Recovering a Mirror after crash

Here we are again with an article related to the Mirror!

In the previous article we saw how we could configure a Mirror between two IRIS instances, one acting as an active node and the other as a passive one. This mirroring system works on the transfer of a journal file that keeps the instance that works as a passive node continuously updated, but what happens if due to some communication failure or permissions of the journal file it is not transferred correctly?

The most common is that in this situation our Mirror remains unconfigured, in such a way that the passive node does not receive the changes made to the active node:

In the previous image we see a Mirror configured with two nodes in failover mode (mirrorA as active node and mirrorB as passive node), but in this case the one defined as passive node (mirrorB) has lost the connection with the active or primary node and therefore has stopped receiving data from it. of the databases included in the Mirror (COMPANY, CUSTOMER and PERSONAL). In this situation, the best thing to do is to discard the data that we have in our passive node and restore the databases from the active node to the passive one again, at this URL you can find the documentation related to the restoration. For this example we are going to opt for the option defined in the documentation as Online Backupwhich will make use of the backup and restore features included in IRIS.

Alright, let's get started then:

Configuring the passive node

As a first step we must leave the passive node that has been deconfigured in a state that allows the restoration of the backup on the mirror databases.

Dismounting the Mirror databases

In order to operate correctly with the affected databases, we must first dismount them and for this we will access the option directly from the management portal: System Operation --> Databases

Let's proceed to dismount the affected databases COMPANY, CUSTOMER and PERSONAL by clicking on the name of each of them and pressing the Dismount button that appears in the new window:

Once the 3 affected databases have been dismounted, we can now operate on them and restore the databases from the active node (mirrorA).

Configuring the active or primary node

As we have previously indicated, we have opted for Online backup as the best option for recovery and for this we must make a backup of the databases configured in the Mirror. As a step prior to generating our backup we are going to define the database list on which we want to work.

Setting the database list

Our idea is to perform a Full Backup of the databases configured in the Mirror and for this we are going to create the list of databases to be saved in our backup file and we are going to do it from the management portal option System Administration --> Configuration --> Database Backup --> Database Backup List

From this menu option we can set our list:

Once the databases that we have configured in our Mirror have been selected, we can proceed to make their backup.

Running the Full Backup

We will access from the management portal the option System Operation --> Backup --> Full Backup List and we will define the path of our server (mirrorA) where we want to store the backup file, in this case, as we have defined a folder accessible from Visual Studio Code (/shared) we will use it to store the backup and be able to easily copy it to the other node (mirrorB):

As you can see when selecting the Full Backup List option we have automatically loaded the previously selected list of databases. Well then, we would only need to press Ok to start the generation of the full backup file. Depending on the size of the databases, it may take more or less, but we can check the status of the backup at all times from the Backup Status screen:

Let's briefly see the log that has been generated:

As you can see, it indicates the databases on which the backup is going to be made and an important detail, the journal file has changed. What does that mean? It means that when we restore the backup on our passive node (mirrorB) when the active journal is synchronized, we will have all the changes made since the creation of the backups fully available and, therefore, we will not need to copy the journal file over our passive node.

For those of you wondering what the journal is exactly, it is nothing more than a file where we are recording the write operations (updating or deleting records) of our database.

Very well, our backup is complete, let's check our /shared/Backup path on our active node (mirrorA):

There is our file FullDBList_20230522_003.cbk with the full backup of our database in our active node (mirrorA). Let's copy it to the server of our passive node (mirrorB), remember that we are using Docker instances for this example, so we will directly use the path accessible from our Visual Studio Code project, in case we have the IRIS instances running on normal servers you can move the files as you usually do.

Once copied, let's check that it is correctly located on the server of our passive node (mirrorB):

Perfect! There it is waiting for our restoration of the passive node (mirrorB), well, let's not keep it waiting.

Restoring the passive node (mirrorB)

We already have our backup in our passive node (mirrorB), so let's open a terminal:

iris terminal IRIS

Let's now follow the steps indicated by the documentation

Let's see each step in detail:

1. zn "%SYS": we change the default namespace USER to %SYS
2. do ^DBREST: we run the restore command
3. Restore: 2. Selected and/or renamed directories: we select this option simply to show all the directories/databases that we are going to restore.
4. Device: /shared/Backup/FullDBList_20230522_003.cbk:  we introduce the name of the backup file that we have copied to our passive node (mirrorB).

We confirm that it is the file that we want to restore:

Once confirmed, we will be informed of the following:

It has been detected that the export has been carried out in the other member of the mirror and asks us if we only want to restore the databases that were configured in the mirror. Since in our export we have only included these databases, it does not matter what we answer, it will take the same ones. Let's set it to YES.

For each backup database we will be able to modify the destination directory, something that does not interest us, so we will simply press enter.

Finally, it will ask us for confirmation of whether we want to modify the list of directories previously selected and finally we must write YES to confirm the restoration.

As you can see, the restoration of the databases will be carried out and finally it will show us the possibility of continuing with more restoration files, since we do not have any more we could write STOP directly, if we press enter it will ask us if we have more backups to restore, in this case we will answer NO.

As you can see, at the end of the restoration the databases are automatically mounted.

Checking the Mirror after the restore

If we access the mirror monitor again we can see the state of it after the restoration:

As we can see, the passive node (mirrorB) has recovered from the failure state it was in with the restore and has started to receive the journal file from the active node (mirrorA) and which was created after performing the backup.

Well done, we have our Mirror fully operational again!