Open ExchangeChecking your Backups and the number of journals to keep
Working in support, I usually get asked how many days I should keep journals. Should it be two days or after two backups? More? Less? Why two?
The correct answer (for most of the environments) is that you should keep the journals since the last validated Backup. I.e., until you don't check if a Backup is valid (restoring the file and checking with the Integrity utility), you can't be sure there is a good copy of your data and can't purge the journals safely.
For example, imagine you need to restore your system after a hardware failure that corrupted some databases. The first step is to go to the last backups and restore them. But, what happens if the Backup is corrupt or saved into a defective disk? You will need to look for a previous Backup until you find a correct and clean copy. Then, if you want to recover to the very last moment, you will need to apply the journals. If you only have one or two days, it won't be enough, and you may lose data.
The only way to make sure that your backups are valid is by checking them! And for checking, I mean restoring the databases and validating the data on them. The way to check the data inside a database is by using the Integrity utility.
When I explain this, most administrators consider it a complex task, which takes a lot of time and is hard to automate. So, I decided to build a super simple Backup checker/validator that helps you validate Backups easily. I am uploading the utility to the open exchange site.
The utility is a simple Backup checker/validator for Backups done with InterSystems Iris. It will restore your Backup file (.cbk) automatically and will run an integrity report afterward. All the "magic" is done in the restoreAll method of the Installer class. You can borrow the code and improve it to send you an email when it finishes with the results.
Once the Backup has been restored and the Integrity check run, the docker log (and messages.log) will contain the restore and integrity check results. The databases restored will appear in a Restore folder.
Mario, where is the link to the utility?
Added link ;-)
Well done @Mario Sanchez Macias !! A great utility to keep everyone from having to re-invent the wheel themselves :) Thank you for contributing
running the docker image uncovered the result (interactive:=> docker-compose up)
only at stopping of the container. So I changed installer.sh and run the method from iris session
This gave me the full dimension of this EXCELLENT TOOL! especially the integrity check!
I just can say WOW.
So delegating of routine jobs of a System Manager becomes safe and easy.
This is the output that kicked me:
------------------------------------------------------------------------------- irisowner@7aa9b053e544:/opt/irisapp$ iris session iris -U "%SYS" Node: 7aa9b053e544, Instance: IRIS %SYS>do ##class(BackupChecker.Installer).restoreAll("backup.cbk") Starting to restore the Backup... It will take some time, be patient! The following directories will be restored: c:\intersystems\iris\mgr\appcode\ => /ISC/utiles/Restore/c:\intersystems\iris\mgr\appcode\/ c:\intersystems\iris\mgr\appdata\ => /ISC/utiles/Restore/c:\intersystems\iris\mgr\appdata\/ c:\intersystems\iris\mgr\cacheuser\ => /ISC/utiles/Restore/c:\intersystems\iris\mgr\cacheuser\/ c:\intersystems\iris\mgr\remote\ => /ISC/utiles/Restore/c:\intersystems\iris\mgr\remote\/ c:\intersystems\iris\mgr\user\ => /ISC/utiles/Restore/c:\intersystems\iris\mgr\user\/ Cannot mount database in /ISC/utiles/Restore/c:\intersystems\iris\mgr\cacheuser\/ because its default collation is not available (see messages.log). c:\intersystems\iris\mgr\cacheuser\ will not be restored. Cannot mount database in /ISC/utiles/Restore/c:\intersystems\iris\mgr\remote\/ because its default collation is not available (see messages.log). c:\intersystems\iris\mgr\remote\ will not be restored. ***Restoring c:\intersystems\iris\mgr\appcode\ to /ISC/utiles/Restore/c:\intersystems\iris\mgr\appcode\/ at 10:18:55 469 blocks restored in 0.0 seconds for this pass, 469 total restored. ***Restoring c:\intersystems\iris\mgr\appdata\ to /ISC/utiles/Restore/c:\intersystems\iris\mgr\appdata\/ at 10:18:55 413 blocks restored in 0.0 seconds for this pass, 413 total restored. Starting skip of c:\intersystems\iris\mgr\cacheuser\. skipped 77 blocks in .007096 seconds. Starting skip of c:\intersystems\iris\mgr\remote\. skipped 77 blocks in .006635 seconds. ***Restoring c:\intersystems\iris\mgr\user\ to /ISC/utiles/Restore/c:\intersystems\iris\mgr\user\/ at 10:18:55 656 blocks restored in 0.1 seconds for this pass, 656 total restored. ***Restoring c:\intersystems\iris\mgr\appcode\ to /ISC/utiles/Restore/c:\intersystems\iris\mgr\appcode\/ at 10:18:55 1 blocks restored in 0.0 seconds for this pass, 470 total restored. ***Restoring c:\intersystems\iris\mgr\appdata\ to /ISC/utiles/Restore/c:\intersystems\iris\mgr\appdata\/ at 10:18:55 1 blocks restored in 0.0 seconds for this pass, 414 total restored. Starting skip of c:\intersystems\iris\mgr\cacheuser\. skipped 1 blocks in .000007 seconds. Starting skip of c:\intersystems\iris\mgr\remote\. skipped 1 blocks in .000008 seconds. ***Restoring c:\intersystems\iris\mgr\user\ to /ISC/utiles/Restore/c:\intersystems\iris\mgr\user\/ at 10:18:55 1 blocks restored in 0.0 seconds for this pass, 657 total restored. ***Restoring c:\intersystems\iris\mgr\appcode\ to /ISC/utiles/Restore/c:\intersystems\iris\mgr\appcode\/ at 10:18:55 1 blocks restored in 0.0 seconds for this pass, 471 total restored. ***Restoring c:\intersystems\iris\mgr\appdata\ to /ISC/utiles/Restore/c:\intersystems\iris\mgr\appdata\/ at 10:18:55 1 blocks restored in 0.0 seconds for this pass, 415 total restored. Starting skip of c:\intersystems\iris\mgr\cacheuser\. skipped 1 blocks in .000022 seconds. Starting skip of c:\intersystems\iris\mgr\remote\. skipped 1 blocks in .000008 seconds. ***Restoring c:\intersystems\iris\mgr\user\ to /ISC/utiles/Restore/c:\intersystems\iris\mgr\user\/ at 10:18:55 1 blocks restored in 0.0 seconds for this pass, 658 total restored. Mounting /ISC/utiles/Restore/c:\intersystems\iris\mgr\appcode\/ /ISC/utiles/Restore/c:\intersystems\iris\mgr\appcode\/ ... (Mounted) Mounting /ISC/utiles/Restore/c:\intersystems\iris\mgr\appdata\/ /ISC/utiles/Restore/c:\intersystems\iris\mgr\appdata\/ ... (Mounted) Mounting /ISC/utiles/Restore/c:\intersystems\iris\mgr\user\/ /ISC/utiles/Restore/c:\intersystems\iris\mgr\user\/ ... (Mounted) [Journal not applied to any directory] Restored completed! Starting the Integrity report Directory: /ISC/install/mgr/ No errors found Summary of blocks in /ISC/install/mgr/ 87 Pointer Level blocks 696kb (16% full) 5,908 Data Level blocks 46MB (84% full) 2,553 Big String blocks 19MB (86% full) # = 1,011 8,563 Total blocks 66MB (84% full) 2,951 Free blocks 23MB Elapsed time = 14.3 seconds. Directory: /ISC/install/mgr/irisaudit/ No errors found Summary of blocks in /ISC/install/mgr/irisaudit/ 34 Pointer Level blocks 272kb (0% full) 54 Data Level blocks 432kb (33% full) 4 Big String blocks 32kb (99% full) # = 1 107 Total blocks 856kb (20% full) 21 Free blocks 168kb Elapsed time = 6.1 seconds. Directory: /ISC/install/mgr/irislocaldata/ No errors found Summary of blocks in /ISC/install/mgr/irislocaldata/ 35 Pointer Level blocks 280kb (1% full) 213 Data Level blocks 1704kb (58% full) 25 Big String blocks 200kb (74% full) # = 13 288 Total blocks 2304kb (49% full) 1,130 Free blocks 9040kb Elapsed time = 6.5 seconds. Directory: /ISC/install/mgr/iristemp/ No errors found Summary of blocks in /ISC/install/mgr/iristemp/ 46 Pointer Level blocks 368kb (0% full) 81 Data Level blocks 648kb (33% full) 0 Big String blocks 142 Total blocks 1136kb (19% full) 1,177 Free blocks 9416kb Elapsed time = 8.5 seconds. Directory: /ISC/install/mgr/user/ No errors found Summary of blocks in /ISC/install/mgr/user/ 50 Pointer Level blocks 400kb (0% full) 73 Data Level blocks 584kb (27% full) 0 Big String blocks 138 Total blocks 1104kb (14% full) 1,270 Free blocks 10160kb Elapsed time = 8.9 seconds. Directory: /ISC/utiles/Restore/c:\intersystems\iris\mgr\appcode\/ No errors found Summary of blocks in /ISC/utiles/Restore/c:\intersystems\iris\mgr\appcode\/ 32 Pointer Level blocks 256kb (2% full) 379 Data Level blocks 3032kb (71% full) 43 Big String blocks 344kb (88% full) # = 12 469 Total blocks 3752kb (66% full) 1,067 Free blocks 8536kb Elapsed time = 0.1 seconds. Directory: /ISC/utiles/Restore/c:\intersystems\iris\mgr\appdata\/ No errors found Summary of blocks in /ISC/utiles/Restore/c:\intersystems\iris\mgr\appdata\/ 57 Pointer Level blocks 456kb (0% full) 241 Data Level blocks 1928kb (66% full) 100 Big String blocks 800kb (92% full) # = 30 413 Total blocks 3304kb (61% full) 995 Free blocks 7960kb Elapsed time = 0.1 seconds. Directory: /ISC/utiles/Restore/c:\intersystems\iris\mgr\user\/ No errors found Summary of blocks in /ISC/utiles/Restore/c:\intersystems\iris\mgr\user\/ 68 Pointer Level blocks 544kb (1% full) 447 Data Level blocks 3576kb (70% full) 126 Big String blocks 1008kb (90% full) # = 33 656 Total blocks 5248kb (65% full) 752 Free blocks 6016kb Elapsed time = 0.2 seconds. Directory: /usr/irissys/mgr/enslib/ No errors found Summary of blocks in /usr/irissys/mgr/enslib/ 130 Pointer Level blocks 1040kb (33% full) 14,684 Data Level blocks 114MB (97% full) 4,407 Big String blocks 34MB (88% full) # = 1,606 19,236 Total blocks 150MB (94% full) 1,500 Free blocks 11MB Elapsed time = 12.6 seconds. Directory: /usr/irissys/mgr/irislib/ No errors found Summary of blocks in /usr/irissys/mgr/irislib/ 180 Pointer Level blocks 1440kb (60% full) 39,001 Data Level blocks 304MB (98% full) 11,177 Big String blocks 87MB (86% full) # = 4,234 50,373 Total blocks 393MB (95% full) 59 Free blocks 472kb Elapsed time = 7.9 seconds. Directory: /usr/irissys/mgr/zpm/ No errors found Summary of blocks in /usr/irissys/mgr/zpm/ 44 Pointer Level blocks 352kb (11% full) 2,129 Data Level blocks 16MB (97% full) 453 Big String blocks 3624kb (88% full) # = 175 2,641 Total blocks 20MB (94% full) 431 Free blocks 3448kb Elapsed time = 7.7 seconds. Run successfully and found NO errors! Databases are VALIDATED! %SYS> -------------------------------------------------------------------------------It was an additional pleasure to see your verbose comments in dock-compose.yml.
not just the usual plain parameters with no explanation of what the intention is!
Thanks a lot!
That was my intention, trying to show something as simple as possible so anyone can look inside and adapt the code to what they need.
To make it simpler I could have used the API methods and calls provided by default, but I preferred to "hack" those methods to allow restore where I wanted without requesting any prompt and finally running the Integrity check with multiple process. This also allow me to control the return of the integrity report and execute an IF to check if there was an error or not.
I was logging to the messages.log (which is also displayed in the docker logs) just in case you prefer to run it not interactively or in a docker up session. You can of course leave the instance up and call the method.
I didn't want to spend much time in the way of calling the docker instance as I am sure that in a real environment most of the customers will have to write a script moving or renaming the .cbk files and executing the app.
Social networks
InterSystems resources
Log in or sign up
Log in or create a new account to continue
Log in or sign up
Log in or create a new account to continue
Log in or sign up
Log in or create a new account to continue