External backup - exceeding the maximum duration specified

Question

Question

Eyal Levin · Jan 28

#Backup #Kubernetes #InterSystems IRIS #InterSystems IRIS for Health

Hi, I was wondering if anyone already dealt with this issue:
"System has been suspended for over X seconds, exceeding the maximum duration specified. Allowing system activity to resume. Any ongoing backup has presumably failed. Next InterSystems IRIS backup must be a full one"

our backup system "Commvault" is automatic, how do you tell it once you get this message that the next backup should be full?

thanks,

Eyal

Product version: IRIS 2023.3

Discussion (11)2

Log in or sign up to continue

Ambrogio De Lorenzo · Jan 30

Your scripts are a bit complex but the backup pre-post command should return 0 if completed correctly.

So backup software can understand if the pre-command (preparation for backup) is completed ok.
If return code is 0 the backup can start saving data.
When backup completes commvault will start the post command.

Also the post command should return 0 if completed correctly.

In your script maybe it's better you should specify exit 0 when needed.

2 0

score 0 · Answer 1 · 2024-01-28T05:30:22-05:00

Ambrogio De Lorenzo · Jan 28

Hello, have you configured some pre and post backup script?

0 0

score 0 · Answer 2 · 2024-01-29T03:00:37-05:00

Hi Ambrogio,

I have configured pre and post scripts and the system should thaw itself after 8 min if an external thaw was not issued, but I need to pass this to my backup software (commvault) in order for it to know it actually failed.

score 0 · Answer 3 · 2024-01-29T04:29:40-05:00

Hi Eyal,

I can't figuring how the system can now that a backup is started without the pre-post script on commvault.
Can you provide the pre-post script to understand what you would like to do?

score 0 · Answer 4 · 2024-01-30T01:20:53-05:00

Hi Ambrogio,

here are my scripts:
Pre:

#!/bin/bash

LOG_DIR=~/Commvault_backup
LOG_FILE=$LOG_DIR/backup-log_$(date +'%d-%m-%y').txt
mkdir -p $LOG_DIR 2>/dev/null
touch $LOG_FILE
if [ `hostname | grep data` ]; then
#Freeze Write Daemon
echo -e "##################################\n" | tee -a $LOG_FILE >> /irissys/data/IRIS/mgr/messages.log
echo "Freezing IRIS Write Daemon" | tee -a $LOG_FILE >> /irissys/data/IRIS/mgr/messages.log
iris session iris -U%SYS "##Class(Backup.General).ExternalFreeze(,,,,,,,,,480)"
status=$?
if [[ $status -eq 5 ]]; then
echo -e "$(date +'%m/%d/%y-%T.%3N') (Wizards) [Backup.Event] [Commvault Backup] IRIS WD IS FROZEN Performing backup (UTC time! +03:00) STATUS = $status (need to be 5) " | tee -a $LOG_FILE >> /irissys/data/IRIS/mgr/messages.log
while [ true ]; do
   if [ "$(tail -n 1 /irissys/data/IRIS/mgr/messages.log | grep "Journal File Compression" | awk '{print $8}')" = "Compressed" ]; then
     echo -e "$(date +'%m/%d/%y-%T.%3N') (Wizards) [Backup.Event] [Commvault Backup] Running... " | tee -a $LOG_FILE >> /irissys/data/IRIS/mgr/messages.log
     echo -e "$(date +'%m/%d/%y-%T.%3N') (Wizards) [Backup.Event] [Commvault Backup] Starting Backup... (UTC time! +03:00) " | tee -a $LOG_FILE >> /irissys/data/IRIS/mgr/messages.log
              break
   fi
          done
elif [[ $status -eq 3 ]]; then
echo -e "$(date +'%m/%d/%y-%T.%3N') (Wizards) [Backup.Event] [Commvault Backup] IRIS WD FREEZE FAILED (UTC time! +03:00) STATUS = $status (need to be 5)" | tee -a $LOG_FILE >> /irissys/data/IRIS/mgr/messages.log
exit 1
fi
echo
else
echo -e "Not data Pod" | tee -a $LOG_FILE

fi

The question is how do I tell commvault that it's backup actually failed?

thanks,

Eyal

Post:

#!/bin/bash
LOG_DIR=~/Commvault_backup
LOG_FILE=$LOG_DIR/backup-log_$(date +'%d-%m-%y').txt
mkdir -p $LOG_DIR 2>/dev/null
touch $LOG_FILE
if [ `hostname | grep data` ]; then
#Thaw Write Daemon
echo -e "\nThaw Write Daemon" | tee -a $LOG_FILE >> /irissys/data/IRIS/mgr/messages.log
iris session iris -U%SYS "##Class(Backup.General).ExternalThaw()"
status=$?
if [[ $status -eq 5 ]]; then
    while [ true ]; do
      if [ "$(tail -n 1 /irissys/data/IRIS/mgr/messages.log | grep "Backup.General.ExternalThaw: System resumed" | awk '{print $6,$7}')" = "System resumed" ]; then
        echo -e "$(date +'%m/%d/%y-%T.%3N') (Wizards) [Backup.Event] [Commvault Backup] IRIS WD IS THAW! (UTC time! +03:00) STATUS = $status (need to be 5) " | tee -a $LOG_FILE >> /irissys/data/IRIS/mgr/messages.log
        break
      else
        echo "wait"
      fi
    done
elif [[ $status -eq 3 ]]; then
    echo -e "$(date +'%m/%d/%y-%T.%3N') (Wizards) [Backup.Event] [Commvault Backup] IRIS WD THAW FAILED (UTC time! +03:00) STATUS = $status (need to be 5)" | tee -a $LOG_FILE >> /irissys/data/IRIS/mgr/messages.log
    exit 1
fi
echo -e "##################################\n" | tee -a $LOG_FILE >> /irissys/data/IRIS/mgr/messages.log
else
echo -e "Not data Pod" | tee -a $LOG_FILE
fi

score 0 · Answer 5 · 2024-01-30T03:47:34-05:00

ok, so I need my post script to return something other than 0 so commvault knows it actually failed

Thanks Ambrogio

score 1 · Answer 6 · 2024-01-30T03:59:30-05:00

Yes, but only if it really fails.

Commvault should know of the backup cause it's do a backup.
The pre-script is used to configure the DB in freeze mode, so you can take a snapshot or a backup.

If the pre-script is ok it should return 0 so commvault can start the backup process.

After the backup completes commvault will launch the post-script, so the DB can return in normal way (thaw).

If the post-script is ok it must return 0 so commvault can understand that the process is completed.

I think you should reengineer your entire backup process with those steps in mind.

score 1 · Answer 7 · 2024-02-01T01:49:00-05:00

Having an instance frozen for 8minutes is not so good in my experience.

Have you considered to move to snapshot based external backups? Using ShadowCopies on windows and LVM snaps on Linux? This will reduce the freeze time to the time used to actually take the snapshot. Then CommVault can backup the snapshot drive while IRIS continues on unfrozen.

score 0 · Answer 8 · 2024-02-01T02:48:38-05:00

I agree with your comment, even if I didn't mentioned it.

Maybe he is using a VM so CommVault can be configured to take a snap on VMFS and copy it.

The bad is the logic in scripts, cause Commvault will not be notified of freeze is ok and to start the backup.

score 0 · Answer 9 · 2024-02-01T09:23:15-05:00

Eyal Levin · Feb 1

It's a kubernetes setup, I can't use shadow copy or something like that

Thanks

Eyal

0 0

score 0 · Answer 10 · 2024-02-01T18:22:56-05:00

hmm i just did a bit of searching and found this Volume Snapshots | Kubernetes
As i have never worked with kubernetes i don't know if this can be leveraged to get a "Backup" volume.