Question
Shawn McCartt · Aug 16, 2017

Dynamically modifying RetryInterval and FailureTimeout

Is it possible to dynamically adjust the RetryInterval andFailureTimeout settings in a BPL?

I've got a business process that calls a web service operation to get a session ID from an external system.  There is a string property returned in the body of the response that indicate an exception occurred in the external system. I have code in the BPL that examines the property and sets the status property to an error status when that occurs.

Depending on what the value is I want to adjust the RetryInterval and FailureTimeout values used in by the system when the ReplyCodeActions is set to E=RD.

Example: If the string is "No sessions currently available" I want to retry every 15 seconds for 10 minutes before disabling the process and raising an alert.

If instead the string is "System is down for nightly maintenance"  I want to retry every 5 minutes for 4 hours before disabling the process and raising an alert.

Suggestions?

00
1 0 3 481
Log in or sign up to continue

There are several ways to handle this:

  1. Dynamically adjust BP settings (your suggestion). To do that you need to modify object(s) of Ens.Config.Item class and update the production. Check GetItemSettingValue method in Ens.Director class for an example of getting setting value. After that update the production and that's it. The problem with that approach are:
    • Updated settings take effect for all new incoming/outgoing messages
    • Updating production is a time-consuming operation, it should be used as rarely as possible
  2. Proxy process. Add proxy process that calls your target process. The proxy process should only contain logic related to error processing. And the target process should return the message immediately without any error checks.
  3. Modify the process so it does not return an error each time, but rather sleeps and calls the operation again depending on an error.

Code sample for 1 - utility method to search for setting in a production object.

/// Find BH in a current production by a setting value
ClassMethod findConfigItemBySettingValue(settingName As %String, settingValue As %String, businessType As %String = "", enabledOnly As %Boolean = 0) As Ens.Config.Item
{
    #dim sc As %Status
    
    // Get current production object
    #dim prod As Ens.Config.Production = ..getCurrentProduction()
    if '$isObject(prod) quit ""

    // Cycle over production elements
    #dim item As Ens.Config.Item
    #dim result As Ens.Config.Item = ""
    for i = prod.Items.Count():-1:1
    {
        set item = prod.Items.GetAt(i)
        
        // Search only for specified type
        if ((businessType '= "") && (item.BusinessType() '= businessType)) || (enabledOnly && 'item.Enabled) continue
        
        // Cycle over settings
        do item.PopulateModifiedSettings()
        set ind = ""
        for
        {
            set setting = item.ModifiedSettings.GetNext(.ind)
            if (ind = "") quit
            
            // Found it?
            if (setting.Name = settingName)
            {
                if ($ZStrip(setting.Value, "<>W") = $ZStrip(settingValue, "<>W")) set result = item
                quit
            }
        }
        
        if $isObject(result) quit
    }
    
    quit result
}

/// Get current running production object
ClassMethod getCurrentProduction() As Ens.Config.Production
{
    #dim sc As %Status
    #dim prodName As %String
    #dim prodState As %Integer
    
    // Find the name and status of current production
    set sc = ##class(Ens.Director).GetProductionStatus(.prodName, .prodState)
    if $$$ISERR(sc)
    {
        $$$LOGERROR($System.Status.GetErrorText(sc))
        quit ""
    }
    
    //Status should be "Running"
    if (prodState '= $$$eProductionStateRunning) quit ""
    
    // Open object by name
    #dim prod As Ens.Config.Production = ##class(Ens.Config.Production).%OpenId(prodName, , .sc)
    if $$$ISERR(sc)
    {
        $$$LOGERROR($System.Status.GetErrorText(sc))
        quit ""
    }
    
    quit prod
}

You can use simply:

..RetryInterval

Instead of

%Ensemble("%Process").RetryInterval

Since current object is %Ensemble("%Process").

The requirement to alter the failure timeout was dropped in favor of just retrying indefinitely, so I went with the following solution for the dynamic retry interval and alert grace period:

  • Added new settings to the BPL for the retry interval and alert grace period to use for the different error types
  • When error is detected, modify the RetryInterval and AlertRetryGracePeriod on the current process to use the new values.
<if name="" condition='(context.LogonResponse.LogonResult [ "No sessions are currently available")'>
<true>
<code name='' xpos='200' ypos='350' >
<![CDATA[
 $$$TRACE("LogonResponse.LogonResult indicates all sessions are used [" _ context.LogonResponse.LogonResult _ "]")
 set context.ExceptionString = $$$ERRNOAVAILABLESESSION
 set %Ensemble("%Process").RetryInterval = %Ensemble("%Process").RetryNoSessionAvailableInterval
 set %Ensemble("%Process").AlertRetryGracePeriod = %Ensemble("%Process").AlertNoSessionAvailableGracePeriod
 set status = $System.Status.Error( context.ExceptionString,"no session is available")
 ]]>
</code>
</true>
</if>
<if name="" condition='(context.LogonResponse.LogonResult [ "System Unavailable")'>
<true>
<code name='' xpos='200' ypos='350' >
<![CDATA[
 $$$TRACE("LogonResponse.LogonResult indicates system is offline [" _ context.LogonResponse.LogonResult _ "]")
 set context.ExceptionString = $$$ERRSYSTEMUNAVAILABLE
 set %Ensemble("%Process").RetryInterval = %Ensemble("%Process").RetrySystemUnavailableInterval
 set %Ensemble("%Process").AlertRetryGracePeriod = %Ensemble("%Process").AlertSystemUnavailableGracePeriod
 set status = $System.Status.Error( context.ExceptionString,"no session is available")
 ]]>
</code>
</true>
</if>