Issues with Automatic Resending in a Business Operation

Hi Pietro,

Not sure if this will help you but I written a retry mechanisme in the BO with 2 additional settings MaxRetries and RetryWaitSeconds. 

See code below:

Class BO.Rest.PostBodyHeaderAuth Extends EnsLib.REST.Operation
{

Parameter ADAPTER = "EnsLib.HTTP.OutboundAdapter";

Parameter INVOCATION = "Queue";

Property HeaderName As %String(MAXLEN = 1000) [ InitialExpression = "Authorization", Required ];

Property HeaderValue As %String(MAXLEN = 10000) [ InitialExpression = "Bearer", Required ];

Property UseSSL As %Integer [ InitialExpression = 1, Required ];

Property MaxRetries As %Integer [ InitialExpression = 3, Required ];

Property RetryWaitSeconds As %Integer [ InitialExpression = 300, Required ];

Property ContentType As %String(MAXLEN = 300) [ InitialExpression = "application/json; charset=utf-8", Required ];

Parameter SETTINGS = "HeaderName:Auth,HeaderValue:Auth,MaxRetries:Retry,RetryWaitSeconds:Retry,UseSSL:Connection,ContentType:ContentType";

Method PostBodyData(pRequest As BO.Rest.PostBodyHeaderAuth.PostBodyHeaderAuthReq, Output pResponse As BO.Rest.PostBodyHeaderAuth.PostBodyHeaderAuthResp) As %Status
{
      Set tSc	= $$$OK
      
      Set tCounter = 0
      
      While (tCounter <= ..MaxRetries)
      {
        //Create a new %Net.HttpRequest
        Set tHttpRequest	= ##class(%Net.HttpRequest).%New()
            
        //Get the serversetting from the Adapter
        Set tHttpRequest.Server	= ..Adapter.HTTPServer
        Set tHttpRequest.Port		= ..Adapter.HTTPPort
            
        //Set Content-Type
        Set tHttpRequest.ContentType = ..ContentType
      
        //Set Body from Request Object
        Do tHttpRequest.EntityBody.Write(pRequest.Body)
        
        // Set Https Flag
        If (..UseSSL = 1) {
            Set tHttpRequest.Https	= 1
            Set tHttpRequest.SSLConfiguration	= ..Adapter.SSLConfig
        } Else {
            Set tHttpRequest.Https	= 0
        }
          
        //Set the Header value for Authentication
        If ($LENGTH(pRequest.HeaderValue) > 0) {
            Set tSc = tHttpRequest.SetHeader(..HeaderName, pRequest.HeaderValue)
        } Else {
            Set tSc = tHttpRequest.SetHeader(..HeaderName, ..HeaderValue)  
        }
        
        //Set Custom Timeouts from Adapter
        Set tHttpRequest.WriteTimeout = ..Adapter.WriteTimeout
        Set tHttpRequest.Timeout = ..Adapter.ResponseTimeout
        Set tHttpRequest.OpenTimeout  = ..Adapter.ConnectTimeout
            
        If ($LENGTH(pRequest.Url) > 0) { 
            Set tUrl = pRequest.Url
        } Else {
            Set tUrl = ..Adapter.URL
        }
      
        //Create a trace with the complete url
        $$$TRACE(..Adapter.HTTPServer_"/"_tUrl)
            
        //Do the post call
        Set tSc = tHttpRequest.Post(tUrl)
    
        //Create the tHttpResponse object
        #dim tHttpResponse As %Net.HttpResponse
          
        //Set the response object	
        Set tHttpResponse = tHttpRequest.HttpResponse
                  
        //If an technical error has occured
        If ($$$ISERR(tSc)) 
        { 
            $$$TRACE("Error:"_$System.Status.GetErrorText(tSc))
            
        } ElseIf ('$IsObject(tHttpResponse)) 
        {
            $$$TRACE("Invalid HTTP Response Object")
            Set tSc=$$$ERROR($$$EnsErrGeneral, "Invalid HTTP Response Object.")
        } 
        Else 
        {
          ;Build Operation Response
          Set pResponse = ##class(BO.Rest.PostBodyHeaderAuth.PostBodyHeaderAuthResp).%New()
          Set pResponse.HTTPStatusCode = tHttpResponse.StatusCode
          Set pResponse.HTTPStatusLine = tHttpResponse.StatusLine
          
          ;Check the HTTPStatusCode
          Set tSuccess = ##class(Util.Functions).IsSuccess(tHttpResponse.StatusCode)
          
          If (tSuccess) {
            Set tCounter = ..MaxRetries + 1
              Do tHttpResponse.Data.Rewind()
              Set pResponse.HTTPResponseData = tHttpResponse.Data
          } Else {
            $$$TRACE("An error HTTP code has been received. HTTPSCode="_tHttpResponse.StatusCode)  
            Try {
                Do tHttpResponse.Data.Rewind()
                Set pResponse.HTTPResponseData = tHttpResponse.Data
            } Catch exception {}
          }
        }
        
        Set tCounter = tCounter+1
        If (tCounter <= ..MaxRetries) {
            $$$TRACE("Retry: "_tCounter_". Wait for "_..RetryWaitSeconds_" seconds")
            HANG ..RetryWaitSeconds	
        }	  
      }
     return tSc
}
}

Sorry if I am a bit late to the discussion.  Let me address your issues in order

• This is not necessarily a problem, but I noticed that the method SendPostRequest is executed again from start to finish for each reply, reinitializing all variables as if it were the first execution, except for the RetryCount variable, which indicates the current resend count.
  • This is expected behavior.  Messages are handled as a specific unit of work (atomic).  When a message errors and is going to retry it is basically put back on the queue to process.  When the retry occurs it is treated as a new message with the exception, as you noted, of the retry count. 
• Setting Retry = 0 prevents message resending, even with active Reply Code Actions (e.g., E=R).
  • Also expected.  With this configuration the reply action code may be set to retry , but this setting is indicating that no (0) retries are to be attempted.
• The timeouts are not respected. Instead, the message is resent every 30 seconds instead of 18 (I also tried other Response Timeout values like 20 or 35 seconds, but nothing changes). The Retry Interval is not respected either; the resending is attempted a number of times equal to FailureTimeout/30 instead of FailureTimeout/RetryInterval, as written in the documentation.
  • This seems to be the main issue.  The ResponseTimeout is a property of the Adapter. The default value of this setting is 30 which is what you are seeing.  It appears that you have a custom Business Operation here that is using the EnsLib.HTTP.OutboundAdapter and that you assign this adapter in code rather than using the ADAPTER parameter.  When you initialize this you should assign the reference to the Adapter property of the Business Operation.  Then when you want to set the response timeout in code you would update the property of the Adapter. I would highly recommend using the ADAPTER parameter unless you need to dynamically change the adapter at runtime.  If done in code at runtime you need to be sure that everything is initialized properly
• After the Failure Timeout expires, an error is generated even if one or more responses have been received. Despite the responses arriving, the BO does not detect them and generates the error: "ERROR 5922: Timed out waiting for response". The message continues to be resent until the Failure Timeout expires, and finally the "ERROR <Ens>ErrFailureTimeout: FailureTimeout of 90 seconds exceeded in lombardia.OMR.BO.Sender; status from last attempt was OK".
  • What is being reported here is the last known status of the operation.  In this case that is the error indicated.   There is a difference between a reply and a response.  The Visual trace is indicating that the operation receive a  reply.  This reply was an error.   The 'Response' is what is received from the Web Server.  This never arrived hence the error.  Business Operations will always receive a reply of some kind.  This reply may indicate an error as in this case or success

If you continue to have issues I would encourage you to reach out to the Worldwide Response Center (WRC) for support.  Additionally you could contact your assigned Sales Engineer.

Regards,

Rich Taylot

Though this post does not address the question/issue raised directly it might help shed some light on some of the settings used.