New post
Question
· Nov 23, 2022

What is the likelihood of encountering "missing messages" in message bank?

#Containerization #Enterprise Message Bank #InterSystems IRIS

My team works on implementing an Interoperability solution utilizing InterSystems Kubernetes Operator on Red Hat OpenShift container platform. 

We are trying to determine how many messages we can process in any given time. We have a Feeder app running in 10 containers sending 50k messages each to a load balancer all beginning at the same time.

Messages are received via HTTPS protocol by webgateway containers. 

Interoperability production runs in compute pods with persistent data, journals, and WIJ volumes.

We implemented Horizontal Pod Autoscaler to scale compute pods when CPU utilization is high.

We utilize Enterprise Message Bank to have one place to find any message processed by any compute.

We observe the queue for Message Bank operation grows quite large in compute pods.

Sometimes the Autoscaler scales computes down while they are still processing.

How likely is it Messages do not show up in Message Bank if they have been partially or completed processed in compute pods?

Any messages in queues on compute pods which are shutdown will not be processed until the compute pods is started again.

Product version: IRIS 2022.1
Discussion (1)2
Log in or sign up to continue
Mark Bolinsky · Jan 18, 2023

Message delivery is dependent on queuing, so if messages on the compute node(s) haven't completed through the production - especially on the outbound send to the Message Bank.  The Message Bank will only "bank" those messages that have been actually sent to it.  If there are still messages queued in the production, they will remain there in the production queues until those pods are restarted or PreStop hooks are used to allow the POD to have a grace period on container shutdown until all queues are empty.  An Interoperability Production is a stateful set, and the queues are required to support message delivery guarantees.  

0 0