Processing certain messages in sequence when Pool Size > 1
We have an HL7 feed that contains Lab Results. Each message contains a set of results, identified by a LabReportId. Some LabReportIds can get more than one message / set of results - these are updates, as new results become available, and must be processed in order. Results for different LabReportIds can be processed in any order.
As each message is used to build a PDF, which takes time, we want to run these through a production where the key business process has a Pool Size > 1, while still ensuring that messages for each LabReportId are processed in order. The key business process is implemented in BPL, if that's relevant.
One suggestion for doing this we've come across is to set the SessionId of the request messages going to the key business process to be the LabReportId:
- are we right in thinking that Ensemble (we'll eventually upgrade to Iris...) will process the messages for a specific SessionId (LabReportId) in sequence?
- are there any "gotchas" we need to watch out for in this kind of scenario?
Comments
For IRIS, since 2025.3, there is a new option for FIFO handling (even with pool size > 1).
See related Docs - Maintaining First-In First-Out (FIFO) Processing and related video Multi-threading an HL7 Interface and scaling beyond FIFO constraints by @James MacKeith
Messages are processed from the queue based on arrival only and not by the SessionId - it may look in some cases that it is based on the SessionId but that is because the SessionId is the Message Header Id for the first message in the session and the Header Id is an incrementing counter.
One thing to bear in mind with BPL is that even with a pool size 1 it is possible for message ordering to become switched if the BPL makes non blocking calls (the default and strength of BPLs). See the docs on BPL and FIFO and reference to using of blocking coded SendRequestSync().
A custom option that some have used concern having dedicated hosts to do the long running conversion that have pool size 1 and a name using a numbering system so that messages can be routed based on a hashing of the FIFO key (LabReportId for example). Another custom option is maintaining a table of what is in progress and being able to delay processing until the other is finished.
The feature that Tani references has at its core a modification to the framework's DeQueuing mechanism to be able to skip a message if its FIFO key is the same as a message already in progress.
Thanks for those that took the time to reply. Summary of where we've got to:
- "structural" support for what we are trying to do isn't there in Ensemble 2018; another piece of ammunition to use when kicking our management into upgrading to Iris 😁
- we had a further think, and also had a conversation with someone from WRC where were able to go into a bit more detail about our application, which resulted in a (very promising looking) solution. We already have a table whose contents includes the MessageBodyId, so we when we come to process a message for a LabReportId we can see whether there is an entry in the table for that LabReportId, and if there is compare the MessageBodyId in the table with that from the current message to ensure they are processed correctly.