ORU to MDM with large HL7 v2 messages.

Article

Anthony Master · Oct 17, 2024 2m read

#DTL #HL7 #Interoperability #Mapping #Testing #Tips & Tricks #HealthShare #InterSystems IRIS for Health #Health Connect

I was working on a DTL but kept getting ERROR #5002... MAXSTRING errors. The problem was that most of the DTL GUI action steps only support the string data type when working with the segments. A %String has a limit of 3,641,144 characters and my OBX5.1 was 5,242,952 characters long as the example provided. Of course PACS admin stated ultra high quality up to and including 4K resolution files were needed, so we could not get the vendor to compress or reformat these files to compressed jpg or something similar.

Initially this vendor sends a 2.3 ORU^R01 and our EHR (Epic) is expecting a 2.3 MDM^T02. Furthermore, we needed the following transformations:

The embedded image was sent in OBX-5.1, and we needed it moved to OBX-5.5
The image format was sent in OBX-6 and we needed it in OBX-5.3 & 4
Needed to create TXA segment
Support a set 25 OBX segments that may be completely empty (>25 x 5Mb = 125Mb+ Message sizes, yikes!)

Example received message (replace ... with 5+ Mb embedded data):

MSH|^~\&|VENDOR||||20241017125335||ORU^R01|1|P|2.3|
PID|||203921||LAST^FIRST^^^||19720706|M||||||||||100001|
PV1||X||||||||GI6|||||||||100001|
ORC|RE||21||SC||1|||||||||||
OBR|1||21|21^VENDOR IMAGES|||20241017123056|||||||||1001^GASTROENTEROLOGY^PHYSICIAN|||||Y||||F||1|
OBX|1|PR|100001|ch1_image_001.bmp|...^^^^^^^|BMP|||||F|
OBX|2|PR|100001|ch1_image_003.bmp|...|BMP|||||F|
OBX|3|PR|100001|ch1_video_01thumbnail.bmp|...|BMP|||||F|
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||

My normal tools and testing was not up to par with these really large messages. When I used this example replacing the data with the ... of course the normal DTL drag and drop GUI and testing would play nice, but plug in the real data, and it all crumbled.

Eventually I found that I had to use a code block with ObjectScript using the %GlobalCharacterStream data types to work with the large messages correctly.

Sharing my final DTL class for anyone who might come after me and find this helpful

Class OrdRes.VendorMDM Extends Ens.DataTransformDTL [ DependsOn = EnsLib.HL7.Message ]
{

Parameter IGNOREMISSINGSOURCE = 1;

Parameter REPORTERRORS = 1;

Parameter TREATEMPTYREPEATINGFIELDASNULL = 0;

XData DTL [ XMLNamespace = "http://www.intersystems.com/dtl" ]
{
<transform sourceClass='EnsLib.HL7.Message' targetClass='EnsLib.HL7.Message' sourceDocType='2.3:ORU_R01' targetDocType='2.5:MDM_T02' create='new' language='objectscript' >
<assign value='source.{MSH}' property='target.{MSH}' action='set' />
<assign value='"MDM"' property='target.{MSH:MessageType.MessageCode}' action='set' />
<assign value='"T02"' property='target.{MSH:MessageType.TriggerEvent}' action='set' />
<assign value='"2.5"' property='target.{MSH:VersionID.VersionID}' action='set' />
<assign value='source.{MSH:DateTimeofMessage}' property='target.{EVN:2}' action='set' />
<assign value='source.{PIDgrpgrp().PIDgrp.PID}' property='target.{PID}' action='set' />
<assign value='source.{PIDgrpgrp().PIDgrp.PV1grp.PV1}' property='target.{PV1}' action='set' />
<assign value='source.{PIDgrpgrp().ORCgrp().ORC}' property='target.{ORCgrp().ORC}' action='set' />
<assign value='source.{PIDgrpgrp().ORCgrp().OBR}' property='target.{ORCgrp().OBR}' action='set' />
<assign value='source.{PIDgrpgrp().ORCgrp().NTE()}' property='target.{ORCgrp().NTE()}' action='set' />
<assign value='"Endoscopy Image"' property='target.{TXA:DocumentType}' action='set' />
<assign value='"AU"' property='target.{TXA:DocumentCompletionStatus}' action='set' />
<assign value='"AV"' property='target.{TXA:DocumentAvailabilityStatus}' action='set' />
<assign value='source.{PID:18}' property='target.{TXA:12.3}' action='set' />
<code>
<![CDATA[
 set OBXCount=source.GetValueAt("PIDgrpgrp(1).ORCgrp(1).OBXgrp(*)")
 For k1 = 1:1:OBXCount
 {
   // if OBX-1 is empty then it is assumed the rest of the segment will be empty too, so disregard it.
   If source.GetValueAt("PIDgrpgrp(1).ORCgrp(1).OBXgrp("_k1_").OBX:SetIDOBX") '= ""
   {
     // create new stream to read source OBX
     set srcOBXStream=##class(%GlobalCharacterStream).%New()
     // get stream data from source OBX
     set tSC=source.GetFieldStreamRaw(srcOBXStream,"PIDgrpgrp(1).ORCgrp(1).OBXgrp("_k1_").OBX")
     // get the positions of needed delimitters:
     set p1=srcOBXStream.FindAt(1,"|")    // 0>p1="OBX"
     set p2=srcOBXStream.FindAt(p1+1,"|") // p1>p2=OBX-1
     set p3=srcOBXStream.FindAt(p2+1,"|") // p2>p3=OBX-2
     set p4=srcOBXStream.FindAt(p3+1,"|") // p3>p4=OBX-3
     set p5=srcOBXStream.FindAt(p4+1,"|") // p4>p5=OBX-4
     set p6=srcOBXStream.FindAt(p5+1,"^") // p5>p6=OBX-5.1
     set p7=srcOBXStream.FindAt(p6+1,"|") // p6>p7=OBX-5.2 -> OBX 5.*
     // if no OBX-5.2 then there will not be the `^` and p6 and p7 will be `-1`
     // when that is the case, find p7 starting at `p5+1` and make p6 = p7
     if (p6 < 0) {
       set p7=srcOBXStream.FindAt(p5+1,"|") // p5>p7=OBX-5
       set p6=p7
     }
     set p8=srcOBXStream.FindAt(p7+1,"|") // p7>p8=OBX-6
     set tStream=##class(%GlobalCharacterStream).%New()

     // renumber OBX-1 to OBX 
     set tSC=tStream.Write("OBX|"_k1_"|")

     // set OBX2-2 to "ED"
     set tSC=tStream.Write("ED|")

     // copy source OBX-3 to target OBX-3
     set tSC=srcOBXStream.MoveTo(p3+1)
     set tSC=tStream.Write(srcOBXStream.Read(p4-p3-1))
     set tSC=tStream.Write("|")

     // copy source OBX-4 to target OBX-4
     set tSC=srcOBXStream.MoveTo(p4+1)
     set tSC=tStream.Write(srcOBXStream.Read(p5-p4-1))

     // copy source OBX-6 to OBX-5.3 & OBX-5.4
     set tSC=srcOBXStream.MoveTo(p7+1)
     set docType=srcOBXStream.Read(p8-p7-1)
     set tSC=tStream.Write("|^^"_docType_"^"_docType_"^")

     // copy source OBX-5.1 to target OBX-5.5
     // can only set up to 3,641,144 chars at once, so do while loop...
     set startPos=p5+1
     set remain=p6-p5-1
     // characters to read/write in each loop, max is 3,641,144 since .Write limit is a %String
     set charLimit=3000000
     while remain > 0 {
       set tSC=srcOBXStream.MoveTo(startPos)
       set toRead = charLimit
       if toRead > remain {
         set toRead=remain
       }
       set tSC=tStream.Write(srcOBXStream.Read(toRead))
       set remain=remain-toRead
       set startPos=startPos+toRead
     }
     set tSC=tStream.Write("|")

     set obxSegment=##class(EnsLib.HL7.Segment).%New()
     set obxSegment.SegType="2.5:OBX"
     set tSC=obxSegment.StoreRawDataStream(tStream)
     set tSC=target.setSegmentByPath(obxSegment,"OBXgrp("_k1_").OBX")
   }
 }
]]></code>
</transform>
}

}

For developing and testing this, I used the VSCode Plugins for InterSystems because the integrating testing tools could not handle the message size.

I will also add, that getting HL7 over HTTPS to Epic's InterConnect also involved creating a custom HTTP class and sending the custom Content-Type x-application/hl7-v2+er7

// Get the stream data (no need to instantiate a stream object in advance) do source.GetFieldStreamRaw(.tStream,"PIDgrpgrp(1).ORCgrp(1).OBXgrp("_k1_").OBX:5(1).1",.tRem) // // Insert code here to modify tRem to accommodate any changes needed to // fields after OBX:5(1).5 // // Store the stream to the appropriate target field do target.StoreFieldStreamRaw(tStream,"OBXgrp("_k1_").OBX:5(1).5",tRem)

Class OrdRes.VendorMDM Extends Ens.DataTransformDTL [ DependsOn = EnsLib.HL7.Message ] { Parameter IGNOREMISSINGSOURCE = 1; Parameter REPORTERRORS = 1; Parameter TREATEMPTYREPEATINGFIELDASNULL = 0; XData DTL [ XMLNamespace = "http://www.intersystems.com/dtl" ] { <transform sourceClass='EnsLib.HL7.Message' targetClass='EnsLib.HL7.Message' sourceDocType='2.3:ORU_R01' targetDocType='2.5:MDM_T02' create='new' language='objectscript' > <assign value='source.{MSH}' property='target.{MSH}' action='set' /> <assign value='"MDM"' property='target.{MSH:MessageType.MessageCode}' action='set' /> <assign value='"T02"' property='target.{MSH:MessageType.TriggerEvent}' action='set' /> <assign value='"2.5"' property='target.{MSH:VersionID.VersionID}' action='set' /> <assign value='source.{MSH:DateTimeofMessage}' property='target.{EVN:2}' action='set' /> <assign value='source.{PIDgrpgrp(1).PIDgrp.PID}' property='target.{PID}' action='set' /> <assign value='source.{PIDgrpgrp(1).PIDgrp.PV1grp.PV1}' property='target.{PV1}' action='set' /> <assign value='source.{PIDgrpgrp(1).ORCgrp(1).ORC}' property='target.{ORCgrp(1).ORC}' action='set' /> <assign value='source.{PIDgrpgrp(1).ORCgrp(1).OBR}' property='target.{ORCgrp(1).OBR}' action='set' /> <assign value='source.{PIDgrpgrp(1).ORCgrp(1).NTE()}' property='target.{ORCgrp(1).NTE()}' action='set' /> <assign value='"Endoscopy Image"' property='target.{TXA:DocumentType}' action='set' /> <assign value='"AU"' property='target.{TXA:DocumentCompletionStatus}' action='set' /> <assign value='"AV"' property='target.{TXA:DocumentAvailabilityStatus}' action='set' /> <foreach property='source.{PIDgrpgrp(1).ORCgrp(1).OBXgrp()}' key='k1' > <assign value='source.{PIDgrpgrp(1).ORCgrp(1).OBXgrp(k1).OBX:SetIDOBX}' property='target.{OBXgrp(k1).OBX:SetIDOBX}' action='set' /> <assign value='source.{PIDgrpgrp(1).ORCgrp(1).OBXgrp(k1).OBX:ValueType}' property='target.{OBXgrp(k1).OBX:ValueType}' action='set' /> <assign value='source.{PIDgrpgrp(1).ORCgrp(1).OBXgrp(k1).OBX:ObservationIdentifier}' property='target.{OBXgrp(k1).OBX:ObservationIdentifier}' action='set' /> <assign value='source.{PIDgrpgrp(1).ORCgrp(1).OBXgrp(k1).OBX:ObservationSubID}' property='target.{OBXgrp(k1).OBX:ObservationSubID}' action='set' /> <assign value='source.{PIDgrpgrp(1).ORCgrp(1).OBXgrp(k1).OBX:Units.identifier}' property='target.{OBXgrp(k1).OBX:5.3}' action='set' /> <assign value='source.{PIDgrpgrp(1).ORCgrp(1).OBXgrp(k1).OBX:Units.identifier}' property='target.{OBXgrp(k1).OBX:5.4}' action='set' /> <if condition='source.{PIDgrpgrp(1).ORCgrp(1).OBXgrp(k1).OBX:SetIDOBX}' > <true> <code> <![CDATA[ do source.GetFieldStreamRaw(.tStream,"PIDgrpgrp(1).ORCgrp(1).OBXgrp("_k1_").OBX:5(1).1",.tRem) // set tRem = "|PDF|||||F|" // // Store the stream to the appropriate target field do target.StoreFieldStreamRaw(tStream,"OBXgrp("_k1_").OBX:5(1).5",tRem)]]></code> </true> <false> <assign value='source.{PIDgrpgrp(1).ORCgrp(1).OBXgrp(k1).OBX}' property='target.{OBXgrp(k1).OBX}' action='set' /> </false> </if> </foreach> <assign value='source.{PID:18}' property='target.{TXA:12.3}' action='set' /> </transform> } }

Julian Matthews · Oct 21, 2024

Hey Anthony.

Depending on your version of Iris, I would recommend swapping out your use of %GlobalCharacterStream with %Stream.GlobalCharacter as the former is depreciated. Additionally, I would recommend swapping them out for their temp couterparts so you're not inadvertently creating loads of orphaned global streams, especially where you're dealing with files of this size.

1 0

Anthony Master · Oct 21, 2024

I am open for performance suggestions. We are using version: IRIS for Windows (x86-64) 2023.1.3 (Build 517U) Wed Jan 10 2024 13:36:58 EST [HealthConnect:5.1.0-2.m3]

As I am fairly new to InterSystems, I had to sort of figure this out by perusing the docs and trial and error. Can you explain what you mean by the "temp counterparts"?

0 0

Jeffrey Drumm · Oct 21, 2024

Hi Anthony,

I think the issue is that you're using GetFieldStreamRaw() against the entire OBX segment, when you should be using it against the field that contains the stream: OBX:5.1. The method can take 3 arguments, the 3rd being a variable passed by reference that contains the remainder of the current OBX segment. That variable is of type %String and can be modified to include different values for the remaining fields, and then supplied as the 3rd argument to StoreFieldStreamRaw() ... which you would use to populate OBX:5.5.

These methods are usually used in a code block, where passing a variable by reference is supported (precede it with a period). You'll need to do that with both the first and 3rd arguments in GetFieldStreamRaw().

It's also important to note that once you've used StoreFieldStreamRaw(), the target segment becomes immutable; no further changes can be made to it. That's why the remainder variable is so important as it populates the remainder of the segment at the time the stream is stored to the field.

The DTL flow would Look like this:

Populate everything in the target message, up to the OBX
In a Foreach over the OBX:
1. Populate everything in the target OBX preceding OBX:5.5
2. Execute a code block similar to the following:

Then populate any remaining segments as you normally would.

Given the length of the OBX segment, I could not get the OBX-6 from source and set it in the OBX-5.3 & 5.4 as needed. Once I went down the path of reading the OBX entire segment from the stream, it just made sense to continue using a stream and code block for the entire OBX segment.

I also tried to do a foreach loop in the GUI fashion, and then use the `k1` in the code block, but that didn't seem to work, probably just my lack of experience, so I found something that worked and just utilized it instead.

Thank you for the feedback though ;)

The "length" of the OBX segment is only relevant if you're attempting to treat it as a string. If you treat it as an object and use the GUI's copy rules (which leverage the EnsLib.HL7.Message and EnsLib.HL7.Segment classes' methods), those fields should be readily accessible.

Try it yourself, you will not be able to access OBX-6 through regular means of the GUI if the OBX-5 is over the limitation of %String. I tested this and spent a lot of time troubleshooting around it.

I did:

Now, I used PDFs rather than BMPs, I'm a little OCD, so my output looks slightly different from yours. But it does work. Notice that I used the numeric syntax to reference OBX:5's components, though. There are no symbolic names for those components in HL7, but they're still recognized using the numeric syntax.

Also, I think one of the OBX:5 components should probably contain "Base64" since that's probably how OBX:5.5 is encoded.

Here's the output:

2 0