ORU to MDM with large HL7 v2 messages.

I was working on a DTL but kept getting ERROR #5002... MAXSTRING errors. The problem was that most of the DTL GUI action steps only support the string data type when working with the segments. A `%String` has a limit of 3,641,144 characters and my OBX5.1 was 5,242,952 characters long as the example provided. Of course PACS admin stated ultra high quality up to and including 4K resolution files were needed, so we could not get the vendor to compress or reformat these files to compressed jpg or something similar.

Initially this vendor sends a 2.3 `ORU^R01` and our EHR (Epic) is expecting a 2.3 `MDM^T02`. Furthermore, we needed the following transformations:

1. The embedded image was sent in OBX-5.1, and we needed it moved to OBX-5.5
2. The image format was sent in OBX-6 and we needed it in OBX-5.3 & 4
3. Needed to create TXA segment
4. Support a set 25 OBX segments that may be completely empty (>25 x 5Mb = 125Mb+ Message sizes, yikes!)

Example received message (replace `...` with 5+ Mb embedded data)

MSH|^~\&|VENDOR||||20241017125335||ORU^R01|1|P|2.3|
PID|||203921||LAST^FIRST^^^||19720706|M||||||||||100001|
PV1||X||||||||GI6|||||||||100001|
ORC|RE||21||SC||1|||||||||||
OBR|1||21|21^VENDOR IMAGES|||20241017123056|||||||||1001^GASTROENTEROLOGY^PHYSICIAN|||||Y||||F||1|
OBX|1|PR|100001|ch1_image_001.bmp|...^^^^^^^|BMP|||||F|
OBX|2|PR|100001|ch1_image_003.bmp|...|BMP|||||F|
OBX|3|PR|100001|ch1_video_01thumbnail.bmp|...|BMP|||||F|
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||
OBX||||||||||||

My normal tools and testing was not up to par with these really large messages. When I used this example replacing the data with the `...` of course the normal DTL drag and drop GUI and testing would play nice, but plug in the real data, and it all crumbled.

Eventually I found that I had to use a code block with ObjectScript using the `%GlobalCharacterStream` data types to work with the large messages correctly.

Sharing my final DTL class for anyone who might come after me and find this helpful

Class OrdRes.VendorMDM Extends Ens.DataTransformDTL [ DependsOn = EnsLib.HL7.Message ]
{
Parameter IGNOREMISSINGSOURCE = 1;
Parameter REPORTERRORS = 1;
Parameter TREATEMPTYREPEATINGFIELDASNULL = 0;
XData DTL [ XMLNamespace = "http://www.intersystems.com/dtl" ]
{
<transform sourceClass='EnsLib.HL7.Message' targetClass='EnsLib.HL7.Message' sourceDocType='2.3:ORU_R01' targetDocType='2.5:MDM_T02' create='new' language='objectscript' >
<assign value='source.{MSH}' property='target.{MSH}' action='set' />
<assign value='"MDM"' property='target.{MSH:MessageType.MessageCode}' action='set' />
<assign value='"T02"' property='target.{MSH:MessageType.TriggerEvent}' action='set' />
<assign value='"2.5"' property='target.{MSH:VersionID.VersionID}' action='set' />
<assign value='source.{MSH:DateTimeofMessage}' property='target.{EVN:2}' action='set' />
<assign value='source.{PIDgrpgrp().PIDgrp.PID}' property='target.{PID}' action='set' />
<assign value='source.{PIDgrpgrp().PIDgrp.PV1grp.PV1}' property='target.{PV1}' action='set' />
<assign value='source.{PIDgrpgrp().ORCgrp().ORC}' property='target.{ORCgrp().ORC}' action='set' />
<assign value='source.{PIDgrpgrp().ORCgrp().OBR}' property='target.{ORCgrp().OBR}' action='set' />
<assign value='source.{PIDgrpgrp().ORCgrp().NTE()}' property='target.{ORCgrp().NTE()}' action='set' />
<assign value='"Endoscopy Image"' property='target.{TXA:DocumentType}' action='set' />
<assign value='"AU"' property='target.{TXA:DocumentCompletionStatus}' action='set' />
<assign value='"AV"' property='target.{TXA:DocumentAvailabilityStatus}' action='set' />
<assign value='source.{PID:18}' property='target.{TXA:12.3}' action='set' />
<code>
<![CDATA[
 set OBXCount=source.GetValueAt("PIDgrpgrp(1).ORCgrp(1).OBXgrp(*)")
 For k1 = 1:1:OBXCount
 {
   // if OBX-1 is empty then it is assumed the rest of the segment will be empty too, so disregard it.
   If source.GetValueAt("PIDgrpgrp(1).ORCgrp(1).OBXgrp("_k1_").OBX:SetIDOBX") '= ""
   {
     // create new stream to read source OBX
     set srcOBXStream=##class(%GlobalCharacterStream).%New()
     // get stream data from source OBX
     set tSC=source.GetFieldStreamRaw(srcOBXStream,"PIDgrpgrp(1).ORCgrp(1).OBXgrp("_k1_").OBX")
     // get the positions of needed delimitters:
     set p1=srcOBXStream.FindAt(1,"|")    // 0>p1="OBX"
     set p2=srcOBXStream.FindAt(p1+1,"|") // p1>p2=OBX-1
     set p3=srcOBXStream.FindAt(p2+1,"|") // p2>p3=OBX-2
     set p4=srcOBXStream.FindAt(p3+1,"|") // p3>p4=OBX-3
     set p5=srcOBXStream.FindAt(p4+1,"|") // p4>p5=OBX-4
     set p6=srcOBXStream.FindAt(p5+1,"^") // p5>p6=OBX-5.1
     set p7=srcOBXStream.FindAt(p6+1,"|") // p6>p7=OBX-5.2 -> OBX 5.*
     // if no OBX-5.2 then there will not be the `^` and p6 and p7 will be `-1`
     // when that is the case, find p7 starting at `p5+1` and make p6 = p7
     if (p6 < 0)
     {
       set p7=srcOBXStream.FindAt(p5+1,"|") // p5>p7=OBX-5
       set p6=p7
     }
     set p8=srcOBXStream.FindAt(p7+1,"|") // p7>p8=OBX-6
     set tStream=##class(%GlobalCharacterStream).%New()
  
     // renumber OBX-1 to OBX 
     set tSC=tStream.Write("OBX|"_k1_"|")
  
     // set OBX2-2 to "ED"
     set tSC=tStream.Write("ED|")
  
     // copy source OBX-3 to target OBX-3
     set tSC=srcOBXStream.MoveTo(p3+1)
     set tSC=tStream.Write(srcOBXStream.Read(p4-p3-1))
     set tSC=tStream.Write("|")
  
     // copy source OBX-4 to target OBX-4
     set tSC=srcOBXStream.MoveTo(p4+1)
     set tSC=tStream.Write(srcOBXStream.Read(p5-p4-1))
  
     // copy source OBX-6 to OBX-5.3 & OBX-5.4
     set tSC=srcOBXStream.MoveTo(p7+1)
     set docType=srcOBXStream.Read(p8-p7-1)
     set tSC=tStream.Write("|^^"_docType_"^"_docType_"^")

     // copy source OBX-5.1 to target OBX-5.5
     // can only set up to 3,641,144 chars at once, so do while loop...
     set startPos=p5+1
     set remain=p6-p5-1
     // characters to read/write in each loop, max is 3,641,144 since .Write limit is a %String
     set charLimit=3000000
     while remain > 0
     {
       set tSC=srcOBXStream.MoveTo(startPos)
       set toRead = charLimit
       if toRead > remain
       {
         set toRead=remain
       }
       set tSC=tStream.Write(srcOBXStream.Read(toRead))
       set remain=remain-toRead
       set startPos=startPos+toRead
     }
     set tSC=tStream.Write("|")

     set obxSegment=##class(EnsLib.HL7.Segment).%New()
     set obxSegment.SegType="2.5:OBX"
     set tSC=obxSegment.StoreRawDataStream(tStream)
     set tSC=target.setSegmentByPath(obxSegment,"OBXgrp("_k1_").OBX")
   }
 }
]]></code>
</transform>
}
}

For developing and testing this, I used the VSCode Plugins for InterSystems because the integrating testing tools could not handle the message size.

I will also add, that getting HL7 over HTTPS to Epic's InterConnect also involved creating a custom HTTP class and sending the custom Content-Type `x-application/hl7-v2+er7`