Remove delimiters which hold not present values in a HL7 message, using a Data Transformation?

Question

Question

Yone Moreno · Feb 14, 2020

Hello,

We need to convert this structure, where we observe empty component separators "^":

MSH|^~\&|sistemaExterno|anonymized^anonymized|anonymized|anonymized|anonymized||RSP^K22^RSP_K21|anonymized|P|2.5^&^&|1||AL|AL||ASCII
MSA|AA||AA
QAK||OK
QPD|Q22^^Find Candidates||anonymized
PID|1|anonymized^1^^^^^^^^|35018^^^anonymized^PI^^^^^~anonymized^1^^TISR^HC^^^^^~292213^2^^12^JHN^^^^^~6339140^^^MPI^CIP^^^^^~""^^^MI^PPN^^^^^~""^^^MI^NNESP^^^^^|292213^^^^^^^^^|ApellidoUno35018^Nombre35018^^^^^^^&^&^^&^&|ApellidoDos35018^^^^^^^^&^&^^&^&|19541111000000|M|||""&AVDA EL PUENTE 64PLB4B&""^""^380374^38^38700^034^^^""_""^""^""^&^&^&||922417062^^^""^^^^^^^^~922417062|922415760^^^^^^^^^^^~""|""|""|034^^^Y|^^^^^^^^^|""|||N|38|||12|""|034^ESPA�A^^N||N|N||||5^A||||""^""^""^N^""
PD1|||12024310^120243^^^^^PTN^1^1202430113A^""~^^^^^^^^""||""||anonymized|TSI 002|01|||N

to this one, where there are not empty component separators "^" nor empty subcomponents separators at component's end like "^&":

MSH|^~\&|sistemaExterno|anonymized^anonymized|anonymized|anonymized|anonymized||RSP^K22^RSP_K21|anonymized|P|2.5|1||AL|AL||ASCII
MSA|AA||AA
QAK||OK
QPD|Q22^^Find Candidates||anonymized
PID|1|anonymized^1|35018^^^anonymized^PI~anonymized^1^^TISR^HC~292213^2^^12^JHN~6339140^^^MPI^CIP~""^^^MI^PPN~""^^^MI^NNESP|292213|ApellidoUno35018^Nombre35018|ApellidoDos35018|19541111000000|M|||""&AVDA EL PUENTE 64PLB4B&""^""^380374^38^38700^034^^^""_""^""^""||922417062^^^""~922417062|922415760~""|""|""|034^^^Y||""|||N|38|||12|""|034^ESPAÃ‘A^^N||N|N||||5^A||||""^""^""^N^""
PD1|||12024310^120243^^^^^PTN^1^1202430113A^""~^^^^^^^^""||""||anonymized|TSI 002|01|||N

Is there any default setting included in Data Transformation's setting to handle this use case?

We are currently using the following code to handle trailing field and component separators. It is being extracted from the following topic created by Randy Pallotta:

https://community.intersystems.com/post/removing-all-trailing-delimiters...

/// This function removes trailing pipes "|" and/or up-carats "^" from the given HL7 message.
///
/// Per the HL7 standard, empty fields at the end of a segment need not be sent at all.
/// This also applies to empty sub-components at the end of a field.
/// For example, PID|1|12345||DOE^JOHN^^^^||||||||| is equivalent to PID|1|12345||DOE^JOHN.
///
/// We created this function to help with our file compares, so that Ensemble mimics the
/// behavior of Cerner Open Engine which removes trailing pipes automatically. By calling
/// this function, we ensure that Ensemble produces the same output as Cerner Open Engine.
ClassMethod ConversionScrub(pHL7 As EnsLib.HL7.Message) As EnsLib.HL7.Message
{
   set tSegCount = pHL7.SegCount
   set tCount = pHL7.SegCount
   while tCount > 0 {
    set tSegment = pHL7.GetValueAt(tCount)

    ; Remove trailing up carats
    while ..StartsWith(($REVERSE(tSegment)),"^") {
     set tLength = ..Length(tSegment)
     set tSegment = ..SubString(tSegment,1,(tLength-1))
     do pHL7.SetValueAt(tSegment,tCount)
    }    ; Remove trailing up carats within a field
    set tIndex = 0
    while $FIND(tSegment,"^|",tIndex) > 0 {
     set tSegment = $Replace(tSegment,"^|","|")
     do pHL7.SetValueAt(tSegment,tCount)
    }

   ; Remove subcomponent & separator?
    set tIndex = 0
    while $FIND(tSegment,"&&",tIndex) > 0 {
     set tSegment = $Replace(tSegment,"&&","")
     do pHL7.SetValueAt(tSegment,tCount)
    }

   ; Remove trailing pipes
    while ..StartsWith(($REVERSE(tSegment)),"|") {
     set tLength = ..Length(tSegment)
     set tSegment = ..SubString(tSegment,1,(tLength-1))
     do pHL7.SetValueAt(tSegment,tCount)
    }

   ; Remove empty segment
    if ..Length(tSegment) = 3 {
     do pHL7.SetValueAt("",tCount,"remove")
    }
     set tCount = tCount-1
    }
  quit pHL7
}

We have read:

https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls...

https://cedocs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?K...

https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls...

Discussion (4)2

Log in or sign up to continue

Suman Samanta · Feb 25, 2020

First of all your source message is missing one segment , i corrected it. below code should work for you .

  set target = ##class(EnsLib.HL7.Message).%New()
  set target.DocType = source.DocType
  write $System.Status.GetErrorText(source.BuildMap(1))
  set tSC = $$$OK
  try {
      set segCount = source.SegCount
      for segNo=1:1:segCount {
        set oSegment = source.GetSegmentAt(segNo)
        set segPath = source.GetSegmentPath(segNo)

        set fieldCount = oSegment.GetValueAt("*")
        for fieldNo=1:1:fieldCount {
          set fieldValue = oSegment.GetValueAt(fieldNo)
          set compCount = oSegment.GetValueAt(fieldNo_".*")

          for compNo=1:1:compCount { // considering there is no repeat - if repeat then one more loop is required
            set subCompCount = oSegment.GetValueAt(fieldNo_"."_compNo_".*")
            for subCompNo=1:1:subCompCount {
                set path = segPath_":"_fieldNo_"."_compNo_"."_subCompNo
                set subCompValue = source.GetValueAt(path)

                if (subCompValue '= "") && (subCompValue '= ($C(34)_$C(34))) {
                  //if (oSegment.Name = "PID")&&(fieldNo=2) break
                  if path="PIDgrp.PID:13.4.1" break
                  write path_"--->"_subCompValue,!
                  do target.SetValueAt(subCompValue,path)
                }
            }

          }
          //set componentCount = oSegment.GetValueAt(fieldNo
        }
      }
      do target.OutputToFile("C:\Work\Report\target.hl7",1)
    }
  catch oEx {
     }

0 0

score 0 · Answer 1 · 2020-02-19T16:50:31-05:00

As far as I know, there is no configuration option for automatically removing trailing delimiters. So your post-processing approach seems like the way to go.

It looks like Randy's example only removes trailing carats at the end of the segment. I've seen a different version of this code (also by Randy) that includes this additional piece which removes trailing carats from each field in the segment.

       ; Remove trailing up carats within a field
       set tIndex = 0
       while $FIND(tSegment,"^|",tIndex) > 0 {
           set tSegment = $Replace(tSegment,"^|","|")
           do pHL7.SetValueAt(tSegment,tCount)
       }

score 0 · Answer 2 · 2020-02-25T02:23:11-05:00

Suman Samanta · Feb 25, 2020

Wont it create issue with big segments or in case of OBX as OBX:5 may contain different type of values ?

0 0

score 0 · Answer 3 · 2020-02-25T14:39:09-05:00

You're right, this code will not work as-is with fields that exceed the max string size (~ 3 MB). If you expect large fields/segments (such as attachments in OBX:5), there are stream-handling methods that you can use.

If OBX:5 is smaller than the max string size then I don't see any problem regardless of what type of data it contains as the OBX:5 content shouldn't include "^".