Remove delimiters which hold not present values in a HL7 message, using a Data Transformation?
Hello,
We need to convert this structure, where we observe empty component separators "^":
MSH|^~\&|sistemaExterno|anonymized^anonymized|anonymized|anonymized|anonymized||RSP^K22^RSP_K21|anonymized|P|2.5^&^&|1||AL|AL||ASCII
MSA|AA||AA
QAK||OK
QPD|Q22^^Find Candidates||anonymized
PID|1|anonymized^1^^^^^^^^|35018^^^anonymized^PI^^^^^~anonymized^1^^TISR^HC^^^^^~292213^2^^12^JHN^^^^^~6339140^^^MPI^CIP^^^^^~""^^^MI^PPN^^^^^~""^^^MI^NNESP^^^^^|292213^^^^^^^^^|ApellidoUno35018^Nombre35018^^^^^^^&^&^^&^&|ApellidoDos35018^^^^^^^^&^&^^&^&|19541111000000|M|||""&AVDA EL PUENTE 64PLB4B&""^""^380374^38^38700^034^^^""_""^""^""^&^&^&||922417062^^^""^^^^^^^^~922417062|922415760^^^^^^^^^^^~""|""|""|034^^^Y|^^^^^^^^^|""|||N|38|||12|""|034^ESPA�A^^N||N|N||||5^A||||""^""^""^N^""
PD1|||12024310^120243^^^^^PTN^1^1202430113A^""~^^^^^^^^""||""||anonymized|TSI 002|01|||N
to this one, where there are not empty component separators "^" nor empty subcomponents separators at component's end like "^&":
MSH|^~\&|sistemaExterno|anonymized^anonymized|anonymized|anonymized|anonymized||RSP^K22^RSP_K21|anonymized|P|2.5|1||AL|AL||ASCII
MSA|AA||AA
QAK||OK
QPD|Q22^^Find Candidates||anonymized
PID|1|anonymized^1|35018^^^anonymized^PI~anonymized^1^^TISR^HC~292213^2^^12^JHN~6339140^^^MPI^CIP~""^^^MI^PPN~""^^^MI^NNESP|292213|ApellidoUno35018^Nombre35018|ApellidoDos35018|19541111000000|M|||""&AVDA EL PUENTE 64PLB4B&""^""^380374^38^38700^034^^^""_""^""^""||922417062^^^""~922417062|922415760~""|""|""|034^^^Y||""|||N|38|||12|""|034^ESPAÑA^^N||N|N||||5^A||||""^""^""^N^""
PD1|||12024310^120243^^^^^PTN^1^1202430113A^""~^^^^^^^^""||""||anonymized|TSI 002|01|||N
Is there any default setting included in Data Transformation's setting to handle this use case?
We are currently using the following code to handle trailing field and component separators. It is being extracted from the following topic created by Randy Pallotta:
https://community.intersystems.com/post/removing-all-trailing-delimiters...
///
/// Per the HL7 standard, empty fields at the end of a segment need not be sent at all.
/// This also applies to empty sub-components at the end of a field.
/// For example, PID|1|12345||DOE^JOHN^^^^||||||||| is equivalent to PID|1|12345||DOE^JOHN.
///
/// We created this function to help with our file compares, so that Ensemble mimics the
/// behavior of Cerner Open Engine which removes trailing pipes automatically. By calling
/// this function, we ensure that Ensemble produces the same output as Cerner Open Engine.
ClassMethod ConversionScrub(pHL7 As EnsLib.HL7.Message) As EnsLib.HL7.Message
{
set tSegCount = pHL7.SegCount
set tCount = pHL7.SegCount
while tCount > 0 {
set tSegment = pHL7.GetValueAt(tCount)
; Remove trailing up carats
while ..StartsWith(($REVERSE(tSegment)),"^") {
set tLength = ..Length(tSegment)
set tSegment = ..SubString(tSegment,1,(tLength-1))
do pHL7.SetValueAt(tSegment,tCount)
} ; Remove trailing up carats within a field
set tIndex = 0
while $FIND(tSegment,"^|",tIndex) > 0 {
set tSegment = $Replace(tSegment,"^|","|")
do pHL7.SetValueAt(tSegment,tCount)
}
; Remove subcomponent & separator?
set tIndex = 0
while $FIND(tSegment,"&&",tIndex) > 0 {
set tSegment = $Replace(tSegment,"&&","")
do pHL7.SetValueAt(tSegment,tCount)
}
; Remove trailing pipes
while ..StartsWith(($REVERSE(tSegment)),"|") {
set tLength = ..Length(tSegment)
set tSegment = ..SubString(tSegment,1,(tLength-1))
do pHL7.SetValueAt(tSegment,tCount)
}
; Remove empty segment
if ..Length(tSegment) = 3 {
do pHL7.SetValueAt("",tCount,"remove")
}
set tCount = tCount-1
}
quit pHL7
}
We have read:
https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls...
https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls...
https://cedocs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?K...
https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls...
As far as I know, there is no configuration option for automatically removing trailing delimiters. So your post-processing approach seems like the way to go.
It looks like Randy's example only removes trailing carats at the end of the segment. I've seen a different version of this code (also by Randy) that includes this additional piece which removes trailing carats from each field in the segment.
; Remove trailing up carats within a field set tIndex = 0 while $FIND(tSegment,"^|",tIndex) > 0 { set tSegment = $Replace(tSegment,"^|","|") do pHL7.SetValueAt(tSegment,tCount) }Wont it create issue with big segments or in case of OBX as OBX:5 may contain different type of values ?
You're right, this code will not work as-is with fields that exceed the max string size (~ 3 MB). If you expect large fields/segments (such as attachments in OBX:5), there are stream-handling methods that you can use.
If OBX:5 is smaller than the max string size then I don't see any problem regardless of what type of data it contains as the OBX:5 content shouldn't include "^".
First of all your source message is missing one segment , i corrected it. below code should work for you .
set target = ##class(EnsLib.HL7.Message).%New()
set target.DocType = source.DocType
write $System.Status.GetErrorText(source.BuildMap(1))
set tSC = $$$OK
try {
set segCount = source.SegCount
for segNo=1:1:segCount {
set oSegment = source.GetSegmentAt(segNo)
set segPath = source.GetSegmentPath(segNo)
set fieldCount = oSegment.GetValueAt("*")
for fieldNo=1:1:fieldCount {
set fieldValue = oSegment.GetValueAt(fieldNo)
set compCount = oSegment.GetValueAt(fieldNo_".*")
for compNo=1:1:compCount { // considering there is no repeat - if repeat then one more loop is required
set subCompCount = oSegment.GetValueAt(fieldNo_"."_compNo_".*")
for subCompNo=1:1:subCompCount {
set path = segPath_":"_fieldNo_"."_compNo_"."_subCompNo
set subCompValue = source.GetValueAt(path)
if (subCompValue '= "") && (subCompValue '= ($C(34)_$C(34))) {
//if (oSegment.Name = "PID")&&(fieldNo=2) break
if path="PIDgrp.PID:13.4.1" break
write path_"--->"_subCompValue,!
do target.SetValueAt(subCompValue,path)
}
}
}
//set componentCount = oSegment.GetValueAt(fieldNo
}
}
do target.OutputToFile("C:\Work\Report\target.hl7",1)
}
catch oEx {
}