How to implement special character stripping in Healthshare DTL Transform

Question

Question

Adam Trahan · Nov 8, 2021

I have an incoming data partner who occasionally passes a non-ASCII character in some free text fields which is causing some downstream systems to be unable to accept the message. Ideally I'm looking for some way to just process the entire message as a data transformation applied in the business rule, as we already have a pre-processor transform to remove the PD1 segment.

ClassMethod Transform(pRequest As EnsLib.HL7.Message, Output pResponse As EnsLib.HL7.Message) As %Status
{ Set $ZT="Trap",tSC=$$$OK
Set pResponse = pRequest.%ConstructClone()

//Clearing PD1 segment from the message
set ind=""
Set pPD1seg = pResponse.FindSegment("PD1", .ind)
If pPD1seg '="" {
set res=pResponse.RemoveSegmentAt(ind)
}
set pRequest = pResponse
Exit
Quit tSC
Trap
Set $ZT=""
Set tSC = $$$EnsError($$$GeneralError, "Problem handling unwanted PD1 segment")
Goto Exit
}

Product version: HealthShare 2020.2

$ZV: IRIS for UNIX (Red Hat Enterprise Linux for x86-64) 2020.1 (Build 217_1_21023U)

Discussion (4)1

Log in or sign up to continue

Adam Trahan · Nov 10, 2021

I ended up opening a WRC for this. We were able to resolve this by utilizing Enslib.HL7.Message's OutputToString and ImportFromString methods to just store the entire contents in a string that could then be manipulated directly.

Including the entire method here, in case someone else stumbles upon this and needs it:

ClassMethod Transform(pRequest As EnsLib.HL7.Message, Output pResponse As EnsLib.HL7.Message) As %Status
{
// Get the MSH values for logging purposes
set MSHID = pRequest.GetValueAt("MSH:10")
set MSHDT = pRequest.GetValueAt("MSH:7") // Save the whole HL7 message into a string so we can manipulate it
set data = pRequest.OutputToString()

// Loop through the string, replacing Non-ASCII characters with spaces until you don't find any more
set tEnd = 1
Set tLocation = $LOCATE(data,"[^\x09-\x0d\x20-\x7e]",,tEnd)
While tLocation {
set ^ASCIIStrip($NOW(),$INCREMENT(i))="Non-ASCII Character Found at Position " _ (tEnd-1) _ " in Msg ID " _ MSHID _ " with MSH Date " _ MSHDT
// replace with a space
Set $EXTRACT(data,tLocation,(tEnd-1)) = " "
Set tEnd = 1
Set tLocation = $LOCATE(data,"[^\x09-\x0d\x20-\x7e]",,tEnd)
} // Save that modified string back into an HL7 message
set pResponse = pResponse.ImportFromString(data) // Manually set the DocType and MessageTypeCategory from the source
set tSC = pResponse.PokeDocType(pRequest.DocType)
set tSC = pResponse.PokeTypeCategory(pRequest.MessageTypeCategory) Quit 1
}

0 0

score 2 · Answer 1 · 2021-11-08T15:54:38-05:00

I know exactly nothing about HealthShare... so I can just suggest two ways to remove the unwanted characters from a string:

use $zstrip()

set inpData="some wild sequence of characters"
set cleanData = $zstrip(inpData, "*C") // this removes all control characters (0x00-0x1f, 0x7f-0x9f)

the other way is to define a set of valid characters and remove all others

set inpData="some wild sequence of characters"
set validChars = "012...89ABC..Zabc..z..."
set badChars = $translate(inpData, validChars) // remove from input all valid chars, leftover are bad chars
set cleanData = $translate(inpData, badChars) // remove all the bad chars

or just the short version

set cleanData = $translate(inpData, $translate(inpData, validChars))

score 0 · Answer 2 · 2021-11-09T09:54:57-05:00

Thanks, that does help, but I still have the problem of figuring out how to address the entirety of the message. There is a RawContent property, but it's truncated.

score 0 · Answer 3 · 2021-11-11T16:27:35-05:00

Here is an approach I used to solve this issue when I was first learning ObjectScript. Outcome I think is the same as what WRC helped you with but maybe a little less verbose. This method would have been in a Utils class that extends Ens.Rule.FunctionSet and called at the end of a DTL like:

set target ..StripMessage(target, "\x7e")

Note I had it doing all segments in reverse order and never doing the MSH segment (tCount > 1) so we didn't accidently remove a encoding character like & or \.

/// Engineer: Craig Regester <br/>
/// Date: 12/09/2016<br/>
/// This function will loop through each segment in the supplied HL7 message EXCEPT MSH <br/>
/// and strip out each instance of the character passed in parameter InvalidChar. <br/>
/// It then returns the transformed message. Call from a DTL passing in the target and setting the target.

ClassMethod StripMessage(pHL7 As EnsLib.HL7.Message, pInvalidChar As %String) As EnsLib.HL7.Message
{
    Set tCount = pHL7.SegCount
    While tCount > 1 {
        Set tSegment = pHL7.GetValueAt(tCount)
        Set tSegment = $ZSTRIP(tSegment,"*",pInvalidChar)
        Do pHL7.SetValueAt(tSegment,tCount)
        Set tCount = tCount-1
    }
    Quit pHL7
}

Anyway, interesting to see alterative approaches when learning so thought I'd share. Good luck!