Data Transformation Testing

"objectscript.conn": {
       "links": {
         "Data Transform Test": "${serverUrl}/csp/healthshare/${namespace}/EnsPortal.Dialog.TestTransform.zen?$ZEN_POPUP=1&$ZEN_SOFTMODAL=0&TRANSFORM=${classname}"
       }
     }

 
For IRIS Data Platform:

"objectscript.conn": {
       "links": {
         "Data Transform Test": "${serverUrl}/csp/${namespace}/EnsPortal.Dialog.TestTransform.zen?$ZEN_POPUP=1&$ZEN_SOFTMODAL=0&TRANSFORM=${classname}"
       }
     }

 

The link uses the following variables exposed by the ObjectScript extension:

• ${serverUrl}: The connected server base URL.
• ${namespace}: The connected namespace.
• ${classname}: The currently open class name.

 

For more details on configuring server actions, see Configure and Perform Server Actions within VS Code | Use Visual Studio Code as a Development Environment for InterSystems Applications | InterSystems Components and Tools.

While the dialog in the portal allows you to select only DTL transformations, the test tool supports any class that extends Ens.DataTransform.

For DTL transformations, it gets the source and target classes from the DTL <transform> element's attributes. For non-DTL transformations (classes that extend  Ens.DataTransform), the tool uses the Transform() method signature.

When the source and/or target class is a virtual document class (a class that extends Ens.VDoc.Interface), a document type (DocType) is required. For DTL data transformations, the tool gets the <transform> element's attributes, SourceDocType, and TargetDocType. For non-DTL transformations, an implementation of GetSourceDocType() and/or GetTargetDocType() is a must.

Example

Below you can find a complete example of a non-DTL data transformation (extending Ens.DataTransform directly) that can be used with the test tool. The transformation clones a source HL7 v2.3 SIU message and sets the target message SCH:7.1 field component value.

Class ut.ks.lib.interop.SampleTranform Extends Ens.DataTransform
{ 
ClassMethod Transform(source As EnsLib.HL7.Message, Output target As EnsLib.HL7.Message, aux As %String) As %Status
{
    #Dim sc as %Status
    #Dim ex as %Exception.AbstractException
    s sc = $$$OK
    try {
        s target = source.%ConstructClone(1)        
        s target.DocType = ..GetTargetDocType()
        d target.SetValueAt("hello","SCH:7.1")   
    } catch (ex) {
      s sc = ex.AsStatus()
    }
    return sc
}
 
ClassMethod GetSourceDocType() As %String
{
  return "2.3:SIU_S12"
}
 
ClassMethod GetTargetDocType() As %String
{
  return "2.3:SIU_S12"
}
 
}

 Take a look at a little screencast of this code in action below:

 

Part 2: Using the Test Tool with the new user interface

Starting with InterSystems IRIS version 2025.1, the DTL editor and test tool got a new, alternative user interface accessible from the Management Portal or VSCode IDE.

To access the new UI from the Management Portal, go to Interoperability > Build > Data Transformation and open a transformation using the ‘open’ button. Remember that the dialog will let you select only DTL transformations. To switch to the new user interface, click the “new UI” button. 
The new user interface can also be accessed as an editor directly in VS Code.

Unlike the previous version, the new test tool supports only DTL data transformations. To use it with a class that extends Ens.DataTransform, you must wrap the call to the Transform() class method in a code action. Review the following example:

Class ut.ks.lib.interop.SampleTransformDTL Extends Ens.DataTransformDTL [ DependsOn = EnsLib.HL7.Message ]
{
 
Parameter IGNOREMISSINGSOURCE = 0;
Parameter REPORTERRORS = 0;
Parameter TREATEMPTYREPEATINGFIELDASNULL = 0;
Parameter GENERATEEMPTYSEGMENTS = 0;
XData DTL [ XMLNamespace = "http://www.intersystems.com/dtl" ]
{
<transform sourceClass='EnsLib.HL7.Message' targetClass='EnsLib.HL7.Message' sourceDocType='2.3:SIU_S12' targetDocType='2.3:SIU_S12' create='new' language='objectscript' >
<code>
<![CDATA[
    return ##class(ut.ks.lib.interop.SampleTranform).Transform(source,.target,.aux)
  ]]></code>
</transform>
}
}

 

Part 3: Programmatic testing

Using the %UnitTest framework, it is easy to write repeatable and robust programmatic tests for data transformations. See @Yuri Marx excellent article on the topic and the official documentation for more.

It is generally a good practice for transformations to depend as little as possible on runtime context data other than the source and auxiliary objects.

Generally speaking, operating anything other than the source, target, auxiliary objects, or the current namespace content should be avoided. Below you can check some examples of what should be bypassed:

• Depending on interoperability production (e.g., sending a request to a business operation).
• Having a stateful Transform() class method.
• Performing a file or external database input/output.

If additional runtime data is needed, the transformation class should provide default values to the maximum extent possible. To achieve that, add such members as class parameters or XData blocks to the transformation class.

Parameters can hold any string value (even multi-line), and when declared as a configuration value, can be changed in a given namespace after class compilation (a nice, yet underrated configuration mechanism).

XData blocks are more suitable for small XML documents, JSON objects or arrays, or text-structured data. Remember to takegeneral system limits into account, though.

Larger data may be made available to the transformation using the following:

• Lookup tables (check out Ens.Util.LookupTable).
• %Persistent objects.
• Globals and other environment-dependent data.

The Auxiliary parameter “aux” is passed by the caller of the Transform() class method. It can be a datatype value or an OREF. When the transformation is used in the ‘send’ action of a message router rule, the router passes itself as an aux parameter, and its properties (e.g., RuleActionUserData) become available to the transformation.

If the transformation class cannot provide the defaults, the test should deliver them and initialize any required state before running the Transform() method. If the transformation is, for instance, using a custom HL7 schema, it should be loaded before the test run. The test should, as an example, do the following:

• Import custom HL7 schemas, using EnsLib.HL7.SchemaXML.Import() class method.
• Import lookup tables, with the help of Ens.Util.LookupTable.%Import() class method.
• Initialize any other data, e.g., %Persistent objects or globals employed by the data transformation class.

A unit test for a transformation begins by initializing context data (e.g., import lookup tables). Then, for each test source object, it calls the Transform() method and compares the resulting target object coming from the transformation to the expected outcome.

Anticipated results can be stored as files loaded by the test at runtime or XData blocks. With many inputs to test (e.g., when testing multiple complete HL7 v2.x ADT scenarios), I find files to be more practical.

Example

Test Class

This test uses the ks-iris-lib module from Open Exchange to import HL7 messages from XData blocks and compare the target message from the transformation to the expected result:

Class dc.SampleTransformTest Extends ks.lib.test.TestCase
{
 
Method TestTransform()
{
    #Dim sc as %Status
    #Dim ex as %Exception.AbstractException
    #Dim source,target,expected As EnsLib.HL7.Message
   
    s sc = $$$OK
    try {
        set source = ##class(ks.lib.hl7.Utils).ImportFromXData($classname(),"SourceHL7","2.3",.sc)
        $$$TOE(sc,sc)
        set expected = ##class(ks.lib.hl7.Utils).ImportFromXData($classname(),"ExpectedHL7","2.3",.sc)
        $$$TOE(sc,sc)
        $$$TOE(sc,##class(NullTranform).Transform(source,.target,$this))
        do $$$AssertTrue(..CompareHL7Messages(target,expected))
    } catch (ex) {
      s sc = ex.AsStatus()
    }
    do $$$AssertStatusOK(sc)
}
 
XData SourceHL7 [ MimeType = application/hl7 ]
{
MSH|^~\&|SPOCARD|EWHIN|JONES|EWHIN|199401040800||SIU^S13|021244SPOCARD|P|2.3|||AL|ER||
SCH|1994047^SCH001|1994567^SCH100|||||047^Referral|NORMAL|30|min|^^^199401091300^199401091330^^^^|0045^Jones^Harold^S^^^MD|555-4685|||087^Jensen^Helen^M^^^MD|555-9255||||BOOKED
NTE||The patient is going to be on vacation so cannot make previous appointmentscheduled on January 6.
PID||4875439|484848||Peterson^Joseph^^Jerome^SR|Brown|19401121|M|Jayjay||N 1234 Newport Highway^Mead^WA^99021||555-4685|||M|||999-99-4413|||||||||||
RGS|001|
AIP|001|032^JENSEN^HELEN|002^CARDIOLOGIST|||||||NO|BOOKED
}
 
XData ExpectedHL7 [ MimeType = application/hl7 ]
{
MSH|^~\&|SPOCARD|EWHIN|JONES|EWHIN|199401040800||SIU^S13|021244SPOCARD|P|2.3|||AL|ER||
SCH|1994047^SCH001|1994567^SCH100|||||047^Referral|NORMAL|30|min|^^^199401091300^199401091330^^^^|0045^Jones^Harold^S^^^MD|555-4685|||087^Jensen^Helen^M^^^MD|555-9255||||BOOKED
NTE||The patient is going to be on vacation so cannot make previous appointmentscheduled on January 6.
PID||4875439|484848||Peterson^Joseph^^Jerome^SR|Brown|19401121|M|Jayjay||N 1234 Newport Highway^Mead^WA^99021||555-4685|||M|||999-99-4413|||||||||||
RGS|001|
AIP|001|032^JENSEN^HELEN|002^CARDIOLOGIST|||||||NO|BOOKED
}
 
}
 

Transform Class

Class dc.NullTransform Extends Ens.DataTransform
{
 
ClassMethod Transform(source As EnsLib.HL7.Message, Output target As EnsLib.HL7.Message, aux As %String) As %Status
{
    #Dim sc as %Status
    #Dim ex as %Exception.AbstractException
    s sc = $$$OK
    try {
        s target = source.%ConstructClone(1)        
        s target.DocType = ..GetTargetDocType()
    } catch (ex) {
      s sc = ex.AsStatus()
    }
    return sc
}
 
ClassMethod GetSourceDocType() As %String
{
  return "2.3:SIU_S12"
}
 
ClassMethod GetTargetDocType() As %String
{
  return "2.3:SIU_S12"
}
 
}