· Oct 23, 2019 2m read

Unit Tests for Data Transforms

Would you like to be sure your data transforms work as expected with a single command? And what about writing unit tests for your data transforms in a quick and simple way? 

When talking about interoperability, there are usually a lot of data transforms involved. Those data transforms are used to convert data between different systems or applications in your code, so they are running a very important job.

Testing strategies

Having a look at the concept of Test Pyramid and some articles about it, we can have a quick idea that having a solid base of low level cheaper automated tests is a better idea that testing only using the UI.

In an interoperability context, I've found in several projects that is really worth investing a little effort in writing data transforms unit tests, especially when we are working with different scenarios (e.g. HL7, custom messages, etc.). This will allow us to be sure our data transform logic is running as expected after introducing new changes. Even after resolving an issue with a data transform, we can easily create a new test with the message that caused the issue, so we are sure we are not getting the same error in the future.  


A little helper that uses %UnitTest framework

To help you writing and running data transforms tests I'm sharing an example that uses IRIS %UnitTest framework. The goal is allowing you to define as external text files different sets of input messages and expected corresponding output messages for each data transform you want to test.

Let's suppose you want to create some unit tests for a data transform called Sample.Health.a31ToPatient, you should:

1. Create a new class that extends DataTransformTestCase. Set the <TestDirectory> parameter to the directory you are going to store your data transform tests.

2. In your <TestDirectory>, create a sub-directory called  Sample.Health.a31ToPatient. This sub-directory will store the sets of input and expected output you want to test in the data transform.

3. In the Sample.Health.a31ToPatient sub-directory add input messages and expected output as *.in.txt and *.out.txt.

4. Run your tests! You can see the results in the %UnitTest Framework portal in IRIS. In case you are getting an error a *.gen.txt file will be generated, so you can compare the actual output Vs. the expected output and see what's wrong.  



Run the example yourself

You can download, execute the example and read more details in Open Exchange.

Discussion (2)2
Log in or sign up to continue

Hi Alberto,

I have an alternative approach that uses Class based Unit Tests for DTL.

Essentially it uses pairs of XDATA blocks to hold the input and expected output messages.

It automatically generates the UnitTest Compare by assertion implementation with assertions, by first analysing the existing DTL path expressions in your real DTLs. code.

The advantage for this is:

1) It doesn't require external folders to store input and output messages.

2) The test input and output message are source controlled with the unit-test class.

3) Easily ignore comparison of specific paths set in DTL that you don't want to compare. For example paths set to todays date or a numeric sequence would always be different so we can easily comment out these specific generated assertions. ie: Not compare the whole input and output file.

4) There is a utility method when given a class package it will seek out and find and generate DTL unit test classes with tailored "compare assertion" methods

5) There is Assertion for paths being found in the input document. For example: If you have a recent change to the schema, and your existing Tests appear to pass. ie: A target field is empty as expected. This unit test will still pick up that the source field is in accessible using the current schema.

6) The ordering of assertions in path loops is re-ordered to be more processing efficient.

7) Its using a more "Row Driven" unit test methodology. ie: An analyst can keep just adding the pairs of XDATA input and expected output messages to the same UnitTest class without the knowledge for how to implement unit-tests. I had considered providing a Studio template to facilitate this, or to pull sample messages into a DTL class where sample messages are available on the system in focus.

Limitations: It works for HL7-to-HL7 and HL7-to Objects. It needs further work for Object-to-Object to auto-generate to compare assertion method.

Note: I also have a classes based UnitTest base for stressing routing rules in an existing production along similar lines.

Let me know if interested. Maybe can share to community.

Kind regards,