Any way to add Ensemble HL7 interfaces via a script in Terminal?

You've written a pretty good description of the Production Generator included in IRIS for Health:
https://docs.intersystems.com/irisforhealthlatest/csp/docbook/DocBook.UI...

This presentation may be of interest:
https://learning.intersystems.com/course/view.php?id=1192

Hi

I have spent the last 4 months developing a generic HL7 and FHIR Codebase. The way it works is as follows:

1) I have a Codebase Namespace that consists of a number of common classes:

a) A Message Queue Service that processes messages in a Message Queue. The Message Queue class has a number of standard properties including: CreateTS, ProcessTS, CompletedTS, Target Message Type (HK7 or FHIR), Fields for the generated Request FHIR or HL7 message, Fields for the Response HL7 or FHIR message.  Fileds for the HL7 ACK/NACK Code and Text, HTTP Response Status, Overall Message Status.

b) There are a number of standard methods for every  Message Queue,: CreateMessage(), Update Message(), GetNextMessage(), CompleteMessage(), ResendMessage(), ResendMessageRange, PurgeMessages(). These methods are called by the Business Service that Processes the Message Queue, Methods that are used by the Busiess Process to Resend Messages or Complete a Message and the UpdateMessage() method is used to update the fields in the message with data it collects as the message passes through the Production. 

c) Code tables to convert some application field values into HL7 or FHIR equivalents.

d) The following Business Operations: HL7HTTPOperation, HL7HTTPSOperation, FHIRHTTPOperation, FHIRHTTPSOperation, HL7FileOperation and FHIRFileOperation 

e) A Business Service that calls the GetNextMessage() method and if it finds a message it puts the MessageId into an Ensemble Request Message and sends it to the Business Process.

f) The Business Process uses the MessageId to open the message Queue Object which apart from the standard fields mentioned in (a) contains any fields derived from the source application that identify the source data that you want to transform into an HL7 or FHIR Message. In the OnRequest() method of the Process you use the fields in the Message to retrieve the data from the data source and then invoke a DTL to transform the data into an HL7 or FHIR Message. The resultant message is written to the Message Queue object. Then depending on how the Interface is configured the Ensemble Request Message is sent on to any of the the Business Operations I listed. The Business Operations then do the work to write the HL7 or FHIR message into the HTTPRequest Message. The HTTP/S Operations then send the HTTP request to the Target Server. When a response is received the response status is noted and the message queue object updated. If there is content in the response then that too is written into the Message Queue Object and an Ensemble response message is instantiated and sent back to the Business Process.

g) The Business Process OnResponse() method processes the response it receives from the HTTP Operation and updates the Message Queue Object with the overall status of the transaction. It also calls the CompleteMessage() Method which tidies everything up and sets the CompletedTS to the current Date/Time. There are two calculated fields: TimeFromCreateToComplete and TimeFromProcessToComplete. The second one is  based on the time from when the message was picked up by the Business Service through to completion. The first time field records the time it took from when the message was originally created through to completion. 

h) There is a House Keeping Business Service that purges the Message Queue, Ensemble Messages, Ensemble Log Files ($$$TRACE etc), the Inbound and outbound file directories. The purges are based on configurable "Number of Days to Keep {messages/logs/etc...}"

i) There is a Debug Logger that you can call anywhere in your code to record some debg information. I did this because you can't use $$$TRACE or any of the $$$LOG... calls in non-Production classes. It also encourages developers to use a standard debugging approach using a standard format rather than having lots of "set ^Nigel($i(^Nigel),$zdt($h,3))="The value of variable x is "_$g(x) approach.

j) There is an Alert Notification System. I have the concept of Error Alert Notifications such as a comms failure in the HTTP Operation or an exception caught in a Try/Catch. Then there are Condition Notifications. They are intended for a different audience and an example would be when the Interface discovers that there is no entry in a code table for a specific code. An email will be sent to the appropriate person/people to update the code table. If the code isn't there a day later a follow-up email will be sent. The Alert Notification Service also checks the number of items in the Production Queues and will send emails to notify someone that the queues are either empty (that could mean that something has broken in the source database/application and it is no longer creating messages for the message queue.

k) There is a single configuration object. There is not a single hard coded value in the entire codebase. Every element of the interface gets its context from the configuration object. The configuration Object has about 40 properties and include the name of the message queue class, the Business Process Production Item Name and Business Process Class Name, the Inbound and Outbound File Directory names, the HL7 or FHIR Request and Response File names (if you are using the File Operation which is intended mainly for testing purposes but can also be used if you are using FTP to send a message onto some other system.

l) The Codebase sits in one database along with the code tables and code table mapping classes and their contents.

m) When you want to create a new Interface you create a new Namespace and map the Codebase classes into that Namespace. You also map the code tables and Code table mapping globals into the new Interface Namespace.

n) In the source application database you import one class. The Interface Mapping Class. If you have an application or ensemble interface in your source then in some appropriate class where you have the details of a Patent or other Entity that you want to send into the new interface you add two lines of code into that class that populate an array of name/value pairs that correspond to the message queue specific fields in the message queue class created for that interface. as mentioned earlier the Message Queue Class has a number of standard properties and methods. When you create a new Interface you create a new Message Queue Class and inherit the Properties and Methods into the class and then add properties for any identifiers that the interface will use to access the data that will be used in the Data Transformation to create your HL7 or FHIR message. The name/value pairs in the array you pass to the Interface Mapping class method correspond to those identifying property names.

o) The alternative is that an Interface will access the source database and walk through a given table and for every entry it finds in that table it will call the Interface Mapping class method to create a new Message Queue Object with the identifier for that item in the table you are processing. This is really useful for Bulk Loads of data from one source into a target destination.

p) There is also a testing module that will take data from any source. You can define test conditions. For example: Scramble the data in one of the source table fields, generate random data for any field, insert or remove items from any collection, remove values from properties. This module uses a combination of the Populate Utils Methods, a whole host of Patient Names, Addresses, Identifiers and you can write custom methods to manipulate the data from the data source. Tests are conducted on one or more source records which are grouped into a "Manifest". You can tell the Manifest how many records you want to process,  where to get the records from. The Manifest Records are then processed by the Test Interface which will run some SQL or object code to get the source data for each Manifest Record. It then calls the DTL you have created to convert that source data into an HL7 or FHIR message. It then runs through the test conditions and applies them to each generated message before the message is sent to the HTTP operations and optionally the File Operations. Details of the Request and Response messages are captured in the Manifest Records. You can also specify documentation on what you expect to happen when the transformed data is processed by the target application that you are testing and once completed there is a manifest report that will generate a .cav file containing the data from every record in the manifest. aving analysed this you update the manifest with the outcome and if the outcome differs from the expected outcome then you either send the whole manifest back to the developers and tell them to fix the bug or you look at why the tests produced a different outcome and whether that is due to issues in the source database or application. Tests are defined in a Tests Table and Tests are then grouped into a Test Set which is linked to a Manifest. That means that Tests can be reused for other test scenarios

Te entire collection of classes are extensively documented. I have determined that once I have monitored the first Interface that is going unto testing in the very near future and I resolve any bugs that haven't been picked up by the various unit tests I have done so far then assuming it is stable (lol I wrote it, how could it not be)  then it will take 1- days to generate any new interface. I wrote this architecture as I have 14 HL7 and FHIR Interfaces that I have to write and they are all almost identical.

The whole framework comes with a few CSP pages that I created to maintain the code tables, code table mappings and the configuration class. I am busy learning Angular at the moment and once I have that under my belt then I will create a REST API that any UI can call so that anyone who uses the Framework can build their own UI on top of it. 

If anyone is interested in this framework please contact me. If there are enough people interested then I will do the work to publish this on Open Exchange.

If anyone wants the Frame work but doesn't want to do any of the development to make it work in your environment then I will do the implementation, create the interfaces and whatever else is required on a contract basis.

There are some aspects f this framework that could make better use of some features in Ensemble such as Code Tables, Uploading CSV files, the Managed Alert Monitoring System (which I rather like but have not built it into this framework as I have my Alert Notification Module which does pretty much the same thing but does not have the 'Help Desk" feel of the Ensemble Alert Monitoring System.

If there are other adapters that are required then I can add them into the framework codebase.

Nigel