New post

Find

Article
· Apr 10, 2023 9m read

Sending DICOM files between IRIS for Health and PACS software

Welcome community members to a new article! this time we are going to test the interoperability capabilities of IRIS for Health to work with DICOM files.

Let's go to configure a short workshop using Docker. You'll find at the end of the article the URL to access to GitHub if you want to make it run in your own computer.

Previously to any configuration we are going to explain what is DICOM:

  • DICOM is the acronym of Digital Imaging and Communication in Medicine and it's a images and medic data transmission standard. In this protocol is included the format of the DICOM file and the communication protocol based on TCP/IP.
  • DICOM files support images and clinical documentation (you can include in a DICOM file images or documents "dicomized" as images).
  • DICOM protocol define services/operations for the DICOM files. You can request the storages of an image (C-STORE), execute queries (C-FIND) o move this images among the systems of the medical organizations (C-MOVE). You can review all these available services from this URL .
  • All the systems involved in a DICOM based communication request a DICOM message as response.

You can see here a typical example of the architecture for a system designed to work with DICOM:

General scheme of DICOM Network architecture

We have some "modalities" (these modalities could be machines as scanners, MRIs or just the software that will store it) identified by the AE Title o AET (Application Entity Title). This AET will be unique for each modality and must be configured in those other modalities or systems that are going to communicate with it, in such a way that communication between both modalities is allowed.

As you can see in the graph, the modalities are configured to store their images in a DICOM file server that may or may not belong to a PACS (Picture Archiving and Communication System) that is later consulted from a PACS web interface. It is increasingly common to include a VNA (Vendor Neutral Archive) system in organizations that is responsible for centralized storage and viewing of all DICOM files used by the organization.

In general, in the most modern modalities, the destination of the generated images can be configured, but on many occasions it may be necessary or to carry out some type of action on the DICOM image fields (modify the patient identifier, include the clinical episode to the one with which it is related, etc) or, due to the inability of the modality, to take charge of capturing and forwarding the generated image to the system responsible for archiving. It is in these cases that the existence of an integration engine that provides us with such functionality is necessary, and there is none better than IRIS for Health!

For our example we will consider the following scenario:

  • A certain modality is generating images that need to be sent to a PACS for registration.
  • Our DICOM or PACS server will receive these images and must forward them to a specific VNA.


To simulate our PACS we will use Orthanc, an open source tool that will provide us with the basic functionalities for archiving and viewing DICOM images (more information here). Orthanc is kind enough to provide us with its use through an image that we can mount in Docker without any complications. Finally we will deploy an IRIS for Health container (it depends on when you read this article, the license may have expired, in that case you just have to update the docker-compose file of the code) in which we can mount our production.

Let's take a look at the docker-compose we've configured:

version: '3.1'  # Secrets are only available since this version of Docker Compose
services:
  orthanc:
    image: jodogne/orthanc-plugins:1.11.0
    command: /run/secrets/  # Path to the configuration files (stored as secrets)
    ports:
      - 4242:4242
      - 8042:8042
    secrets:
      - orthanc.json
    environment:
      - ORTHANC_NAME=orthanc
    volumes:
      - /tmp/orthanc-db/:/var/lib/orthanc/db/
    hostname: orthanc
  iris:
    container_name: iris
    build:
      context: .
      dockerfile: iris/Dockerfile
    ports:
    - "52773:52773"
    - "2010:2010"
    - "23:2323"
    - "1972:1972"
    volumes:
    - ./shared:/shared
    command:
      --check-caps false
    hostname: iris
secrets:
  orthanc.json:
    file: orthanc.json

Access to the Orthanc web viewer will be done through port 8042 (http://localhost:8042), the IP destined to receive images via TCP/IP will be 4242 and its configuration will be done from the orthanc.json file. The management portal of our IRIS for Health will be 52773.

Let's see what orthanc.json contains:

{
    "Name" : "${ORTHANC_NAME} in Docker Compose",
    "RemoteAccessAllowed" : true,
    "AuthenticationEnabled": true,
    "RegisteredUsers": {
        "demo": "demo-pwd"
    },
    "DicomAssociationCloseDelay": 0,
    "DicomModalities" : {
        "iris" : [ "IRIS", "host.docker.internal", 2010 ]
      }
}

 

As you can see we have defined a demo user with a password demo-pwd and we have declared a mode called IRIS that will use port 2010 to receive images from Orthanc, "host.docker.internal" is the mask used by Docker to access other deployed containers.

Let's check that after running the docker-compose build and docker-compose up -d we can access our IRIS for Health and Orthanc without problems:

IRIS for Health is successfully deployed.

Orthanc works too, so come on, get messy!

Let's access the namespace called DICOM and open its production. We can see in it the following business components:

We are going to review just the necessary components to manage the first case that we have presented for now. A modality that generates DICOM images but from which we cannot send them to our PACS. To do this we will use a Business Service of the standard class EnsLib.DICOM.Service.File configured to read all the .dcm files stored in the /shared/durable/in/ directory and send them to the Business Process of the Workshop.DICOM.Production.StorageFile class.

Let's take a closer look at the main method of this Business Process:

/// Messages received here are instances of EnsLib.DICOM.Document sent to this
/// process by the service or operation config items. In this demo, the process is ever
/// in one of two states, the Operation is connected or not.
Method OnMessage(pSourceConfigName As %String, pInput As %Library.Persistent) As %Status
{
    #dim tSC As %Status = $$$OK
    #dim tMsgType As %String
    do {
        
        If pInput.%Extends("Ens.AlarmResponse") {
            
            #; We are retrying, simulate 1st call
            #; Make sure we have a document
            Set pInput=..DocumentFromService
            $$$ASSERT(..CurrentState="OperationNotConnected")
        }
            
        #; If its a document sent from the service
        If pSourceConfigName'=..OperationDuplexName {
            
            #; If the operation has not been connected yet
            If ..CurrentState="OperationNotConnected" {
                
                #; We need to establish a connection to the operation,
                #; Keep hold of the incoming document
                Set ..DocumentFromService=pInput
                
                #; We will be called back at OnAssociationEstablished()
                Set tSC=..EstablishAssociation(..OperationDuplexName)
                
            } elseif ..CurrentState="OperationConnected" {
                
                #; The Operation is connected
                #; Get the CommandField, it contains the type of request, it should ALWAYS be present
                Set tMsgType=$$$MsgTyp2Str(pInput.GetValueAt("CommandSet.CommandField",,.tSC))
                If $$$ISERR(tSC) Quit
                #; We are only handling storage requests at present
                $$$ASSERT(tMsgType="C-STORE-RQ")
        		
        		// set patientId = pInput.GetValueAt("DataSet.PatientID",,.tSC)
        		// Set ^PatientImageReceived(patientId) = pInput.GetValueAt("DataSet.PatientName",,.tSC)
                #; We can forward the document to the operation
                Set tSC=..SendRequestAsync(..OperationDuplexName,pInput,0)
            }
            
        } elseif pSourceConfigName=..OperationDuplexName {
            
            #; We have received a document from the operation
            Set tMsgType=$$$MsgTyp2Str(pInput.GetValueAt("CommandSet.CommandField",,.tSC))
            If $$$ISERR(tSC) Quit
            #; Should only EVER get a C-STORE-RSP
            $$$ASSERT(tMsgType="C-STORE-RSP")

            #; Now close the Association with the operation, we will be called back at
            #; OnAssociationReleased()
            Set tSC=..ReleaseAssociation(..OperationDuplexName)
            
            #; Finished with this document
            Set ..DocumentFromService="",..OriginatingMessageID=""
        }
    } while (0)
    
    Quit tSC
}

As we can see, this class is configured to check the origin of the DICOM file, if it does not come from the Business Operation defined in the OperationDuplexName parameter, it will mean that we must forward it to the PACS and therefore the metadata of the DICOM message located in the CommandSet section under the name CommandField shall be of type C-STORE-RQ (store request) prior to connection establishment. In this URL you can check the different values ​​that this metadata can take (in hexadecimal).

In the case that the message comes from the indicated Business Operation, it is a sign that it corresponds to a DICOM response message to our previously sent DICOM, therefore it is validating that the CommandField of said message is of type C-STORE-RSP.

Let's analyze a little more in detail the key configuration of the Business Operation EnsLib.DICOM.Operation.TCP used to send our DICOM to our PACS via TCP/IP:

We have declared as IP the name of the hostname specified in the docker-compose in which Orthanc is deployed, as well as the port.

We have configured two key elements for sending to PACS: the AET of our IRIS for Health (IRIS) and the AET of our PACS (ORTHANC). Without this configuration, no image sending is possible, as both IRIS and ORTHANC will validate that the sending/receiving modality has permission to do so.

Where do we configure which modalities can send images from IRIS and which modalities can send images to us? It's very simple: we have access to the DICOM configuration functionality from the IRIS management portal:

From this menu we can not only indicate which modalities can send us and to which we can send DICOM images, we can also indicate what type of images we will be able to send and receive, in such a way that we can reject any image that falls outside of this parameterization. As you can see in the image above we have configured connections both from IRIS to Orthanc and from Orthanc to IRIS. By default Orthanc supports any type of image, so we don't need to modify anything in its configuration.

In order not to have problems with the images that we can send and receive from IRIS, we will configure the "Presentation Context" call, made up of "Abstract Syntax" made up of the combination of DICOM services (Store, Get, Find...) and an object (MR images , CT, etc...) and the "Transfer Syntax" that defines how information is exchanged and how data is represented.

Well, we already have configured any possible connection between IRIS and Orthanc and vice versa. Let's proceed to launch a test including a DICOM file in the path defined in our Business Service:


Very good! Here we have registered our DICOM files and we can see how they have gone through our production until they are sent to Orthanc. Let's go into more detail by checking out a message.

Here we have our message with its CommandField set to 1, corresponding to C-STORE-RQ, now let's review the response we received from Orthanc:

We can see that the value of CommandFile 32769 corresponds in hexadecimal to 8001, which, as we have seen in this URL, is equivalent to type C-STORE-RSP. We can also see that the response message is a DICOM message that only contains the values ​​defined in the Command Set.

Let's check from Orthanc that we have received the messages correctly:

Here are our messages successfully archived in our PACS. Goal achieved! We can now store the DICOM images of our modality in our PACS without any problem.

In the next article we will deal with the opposite direction of communication, sending from the PACS to our modality configured in IRIS.

Here you have available the code used for this article: https://github.com/intersystems-ib/workshop-dicom-orthanc

Discussion (0)1
Log in or sign up to continue
Article
· Apr 10, 2023 4m read

テーブル定義のデータが格納されるグローバル変数名について

これは InterSystems FAQ サイトの記事です。

バージョン2017.2以降から、CREATE TABLE文で作成したテーブル定義のデータを格納するグローバル変数の命名ルールが変わり ^EPgS.D8T6.1 のようなハッシュ化したグローバル変数名が設定されます。(この変更はパフォーマンス向上のために追加されました。)

※ バージョン2017.1以前については、永続クラス定義のルールと同一です。詳細は関連記事「永続クラス定義のデータが格納されるグローバル変数名について」をご参照ください。

以下のテーブル定義を作成すると、同名の永続クラス定義が作成されます。

CREATE TABLE Test.Product(
    ProductID VARCHAR(10) PRIMARY KEY,
    ProductName VARCHAR(50),
    Price INTEGER
)

 永続クラス:Test.Productの定義は以下の通りです。(パラメータ:USEEXTENTSETに1が設定されます) 

Class Test.Product Extends %Persistent [ ClassType = persistent, DdlAllowed, Final, Owner = {SuperUser}, ProcedureBlock, SqlRowIdPrivate, SqlTableName = Product ]
{
Property ProductID As %Library.String(MAXLEN = 10) [ SqlColumnNumber = 2 ];

Property ProductName As %Library.String(MAXLEN = 50) [ SqlColumnNumber = 3 ];

Property Price As %Library.Integer(MAXVAL = 2147483647, MINVAL = -2147483648) [ SqlColumnNumber = 4 ];

Parameter USEEXTENTSET = 1;

/// Bitmap Extent Index auto-generated by DDL CREATE TABLE statement.  Do not edit the SqlName of this index.
Index DDLBEIndex [ Extent, SqlName = "%%DDLBEIndex", Type = bitmap ];

/// DDL Primary Key Specification
Index PRODUCTPKEY1 On ProductID [ PrimaryKey, SqlName = PRODUCT_PKEY1, Type = index, Unique ];

Storage Default
{
<Data name="ProductDefaultData">
<Value name="1">
<Value>ProductID</Value>
</Value>
<Value name="2">
<Value>ProductName</Value>
</Value>
<Value name="3">
<Value>Price</Value>
</Value>
</Data>
<DataLocation>^CCar.Wt3i.1</DataLocation>
<DefaultData>ProductDefaultData</DefaultData>
<ExtentLocation>^CCar.Wt3i</ExtentLocation>
<ExtentSize>0</ExtentSize>
<IdFunction>sequence</IdFunction>
<IdLocation>^CCar.Wt3i.1</IdLocation>
<Index name="DDLBEIndex">
<Location>^CCar.Wt3i.2</Location>
</Index>
<Index name="IDKEY">
<Location>^CCar.Wt3i.1</Location>
</Index>
<Index name="PRODUCTPKEY1">
<Location>^CCar.Wt3i.3</Location>
</Index>

<IndexLocation>^CCar.Wt3i.I</IndexLocation>
<StreamLocation>^CCar.Wt3i.S</StreamLocation>

<Type>%Storage.Persistent</Type>
}

}

 

ExtentLocation:このクラスのグローバル名の生成に使用されるハッシュ値

DataLocation:レコードデータが登録されるグローバル変数名です。

Location:各インデックス固有のグローバル変数名が指定されます。

IndexLocation:この定義では、多くの場合使用されません。

StreamLocation:ストリームプロパティのデータが格納される変数です。

 

ストレージ定義に表示される情報について詳細は、ドキュメント「グローバル命名方法:USEEXTENTSET=1 の場合」をご参照ください。

2023.1以前のドキュメントは「ハッシュ化したグローバル名」をご参照ください。

2017.1以前と同様の命名ルール(^スキーマ名.テーブル名D、I、S のグローバル変数名)を使用する場合は、CREATE TABLE文実行時に以下のクラスパラメータを指定します。

WITH %CLASSPARAMETER USEEXTENTSET = 0

CREATE TABLE Test2.Product(
    ProductID VARCHAR(10) PRIMARY KEY,
    ProductName VARCHAR(50),
    Price INTEGER
)
WITH %CLASSPARAMETER USEEXTENTSET = 0

永続クラス定義:Test2.Productのパラメータ:USEEXTENTSETは以下のように定義されます。

Parameter USEEXTENTSET = 0;

 

永続クラス定義:Test2.Productのストレージ定義は以下の通りです。

Storage Default
{
<Data name="ProductDefaultData">
<Value name="1">
<Value>ProductID</Value>
</Value>
<Value name="2">
<Value>ProductName</Value>
</Value>
<Value name="3">
<Value>Price</Value>
</Value>
</Data>
<DataLocation>^Test2.ProductD</DataLocation>
<DefaultData>ProductDefaultData</DefaultData>
<IdFunction>sequence</IdFunction>
<IdLocation>^Test2.ProductD</IdLocation>
<IndexLocation>^Test2.ProductI</IndexLocation>
<StreamLocation>^Test2.ProductS</StreamLocation>

<Type>%Storage.Persistent</Type>
}

 

WITHで指定したテーブルのオプションについては詳しくは、ドキュメント「テーブルのオプション」をご参照ください。


関連記事:永続クラス定義のデータが格納されるグローバル変数名について

Discussion (0)1
Log in or sign up to continue
Question
· Apr 5, 2023

How to know which users did patient searches on clinical viewer

Hi community,

I would like to know how to know which users did patient searches on clinical viewer.

I tried checking the audit database but I cannot seem to know if it's logged there and under which event.

 

Please advise how to go about this.

 

Regards,

Ephraim Malane

2 Comments
Discussion (2)1
Log in or sign up to continue
Article
· Mar 31, 2023 4m read

Web Gateway系列(5)配置Web Gateway使用SSL/TLS连接到IRIS

《WebGateway系列(4): 配置HTTPS访问IRIS的Web服务》中介绍了在Web服务器中配置SSL/TLS以实现从客户端浏览器到Web服务器之间的安全连接,从Web服务器到IRIS之间是否也可以通过配置SSL/TLS建立起安全连接呢?尤其是在Web服务器与IRIS没有安装在同一台Server上的情况下,这段连接的安全性也是需要考虑的。答案是肯定的,接下来我们就来介绍下配置Web Gateway使用SSL/TLS连接到IRIS的基本步骤。

1.首先,我们先准备一下所需要的证书。通讯的双方为Web Gateway 和 IRIS Super Server, 双方都需要准备好各自的证书和key。IRIS自带的Public Key Infrastructure(PKI)功能内置了OpenSSL,可以用来生成服务器端及客户端的证书和key。在使用此功能时,IRIS可以同时作为CA Server和CA Client,作为CA Server时可以生成自签名的证书,可以批准CA Client的证书申请并将证书下发给CA Client。

1)配置本地证书颁发机构服务器,生成sever端的证书和key。

 

 

2)配置本地证书颁发机构客户端,如下

 

3)将证书签名请求提交到证书颁发机构服务器

 

4)进程未决证书签名请求

 

发放证书。

 

至此,Client以及Server端证书和key都已准备完成。

5) 从证书发放机构服务器获取证书。

以下证书和key将用于Web Gateway,

/intersystems/irishealth/mgr/cac.cer

/intersystems/irishealth/mgr/cac.key

以下证书和key将用于IRIS Super Server

/intersystems/irishealth/dev/CAcerts/cas.cer

/intersystems/irishealth/dev/CAcerts/cas.key

当然也可以去申请商用的证书,也可以不使用PKI直接使用OpenSSL自己去生成。

2. 配置SSL/TLS, 名为%SuperServer

 

3. Enable Superserver SSL/TLS Support,

 

4. 在Web Server中配置Web Gateway。

本示例中所使用的Web Server为Apache2.4.6,操作系统为Redhat7.9。注意以下Web Gateway管理页面打开端口为默认的80。另外,上面生成好的证书拷贝至了/tmp/下。

 

对应的CSP.ini中内容如下,

[IRISHEALTH]

Ip_Address=localhost

TCP_Port=1972

Username=CSPSystem

Password=]]]c3lz

Minimum_Server_Connections=3

Maximum_Session_Connections=6

Connection_Security_Level=10

Product=2

SSLCC_Protocol=8@

SSLCC_Key_Type=2

SSLCC_Cipher_Suites=ALL:!aNULL:!eNULL:!EXP:!SSLv2

Env_Parameters=EV6

SSLCC_Certificate_File=/tmp/cac.cer

SSLCC_Certificate_Key_File=/tmp/cac.key

SSLCC_CA_Certificate_File=/tmp/cas.cer

SSLCC_Private_Key_Password=]]]MTIz

SSLCC_Verify_Peer=1

 

5. Web Gateway中Test Connection, 报错Server Unavailable。

CSP.log报错如下,

>>> Time: Wed Mar 29 12:00:36 2023; RT Build: 2101.1776 (linux/apapi:srv=2.4.6/apr=1.4.8/apu=1.5.2/mpm=prefork); Log-Level: -1; Gateway-PID: 4315; Gateway-TID: 139634317305600; Connection-No: 0; Server: IRISHEALTH; InterSystems IRIS-PID: 0; Request-ID: 249b
IRISCONNECT : Error Connecting to InterSystems IRIS : Unable to read SSL configuration

Cannot read SSL certificate file: /tmp/cac.cer

>>> Time: Wed Mar 29 12:00:36 2023; RT Build: 2101.1776 (linux/apapi:srv=2.4.6/apr=1.4.8/apu=1.5.2/mpm=prefork); Log-Level: 0; Gateway-PID: 4315; Gateway-TID: 139634317305600
Information: Terminate Connection

Connection closed by Gateway: Connection-No=0:0; Server=IRISHEALTH; InterSystems IRIS PID=0; Context=205;
>>> Time: Wed Mar 29 12:00:36 2023; RT Build: 2101.1776 (linux/apapi:srv=2.4.6/apr=1.4.8/apu=1.5.2/mpm=prefork); Log-Level: 0; Gateway-PID: 4315; Gateway-TID: 139634317305600; Connection-No: ; Request-ID: 249b; Session-ID: uoyakptQAD; Remote-Addr: ; Page: GET /csp/bin/RunTime/Module.cxw
Diagnostic

Failed to connect to 'IRISHEALTH' - Reason: -201 (SSL Configuration error) (No Retry)

 

解决办法,

将证书和key文件放置在Web Gateway安装路径下之后问题解决。

SSLCC_Certificate_File=/opt/webgateway/cac.cer

SSLCC_Certificate_Key_File=/opt/webgateway/cac.key

SSLCC_CA_Certificate_File=/opt/webgateway/cas.cer

[root@BJSRHELSUP webgateway]# ll

total 16
drwxr-xr-x. 2 root root 23 Sep 17 2021 apache

drwxr-xr-x. 2 root root 4096 Sep 17 2021 bin

-rwxr-xr-x 1 root root 1233 Mar 29 14:48 cac.cer

-rwxr-xr-x 1 root root 1834 Mar 29 14:49 cac.key

-rwxr-xr-x 1 root root 1164 Mar 29 14:49 cas.cer

drwxrwxr-x. 2 apache root 56 Aug 30 2022 conf

drwxrwxr-x. 2 apache root 57 Nov 4 2021 logs

drwx------. 2 apache root 6 Mar 29 14:41 temp

drwxr-xr-x. 2 root root 20 Sep 17 2021 util

6. Web Gateway中再次Test Connection,成功。

 

参考文档,

InterSystems Public Key Infrastructure

Configuring the Web Gateway to Connect to InterSystems IRIS Using TLS

Discussion (0)1
Log in or sign up to continue
Article
· Mar 28, 2023 10m read

Perceived gaps to GPT assisted COS development automation

​Keywords: ChatGPT, COS, Lookup Table, IRIS, AI 

 

Purpose


Here is another quick note before we move on to GPT-4 assisted automation journey. Below are some "little" helps ChatGPT had already been offering,  here and there, during daily works. 

And what could be the perceived gaps, risks and traps to LLMs assisted automation, if you happen to explore this path too. I'd also love to hear anyone's use cases and experiences on this front too. 

 

Lookup tables


One of the simplest tasks could be Lookup tables.  Frequently we have to turn tedious CSV or Excel code tables into some Lookup tables in XML, manually, particularly there are blanks etc randomly distributed in the values across thousand records, and you had to tidy them quite a few times before go-live deadlines. 

Can ChatGPT help a bit? Here is a quick test.

Prompt:

You are a program to automatically convert a CSV file into a XML lookup table file.

The input is a comma delimited CSV file with 3 data columns :
Blood Science ,XCOM,CANCELLED/REJECTED SAMPLE
Blood Science ,ADX ,ADD ON TEST COMMENT
Blood Transfusion,DARA,ALERT FOR DARA/CAMELLIA
Blood Transfusion, BT1 ,REJECT SAMPLE UNLABELLED

The output should be a XML file in the form of:
<?xml version="1.0"?>
<lookupTable>
     <entry table="HIE.ICE.FilterOBR4" key="XCOM">Blood Science</entry>
     <entry table="HIE.ICE.FilterOBR4" key="ADX ">Blood Science</entry>
     <entry table="HIE.ICE.FilterOB
R4" key="DARA">Blood Transfusion</entry>
     <entry table="HIE.ICE.FilterOBR4" key="BT1">Blood Transfusion</entry>
</lookupTable>
Note: the sequence of the XML attribute in each root should be kept as they are


ChatGPT:

Here's a Python script that reads in the CSV file, processes it, and outputs the resulting XML file:

import csv

def convert_csv_to_xml(input_file, output_file):
    """
    Convert a CSV file to an XML lookup table file.

    Parameters:
    input_file (str): Path to the input CSV file.
    output_file (str): Path to the output XML file.

    Returns:
    int: 0 if the conversion was successful, 1 if there was an error.
    """
    try:
        # Create a dictionary to hold the lookup table data
        lookup_table = {}

        # Read in the CSV file and process the data
        with open(input_file, "r") as csvfile:
            csvreader = csv.reader(csvfile)
            for row in csvreader:
                category = row[0].strip()
                key = row[1].strip()
                value = row[2].strip()
                lookup_table[key] = category

        # Generate the XML file
        with open(output_file, "w") as xmlfile:
            xmlfile.write('<?xml version="1.0"?>\n')
            xmlfile.write('<lookupTable>\n')
            for key in lookup_table:
                xmlfile.write(f'<entry table="HIE.ICE.FilterOBR4" key="{key}">{lookup_table[key]}</entry>\n')
            xmlfile.write('</lookupTable>\n')

        return 0
    except Exception as e:
        print("Error:", e)
        return 1

This script first reads in the CSV file using the csv module and creates a dictionary to hold the lookup table data. It then processes each row of the CSV file and adds the relevant data to the dictionary.

Finally, the script generates the XML file by iterating through the dictionary and writing the appropriate XML tags and attributes to the output file. Note that the f string syntax is used to dynamically insert the key and category data into the XML tags.


I ran it briefly in a Jupyter notebook, it checked out. It also stripped out the blanks (was going to be a lot manual tidy-ups). It gave a tool handling lengthy CSVs with messy vlues, without being limited to token size. 

In this specific case, I'd say it increased the productivity about 20+ times over repeated manual approaches handling many messy code tables.


Now that it save me a bit time, so I was wondering whether it can generate a COS sample too to build Util class for future service BAUs etc. 


Prompt:  

Could you convert the above Python function to intersystems COS classmethod, by using correct COS specific function please

ChatGPT:

Certainly! Here's the converted code as an InterSystems ObjectScript class method:

ClassMethod ConvertCSVToXML(inputFile As %String, outputFile As %String) As %Integer
{
    try {
        // Create a dictionary to hold the lookup table data
        set lookupTable = {}

        // Read in the CSV file and process the data
        set csvReader = ##class(%File.CharacterStream).%New()
        do csvReader.LinkToFile(inputFile)
        while (csvReader.AtEnd = 0) {
            set row = csvReader.ReadLine()
            set category = $ZSTRIP($PIECE(row, ",", 1), "B")
            set key = $ZSTRIP($PIECE(row, ",", 2), "B")
            set value = $ZSTRIP($PIECE(row, ",", 3), "B")
            set lookupTable(key) = category
        }
        do csvReader.%Close()

        // Generate the XML file
        set xmlWriter = ##class(%File.CharacterStream).%New()
        do xmlWriter.LinkToFile(outputFile)
        do xmlWriter.WriteLine("<?xml version=""1.0""?>")
        do xmlWriter.WriteLine("<lookupTable>")
        set keys = lookupTable.%GetKeys()
        while ('keys.%AtEnd) {
            set key = keys.%GetNext()
            set category = lookupTable(key)
            do xmlWriter.WriteLine("<entry table=""HIE.ICE.FilterOBR4"" key="""_key_""">"_category_"</entry>")
        }
        do xmlWriter.WriteLine("</lookupTable>")
        do xmlWriter.%Close()

        quit 0
    }
    catch (ex) {
        Write "Error: "_ex.DisplayString(), !
        quit 1
    }
}

In InterSystems ObjectScript, we can use the %File.CharacterStream class to read and write files. We use the LinkToFile() method to open the input and output files, and then use the ReadLine() and WriteLine() methods to read and write data to the files.

To strip leading and trailing spaces from each field, we use the $ZSTRIP() function with the "B" flag.

We also use the %GetKeys() method of the lookupTable object to iterate over the keys in the dictionary.

Finally, the method returns a status code of 0 if the conversion was successful, or 1 if there was an error.


It didn't give an accurate tool that can check out as straightaway as Python.  %File.CharacterStream should be  %Stream.FileCharacter,  stripping blanks should use "<>W", and looping through globals should use $ORDER etc etc. 

Prompts can be improved, but I am particularly not good at asking for favors, so I rushed up a few tweaks along the lines. 

 

ClassMethod ConvertCSVToXML2(inputFile As %String, outputFile As %String) As %Integer
{
  
  s tSC = $$$OK
  try {
		set ^lookupTable = ""
        // Read in the CSV file and process the data
        set csvReader = ##class(%Stream.FileCharacter).%New()
        set tSC =  csvReader.LinkToFile(inputFile)
        while (csvReader.AtEnd = 0) {
            set row = csvReader.ReadLine()
            set category = $ZSTRIP($PIECE(row, ",", 1), "<>W")
            set key = $ZSTRIP($PIECE(row, ",", 2), "<>W")
            set value = $ZSTRIP($PIECE(row, ",", 3), "<>W")
            set ^lookupTable(key) = category
        }
        //zw ^lookupTable
        do csvReader.%Close()
        
        // Generate the XML file
        set xmlWriter = ##class(%Stream.FileCharacter).%New()
        set tSC = xmlWriter.LinkToFile(outputFile)
        set tSC = xmlWriter.WriteLine("<?xml version=""1.0""?>")
        set tSC = xmlWriter.WriteLine("<lookupTable>")
        set key = $O(^lookupTable("")) //lookupTable.%GetKeys()
        while (key '= "") {
	        //w key
            set category = $GET(^lookupTable(key))
            w !,key, " ", category
            set tSC =  xmlWriter.WriteLine("<entry table=""HIE.ICE.FilterOBR4"" key="""_key_""">"_category_"</entry>")
            set key = $O(^lookupTable(key))
        }
        set tSC = xmlWriter.WriteLine("</lookupTable>")
   		set tSC = xmlWriter.%Save("</lookupTable>")
        set tSC = xmlWriter.%Close()
  }
    catch (ex) {
        Write "Error: "_ex.DisplayString(), !
        s tSC = ex
    }
   return tSC
}

 

So what are the perceived gaps here?  

So some potential implications came into my mind would be:

1. Productivity:  The productivity reward would depend on the COS proficiency of the developer.  This tool would further amplify any advantages in coding proficiency.  

2. Gaps: My wondering would be:  how could we drive up its accuracy by few-shot learning via prompts , with or even without fine-tuning?  If you happen to explore this front across LLMs - I'd love to hear your thoughts, or dreams.  
 

 

Unit Tests Auto-generations
 

By talking about development, we can't get away from testing.  

Things seem to be changing now. Very "simple" looking, sleek tools leveraging the power of GPTs, such as RubberDuck, mushroomed out to help.

So I tried RubberDuck extension in VSCode, then set it up with my OpenAI API key.

Then I open the Python function as mentioned above in VSCode, as below:

Then I  selected click "Generate Unit Test ..." menu, and it would auto-generate some unit test coverage in a few seconds, at least 100+ times quicker than typing. It gives a quick place holder for tweaks.

 

 

Can we do such for COS code too, even though neither RubberDuck nor ChatGPT truly understood COS yet (and it's not LLM's fault):

And it generated these unit test placeholder without understanding COS much yet - I'll leave its gaps open for advice for now, and particularly, how or will the gaps need to be addressed with a sense of purpose?

It would need some tweaks, but by harvesting ChatGPT, it seems now able to auto generate codes, analyse codes, auto-comment the code, and auto-generate unit test place holders for you, in today's reality. (Well, sort of, depending on what coding language we are using, and what we really want it to do next).

 

Conclusion?


Again, I don't have quick conclusions, since I am not able to explore its boundaries and traps much yet.  

 

Math foundation??


Maybe one day, just as the concept of "entropy" in information theory was cast in 1948, another math genius might simply come out to enlighten us with another concept to simply quantify the "linguistic" gaps and distances among all sorts of languages, be it human or machine languages. Until then we would not really know the actual potential, limits. risks and traps of these LLMs. Nonetheless it doesn't seem to stop LLMs leap forward on monthly or weekly basis. 

 

Other use cases??


Analytics dashboards driven by human natural language queries:  A few weeks ago I was attempting some R&D for an innovation project, and by chance I noticed another use case:   it "can" turn clinical queries in human language into a machine query in SQL/MDX.  Well, kind of, without any fine-tuning yet by today. It seems to start to bridge the abyss between human languages and machine tones, to a degree?   

It would not be difficult to imagine such a scenario:  that a clinician just types in a clinical query into a chat in her natural language, and automatically a clinical dashboard is generated, highlighting those patients who might have missed the early detection of heart failure, and grouping and ranking them across regions, gender, and age groups. The differences in care quality would be highlighted in minutes instead of months of engineering efforts. 

 

And certainly, personalised AI care assistant.  It seemed to be so remote and so complex last year this time, and with GPT4, it rapidly becomes a reality, it seems.  Theoretically, nothing would stop GPT4 alike LLMs looking into my care records, just bits and pieces of structured and semi-structure data (like Lab tests and my medicines), unstructured data (my clinical notes and reports etc) and image data (X-Rays, CTs and MRIs), to start to make the best sense of it, and able to coordinator my cares and appointments in the coming soon.  


Disclaimer 
 

Sorry to state the obvious, but please note that the purpose of this post is not about how can we do XML Lookup tables quickly, manually or not, and actually I am pretty good at Excels and Notepad++ too. Instead, it's meant to get into touch with minds who'd like to share use cases, reflections, challenges, risks and experiences on the journey to LLM etc assisted automations

And LLMs' power came from developers, from every public repository and posting everybody made in public forums. GPTs are not born out of air. This is not being fully appreciated and acknowledged yet in recent months.

There are risks to humanity by AGI at this speed, but personally I felt slightly lucky, relieved and excused since  we are in healthcare services. 

Other routine risks include data privacy compliances per HIPPA, GDPR and DPAs as usual.  

3 Comments
Discussion (3)1
Log in or sign up to continue