This series of articles would cover Python Gateway for InterSystems Data Platforms. Execute Python code and more from InterSystems IRIS. This project brings you the power of Python right into your InterSystems IRIS environment:

• Execute arbitrary Python code
• Seamlessly transfer data from InterSystems IRIS into Python
• Build intelligent Interoperability business processes with Python Interoperability Adapter
• Save, examine, modify and restore Python context from InterSystems IRIS

Other articles

The plan for the series so far (subject to change).

• Part I: Overview, Landscape, and Introduction
• Part II: Installation and Troubleshooting
• Part III: Basic functionality
• Part IV: Interoperability Adapter <-- you're here
• Part V: Execute function
• Part VI: Dynamic Gateway
• Part VII: Proxy Gateway
• Part VIII: Use cases and ML Toolkit

Intro

You now have tried Python Gateway from a terminal, time to start using it via Interoperability Productons.
In this article, I would cover the main Interoperability interface to Python - isc.py.ens.Operation. It allows us to:

• Execute Python code and return requested variables (string/stream)
• Save/Restore context
• Load data into Python

Generally speaking, it's an Interoperability wrapper over isc.py.Main. Interoperability adapter isc.py.ens.Operation offers the ability to interact with the Python process from Interoperability productions. Currently, five requests are supported:

• Execute Python code via isc.py.msg.ExecutionRequest. Returns isc.py.msg.ExecutionResponse with requested variable values
• Execute Python code via isc.py.msg.StreamExecutionRequest. Returns isc.py.msg.StreamExecutionResponse with requested variable values. Same as above, but accepts and returns streams instead of strings.
• Set dataset from SQL Query with isc.py.msg.QueryRequest. Returns Ens.Response.
• Set dataset faster from Global/Class/Table with isc.py.msg.GlobalRequest/isc.py.msg.ClassRequest/isc.py.msg.TableRequest. Returns Ens.Response.
• Save the Python context via isc.py.msg.SaveRequest. Returns Ens.StringResponse with context id.

• Restore Python context via isc.py.msg.RestoreRequest.

  isc.py.ens.Operation has but two settings:

  • Initializer - select a class implementing isc.py.init.Abstract. It can be used to load functions, modules, classes and so on. It would be executed at process start.
  • PythonLib - (Linux only) if you see loading errors set it to libpython3.6m.so or even to a full path to the shared library.

Writing Business Process

There are two utility classes to ease BP development:
- isc.py.ens.ProcessUtils allows annotation fetching with variable substitution
- isc.py.util.BPEmulator allows easy testing of Python Interoperability business processes. It can execute a business process (python parts) in a current job.

Variable substitution

All business processes inheriting from isc.py.ens.ProcessUtils can use GetAnnotation(name) method to get value of activity annotation by activity name. Activity annotation can contain variables which would be calculated on ObjectScript side before being passed to Python. This is the syntax for variable substitution:

• ${class:method:arg1:...:argN} - execute method
• #{expr} - execute ObjectScript code

Check test isc.py.test.Process business process for example in Correlation Matrix: Graph activity: f.savefig(r'#{process.WorkDirectory}SHOWCASE${%PopulateUtils:Integer:1:100}.png')

In this example:
- #{process.WorkDirectory} returns WorkDirectory property of process object which is an instance of isc.py.test.Process class and current business process.
- ${%PopulateUtils:Integer:1:100} calls Integer method of %PopulateUtils class passing arguments 1 and 100, returning random integer in range 1...100.

Test Business Process

Test Interoperability Production and test Business Process are available by default as a part of the Python Gateway. To use them:

1. In OS bash execute pip install pandas matplotlib seaborn.
2. Execute: do ##class(isc.py.test.CannibalizationData).Import() to populate test data.
3. Start isc.py.test.Production production.
4. Send empty Ens.Request message to the isc.py.test.Process.

Let's see how it all works together. Open isc.py.test.Process in BPL editor (or Studio):

Code execution call

Here's the most important call - to execute Python code:

Request is isc.py.msg.ExecutionRequest and the properties are:
- Code - Python code to execute.
- SeparateLines - Separate incoming message into lines for execution. $c(10) (\n) is used for line separation. Note that it's NOT recommended to process the whole message at once, this feature is only for def and similar multi-line expressions processing. Defaults to 0.
- Variables - Comma-separated list of variables to get in response message.
- Serialization - How to serialize variables we want to return with Str, Repr, JSON, Pickle and Dill options, defaulting to Str.

In our case, we're only setting Code property so all other properties are defaults. We set it by calling process.GetAnnotation("Import pandas") which at runtime returns annotation after performing variable substitution. Eventually, import pandas as pd string would be passed to Python. GetAnnotation can be useful to set up multiline python scripts, but there's no restriction on it. You can set Code property any way you like.

Variable retrieval call

Another interesting call using isc.py.msg.ExecutionRequest is Correlation Matrix: Tabular:

It calculates Correlation Matrix on a Python side and retrieves corrmat baack into InterSystems IRIS in a JSON format, by setting request properties:
- Variables: "corrmat"
- Serialization: "JSON"

We can see results in a visual trace:

And if we need it down the line in BP it can be saved with: callresponse.Variables.GetAt("corrmat")

Data transfer call

Next, let's talk about InterSystems IRIS -> Python data transfer, all data transfer requests extend isc.py.msg.DataRequest which supplies the following common properties:
- Variable - Python variable to set.
- Type - Variable type (Currently supported: dataframe (pandas dataframe) and list.
- Namespace - Namespace in which we get the data. 'isc.py' package must be available in this namespace.

Building on that are 4 concrete classes:
- isc.py.msg.QueryRequest - set Query property to transfer SQL resultset.
- isc.py.msg.ClassRequest - set Class property to transfer class data.
- isc.py.msg.TableRequest - set Table property to transfer a whole table.
- isc.py.msg.GlobalRequest - set Global property to transfer a global.

In the test process check RAW call to see isc.py.msg.QueryRequest in action.

Save/Restore Python context calls

Finally, we can persist Python context to InterSystems IRIS, to do that send isc.py.msg.SaveRequest with:
- Mask - Only variables that satisfy Mask are saved. Wildcards * and ? are accepted. Example: "Data*,Figure?". Defaults to *.
- MaxLength - Maximum length of the saved variable. If variable serialization is longer than that, it would be ignored. Set to 0 to get them all. Defaults to $$$MaxStringLength.
- Name - Context name (optional).
- Description - Extended context info (optional).

Check Save Context call in the test process for example. Returns Ens.StringResponse with Id of a saved context.

Corresponding isc.py.msg.RestoreRequest loads context from InterSystems IRIS to Python:
- ContextId - Context identifier to restore.
- Clear - Clear context before the restore.

Summary

Python Gateway allows seamless integration between InterSystems IRIS and Python. Use it to add Python functionality to your Interoperability productions.

Links

• Python Gateway
• Python Gateway Samples
• Install Python 3.6.7 64 bit
• Python documentation/tutorial

Illustrated guide

There's also an illustrated guide in ML Toolkit user group. ML Toolkit user group is a private GitHub repository set up as part of the InterSystems corporate GitHub organization. It is addressed to the external users that are installing, learning or are already using ML Toolkit components including Python Gateway. To join ML Toolkit user group, please send a short e-mail at the following address: MLToolkit@intersystems.com and indicate in your e-mail the following details (needed for the group members to get to know and identify you during discussions):

• GitHub username
• Full Name (your first name followed by your last name in Latin script)
• Organization (you are working for, or you study at, or your home office)
• Position (your actual position in your organization, or “Student”, or “Independent”)
• Country (you are based in)