Wow, once again, you make my day :)

With love from Paris :)

I love it — thanks a lot, @Muhammad Waseem! It’s so much more comprehensive when you can see a tutorial or presentation of the topic from another point of view.

Once again, you make a breakthrough.

Bravo !

+1 and for community web site too :)

Thanks again for this amazing contribution, once again you make a break through, i guess this is as much important as SQLAlchemy was.

What a deep article, thanks.

Special kudo python examples :).

Hi, have you tried this : https://openexchange.intersystems.com/package/iris-readonly-interop

Official, IRIS support only one version of python and one set of library (default in <installdir/mgr/python).

But you can find here a PoC of venv support in our interoperabilty framework https://grongierisc.github.io/interoperability-embedded-python/venv/.

The good new is that each componement of the interoperabilty has it's own process, it means it can run different version of python.

Remember it's a PoC and it's community based.

Give it a try, and let me know what you think of it.

I mostly use Embedded Python but without the language tag : [ language = python ] in objectscript classes, in my opinion it doesn't make senses to mix two languages in one file.

Why Embedded Python mostly because it's faster and it's in the DNA of intersystems compute close to the data.

Now, time to time, i also use DB-API with sqlalchmy, to manipulate SQL data, thanks to the community edition of DB-API for IRIS, i can chose if i want to use it remotly or with Embedded Python.

At the end of the day, it should not have any difference between embedded python or "native API".

What a terrific article! Thanks.

I hope will see the object %Exception.PythonException:PyExc in the next release of InterSystems IRIS. It will make debugging embedded Python so much easier.

Meanwhile, I implemented a workaround in IoP, it now supports tracebacks and remote debugging. You can find it here:

• IoP

And tracebacks here:

• IoP Tracebacks

Hi, you are on the right tracks.

Here how I would do it:

Input file :

{"id": 1, "name": "Alice", "city": "New York", "age": 28}
{"id": 2, "name": "Bob", "city": "Chicago", "age": 35}
{"id": 1, "name": "Alice", "city": "New York", "age": 28}
{"id": 3, "name": "Charlie", "city": "Boston", "age": 42}
{"id": 4, "name": "David", "city": "Seattle", "age": 31}
{"id": 2, "name": "Bob", "city": "Chicago", "age": 35}
{"id": 5, "name": "Eve", "city": "Miami", "age": 29}
{"id": 3, "name": "Charlie", "city": "Boston", "age": 42}
{"id": 6, "name": "Frank", "city": "Denver", "age": 38}
{"id": 1, "name": "Alice", "city": "New York", "age": 28}

The python code:

import pandas as pd
import iris

def string_to_stream(string:str,buffer=1000000):
    stream = iris.cls('%Stream.GlobalCharacter')._New()
    n = buffer
    chunks = [string[i:i+n] for i in range(0, len(string), n)]
    for chunk in chunks:
        stream.Write(chunk)
    return stream

def stream_to_string(stream,buffer=1000000)-> str:
    string = ""
    stream.Rewind()
    while not stream.AtEnd:
        string += stream.Read(buffer)
    return string

def sort_remove_count(file_name: str):   
    # read the ndjson file
    data = pd.read_json(file_name, lines=True)

    # sort the data by id
    data = data.sort_values(by='id')

    # remove the duplicates based on name, city and age
    data = data.drop_duplicates(subset=['name', 'city', 'age'])

    # count the number of unique ids
    unique_ids = data['id'].nunique()

    print(unique_ids)

    # save the data to a new ndjson string (not a file)
    buffer = data.to_json(orient='records', lines=True)

    # convert it as an iris stream
    stream = string_to_stream(buffer)
    return stream

if __name__ == '__main__':
    stream = sort_remove_count('demo/input.ndjson')
    print(stream_to_string(stream))

Result :

6
{"id":1,"name":"Alice","city":"New York","age":28}
{"id":2,"name":"Bob","city":"Chicago","age":35}
{"id":3,"name":"Charlie","city":"Boston","age":42}
{"id":4,"name":"David","city":"Seattle","age":31}
{"id":5,"name":"Eve","city":"Miami","age":29}
{"id":6,"name":"Frank","city":"Denver","age":38}

Now to industrialize this code, you can use IoP :

from iop import BusinessOperation
import pandas as pd
import iris

class SortRemoveCount(BusinessOperation):
    def string_to_stream(self, string:str,buffer=1000000):
        stream = iris.cls('%Stream.GlobalCharacter')._New()
        n = buffer
        chunks = [string[i:i+n] for i in range(0, len(string), n)]
        for chunk in chunks:
            stream.Write(chunk)
        return stream

    def stream_to_string(self, stream,buffer=1000000)-> str:
        string = ""
        stream.Rewind()
        while not stream.AtEnd:
            string += stream.Read(buffer)
        return string

    def sort_remove_count(self, file_name: str):   
        # read the ndjson file
        data = pd.read_json(file_name, lines=True)

        # sort the data by id
        data = data.sort_values(by='id')

        # remove the duplicates based on name, city and age
        data = data.drop_duplicates(subset=['name', 'city', 'age'])

        # count the number of unique ids
        unique_ids = data['id'].nunique()

        print(unique_ids)

        # save the data to a new ndjson string (not a file)
        buffer = data.to_json(orient='records', lines=True)

        # convert it as an iris stream
        stream = self.string_to_stream(buffer)
        return stream

    def iris_message(self, request: 'iris.Ens.StringRequest') -> 'iris.Ens.StreamContainer':
        stream = self.sort_remove_count(request.StringValue)
        return iris.cls('Ens.StreamContainer')._New(stream)

Hope this helps.

From your screenshot, it seems that you have properly install flask module and it's working.

Now you have to build an flask app.

Maybe a good starting point is here : https://github.com/grongierisc/iris-flask-template

Thanks for this valuable feedback.

Few years ago (2020 ish), i had to do a project based on DocDB. We encontered the same issues :

• API first, not code first
  • Workaround : We relied heavily on scripts to generate the "Databases", "Properties", "Indexes" and so.
• When you create a property it's not automatically indexed
  • Workaround : We created a wrapper around the SDK to ensure that every property was indexed
• No way to enforce a schema
  • Workaround : No workaround, we didn't really care about that at the time

What you didn't mention and that we encountered :

• Composite indexes are not supported
  • Workaround : We solved this with the "wrapper" we created
• No support of neasted objects
  • Workaround : We didn't solve this, we had to flatten all the objects :(
• Some operators was not supported or not working as expected
  • Workaround : We created some wrc tickets and most of them were fixed :) or built our own sql statements based on indexes

What great is that we never get blocked by those issues, we always found a workaround.

I'm glad to see that DocDB is still alive and the team is working on it.

It's a step in the right direction to support "json" databases. I can't wait to see the next steps maybe a client side library, support for nested objects, composite indexes, a great sql function to support json objects, etc.

can you give a try to https://github.com/grongierisc/iris-embedded-python-wrapper.

follow the readme, it will give you the instruction to work with venv and a chosen version of python and bind it to iris.

behind the scene, this module help to setup the PythonPath, PythonRuntimeLibray and PythonRuntimeLibrayVersion

let me know, if you find any issue.

Btw, it will not solve the python 3.13 issue, you need to upgrade to 2025.1 to support it.

Answered here : https://community.intersystems.com/post/hl7v2-fhir-its-easy#comment-279…

Thanks @Dmitry Maslennikov, once again you make things clear. What we do today and what can be improve, i hope, other can have a look at this study and understand the value of such feedback.