Connecting to DynamoDB Using Embedded Python: A Tutorial for Using Boto3 and ObjectScript to Write to DynamoDB

Introduction

As the health interoperability landscape expands to include data exchange across on-premise as well as hosted solutions, we are seeing an increased need to integrate with services such as cloud storage. One of the most prolifically used and well supported tools is the NoSQL database DynamoDB (Dynamo), provided by Amazon Web Services (AWS).

The challenge for IRIS implementations, as it relates to Dynamo and other modern web services, is that there is no native support for interacting with Dynamo and AWS within ObjectScript. Fortunately, we can utilize ObjectScript’s support for embedded Python to leverage AWS connectivity via Boto3 (Amazon’s Python SDK).

In this tutorial we are going to walk through the steps needed in order to put a single item into the Dynamo table.

Prerequisites and Assumptions

In order to connect to Dynamo you will need to have the following in place:

• An AWS Credentials (~/.aws/credentials) file in your .aws directory. This hidden file may be found in your home directory. (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Setting...)
• An existing table in Dynamo to which you want to connect.
• The Embedded Python package loaded. (https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls...)

This tutorial assumes these are already in place and focuses on the actual connection to DynamoDB and writing to the table using ObjectScript and Embedded Python.

Use Case

Given a Dynamo table, “Employees”, we have the task of adding a single new employee to the table. We are going to pass in their “First_Name”, “Last_Name”, and “Employee_Number”. As we are new to the technology we are going to hard code the values, proving a successful connection and insertion into the table using Boto3.

An Overview

Once credentials have been provided the steps to connect to DynamoDB are as follows:

1. Create a calling ClassMethod in ObjectScript
2. Create a Method that will hold the connection logic using Embedded Python
3. Import boto3 and Create the client
4. Call the put_item method
5. Optional: Return a %Status

There are few other hurdles that you may encounter and the end of this article will also address some of those encountered by the author.

Connecting to DynamoDB

Step 1: Create a calling ClassMethod in ObjectScript

Begin by creating a ClassMethod in ObjectScript. Since we are going to hard code the item being put into the table this is all that is needed.

The purpose of this method is to wrap the Python Method so that the ClassMethod may be called by another upstream operation.

ClassMethod writeToDynamoDB(tableName As %String)

For this use case a TableName is being passed in as a %String. The return value of %Status is needed only when the upstream operation needs a success/failure response.

Step 2: Create a Method that will hold the connection logic using Embedded Python

This Python Method will hold the actual logic for connecting to Dynamo. The method signature is written in ObjectScript but the method body is written in Python and syntax is enforced by Python standards.

ClassMethod writeToDynamoDBpython(tableName As %String) [ Language = python ]

We begin by using ClassMethod as when the code is compiled and run ObjectScript won’t see it when it is called. The same “tableName” will be handed to this method for the Embedded Python to work with so we will use the same variable name here (since these are just Strings the program needs no additional adjustments/transformations). We must also indicate the language to follow using the square bracket notation.

Step 3: Import boto3 and Create the Client

Python syntax requires importing libraries at the outset of your implementations. In this case we will be doing it inside the method body. Importing a Python library is normally done at the top of the file but as this an ObjectScript file doing the import outside of the method body would cause an exception.

ClassMethod writeToDynamoDBpython(tableName As %String) [ Language = python ]
{
    import boto3

    client = boto3.client('dynamodb')
}

Next, the client needs to be created in order to connect to Dynamo. Boto3 provides a straightforward way of creating that client however note that the service required MUST be passed in, e.g. ‘dynamodb’, otherwise it won’t connect to the correct services (other examples are ‘s3’, ‘cloudwatch’, etc.)

For this use case we will create a client that points to a docker container that has been set up to run an image of Dynamo so that we don’t accrue too many calls to AWS:

client = boto3.client(
    service_name='dynamodb', 
    region_name='us-east-1', 
    endpoint_url='http://host.docker.internal:8080'
)

Step 4: Call the put_item method

AWS provides extensive documentation regarding the many fields that may be included in the call to put_item(). In this use case we need only to send it the “First_Name”, “Last_Name”, and “Employee_Number” as follows:

response = client.put_item(
            TableName=tableName,
            Item={
                'First_Name': {
                    'S': 'Vic'
                },
                'Last_Name': {
                    'S': 'Cordova'
                },
                'Employee_Number': {
                     'N': '012345',
                }
)

There are two required fields from the standpoint of AWS: tableName and the item being inserted into the table, e.g. “Item”. When using this implementation of the client the format for the object being inserted takes a key:value form. The name of the object, e.g. ‘First_Name’, indicates the table column name. The following ‘S’ inside of the ‘First_Name’ object indicates to AWS that the associated value is of type String (there are of course other data types that may be noted here, i.e. ‘SS’ for Lists, ‘N’ for Numbers, etc.). This marker is required when using the client.

The complete Embedded Python method:

ClassMethod writeToDynamoDBpython(tableName As %String) [ Language = python ]
{
    import boto3

    client = boto3.client(
        service_name='dynamodb', 
        region_name='us-east-1', 
        endpoint_url='http://host.docker.internal:8080'
    )

    response = client.put_item(
            TableName=tableName,
            Item={
                'First_Name': {
                    'S': 'Vic'
                },
                'Last_Name': {
                    'S': 'Cordova'
                },
                'Employee_Number': {
                     'N': '012345',
                }
    )
}

Step 5: Call the Embedded Python Method from the ObjectScript Method

Now that the logic has been completed we are able to call the Python method from the ObjectScript method thus making the connection possible. This will take in the tablename and run the logic through Python, writing to Dynamo.

ClassMethod writeToDynamoDB(tableName As %String)
{
    DO ..writeToDynamoDBpython(tableName)
}

As this stands it is usable however any upstream callers may require a returned status. In order to accomplish this the following step will suggest some ways to indicate a success/failure to any calling operations.

Step 6: Optional: Return a %Status

In order to pass a success/failure status upstream consider the following edits:

ClassMethod writeToDynamoDB(tableName As %String) As %Status
    {
        #dim tSC As %Status = $$$OK

        Set pyStatus = ..writeToDynamoDBpython(tableName)

        If (pyStatus '= "OK") {Set tSC = $$$ERROR($$$GeneralError,"failed to write to DynamoDB")}

        Quit tSC
    }

ClassMethod writeToDynamoDBpython(ByRef tableName As %String) [ Language = python ]
{
    import boto3

    client = boto3.client(
        service_name='dynamodb', 
        region_name='us-east-1', 
        endpoint_url='http://host.docker.internal:8080'
    )

    result = "OK"

    try:
    response = client.put_item(
            TableName=tableName,
            Item={
                'First_Name': {
                    'S': 'Vic'
                },
                'Last_Name': {
                    'S': 'Cordova'
                },
                'Employee_Number': {
                     'N': '012345',
                }
   )
       except:
            result = "BAD"

    return result
}

• Add %Status to the ObjectScript method’s definition.
• Set up a status variable (e.g. “tSC”) to hold the $$OK macro.
• Create a status variable in the Embedded Python method using a simple string, “OK”
• Wrap you put_item() in a try/except. The “except:” should update the status variable to something other than “OK” in the case of a failed write.
• Return the status string from the Python method.
• Set up a variable that stores the value returned by the Python method (e.g. “ Set pyStatus = ..writeToDynamoDBpython(tableName)”)
• Check the result of that response in ObjectScript using an if check. In the case of a failure, update the status variable (“tSC”) to be an $$ERROR($$GeneralError) adding an appropriate error message.
• Finally, terminate the method using Quit and return the value of tSC to the upstream caller.

Potential Hurdles and Caveats

This implementation shows a simple example in order to provide the foundation from which an implementer may continue to explore based on their individual use-case and requirements.

Communication between ObjectScript and AWS may prove to be the most tricky as it necessitates an intermediary passing statuses and values back and forth in potentially disparate data types.

Another thing to note is that this implementation is hard-coded. In order to have a more programmatic approach, for example passing in a JSON string, edits like passing the JSON as a %DynamicObject and utilizing the .toJSON() method in ObjectScript in conjunction with leveraging Python’s ability to convert a JSON to a dictionary object using the load() method can be one approach.

Raising exceptions from Embedded Python is also a hurdle as the data types don’t line up. This implementation used Strings to indicate success/failure in conjunction with try/except and if-statements (as noted in the code snippet above) in order to perform some approximation of error handling. It is also worth mentioning that Python exceptions will be printed out, for example if required data is missing, without having to be passed to ObjectScript. The propagation of exceptions is what may require some finagling based on your individual use-case.

Summary/In Conclusion

In summary, establishing a connection to AWS using a Dynamo client, creating a wrapper method in ObjectScript, housing the put_item() logic in a ClassMethod that uses Embedded Python, and making the necessary adjustments per individual requirements makes interacting with Dynamo possible.

Referencing the Documentation will also prove to be very fruitful as there exists sufficient support on both the part of InterSystems and AWS.

Resources

• Intersystems documentation for Embedded Python
• Intersystems course for Embedded Python
• Amazon’s Boto3 documentation
• Amazon’s documentation for the put_item() method