How to set up an IRIS Sharding cluster in less than a minute

Article

#Beginner #Development Environment #DevOps #Docker #Sharding #InterSystems IRIS

In this article I'll show you how to set up in your laptop, very quickly, a cluster of IRIS nodes in sharding. It's not the goal of this article neither to talk about sharding in detail nor define a deployment of a production ready architecture, but to show how to set up quickly, in your own machine, a cluster of IRIS instances configured as shard nodes, with which you'll able to play and test this functionality. If you're insterested in knowing more about sharding in IRIS, take a look at the documentation clicking here.

First and foremost, I want to remark that IRIS sharding will allow us 2 things:

Define, load and query shard tables, which data will be distributed transparently between the cluster's nodes
Define federated tables, which offer a global and composed view of data belonging to different tables that will be physically stored in different distributed nodes

So, as I said, we let for other article playing with shard or federated tables, and just focus now in the previous step, that is, setting up the cluster of shard nodes.

Well, for our example we're going to use Docker Desktop (for Windows or MacOS) and we will make use of an IRIS feature: CPF Merge; which allow us to use a plain text file within which we'll include IRIS sections/configuration properties that we want to apply to update the default configuration of an IRIS instance. Basically this file overwrites some parts of the iris.cpf file that defines the default configuration of the instance.

This merging is "triggered" automatically when we add the environment variable: ISC_CPF_MERGE_FILE and it's set with a valid path to a file that contains those sections of the cpf file that we want to change. When IRIS starts, it checks if there is a merge file to apply (basically, if it exists that environment variable and points to a valid file). If so, do the merge and starts.

I won't ramble on, here is the file docker-compose.yml that'll do the magic:

docker-compose.yml

services:
  # iris container
  irisnode1:
    init: true 
    hostname: irishost1
    image: shardnode:latest 
    container_name: irisnode1
    build: 
      context: ./cluster
      dockerfile: Dockerfile
    ports:
    - "9991:1972"
    environment: 
    - ISC_DATA_DIRECTORY=/durable/irishost1
    - ISC_CPF_MERGE_FILE=/iris-shared/merge_first_data-node.cpf
    command: 
       --check-caps false --ISCAgent false --key /iris-shared/iris.key -a /iris-shared/configure_first_data-node.sh
    volumes:
    - ./cluster/iris-instance:/iris-shared:delegated
    - ./DDBBs:/durable:delegated

  irisnode2:
    init: true 
    hostname: irishost2
    image: shardnode:latest 
    container_name: irisnode2
    build: 
      context: ./cluster
      dockerfile: Dockerfile
    ports:
    - "9992:1972"
    environment: 
    - ISC_DATA_DIRECTORY=/durable/irishost2
    - ISC_CPF_MERGE_FILE=/iris-shared/merge_data-node.cpf
    command: 
       --check-caps false --ISCAgent false --key /iris-shared/iris.key -a /iris-shared/configure_data-node.sh
    volumes:
    - ./cluster/iris-instance:/iris-shared
    - ./DDBBs:/durable
    depends_on:
      irisnode1:
        condition: service_healthy
    
  # web gateway container
  webgateway:
    image: containers.intersystems.com/intersystems/webgateway:latest-em
    init: true
    container_name: webgateway
    hostname: webgateway
    ports:
    - 7772:80
    - 7773:443
    environment:
    - ISC_CSP_CONF_FILE=/webgateway-shared/CSP.conf
    - ISC_CSP_INI_FILE=/webgateway-shared/CSP.ini
    volumes:
    - ./webgateway/CSP.conf:/webgateway-shared/CSP.conf
    - ./webgateway/CSP.ini:/webgateway-shared/CSP.ini

Here you have also an example of CSP.conf and CSP.ini files:

CSP.conf

# CSP config file
CSPModulePath "${ISC_PACKAGE_INSTALLDIR}/bin/"
CSPConfigPath "${ISC_PACKAGE_INSTALLDIR}/bin/"
# Serve everything via Web Gateway. Conveniently,
# we needn't worry about sharing this container with non-IRIS applications.
<Location />
    CSP On
</Location>
<Directory />
    Options MultiViews FollowSymLinks
    AllowOverride None
    Require all granted
    <FilesMatch "\.(log|ini|pid|exe|so)$">
        Require all denied
    </FilesMatch>
</Directory>
# Redirect Help links
Redirect /csp/docbook/ http://docs.intersystems.com/irislatest/csp/docbook/

CSP.ini

[SYSTEM]
IRISCONNECT_LIBRARY_PATH=/opt/webgateway/bin
System_Manager=*.*.*.*
SM_Timeout=28800
Server_Response_Timeout=60
No_Activity_Timeout=86400
Queued_Request_Timeout=60
Default_Server=IRISHOST1

[SYSTEM_INDEX]
IRISHOST1=Enabled
IRISHOST2=Enabled

[IRISHOST1]
Ip_Address=irishost1
TCP_Port=1972
Minimum_Server_Connections=3
Maximum_Session_Connections=6
Username=CSPSystem
Password=]]]U1lT
Connection_Security_Level=0

[IRISHOST2]
Ip_Address=irishost2
TCP_Port=1972
Minimum_Server_Connections=3
Maximum_Session_Connections=6
Username=CSPSystem
Password=]]]U1lT
Connection_Security_Level=0

[APP_PATH_INDEX]
/=Enabled
/csp=Enabled
/irishost1=Enabled
/irishost2=Enabled
/test=Enabled

[APP_PATH:/]
Default_Server=IRISHOST1
Alternative_Server_0=1~~~~~~IRISHOST1

[APP_PATH:/csp]
Default_Server=IRISHOST1
Alternative_Server_0=1~~~~~~IRISHOST1

[APP_PATH:/irishost1]
Default_Server=IRISHOST1
Alternative_Server_0=1~~~~~~IRISHOST1

[APP_PATH:/irishost2]
Default_Server=IRISHOST2
Alternative_Server_0=1~~~~~~IRISHOST2

[APP_PATH:/test]
Default_Server=IRISHOST2
Alternative_Server_0=1~~~~~~IRISHOST2

In this example, we're instantiating 3 services:

irisnode1 - First node of the cluster, it has a special role, that's the reason we named it as node1
irisnode2 - Data node, an aditional node to the cluster, which role is data (we can have as many as we want of these)
webgateway - Webserver preconfigured to access the IRIS instances (Apache + Webgateway)

To build the image shardnode:latest, I've used the dockerfile below:

Dockerfile

FROM containers.intersystems.com/intersystems/irishealth:2024.3
#FROM containers.intersystems.com/intersystems/iris-community:latest-em
#FROM containers.intersystems.com/intersystems/irishealth-arm64:2024.3
USER root
WORKDIR /opt/irisapp
RUN chown -R irisowner:irisowner /opt/irisapp
USER irisowner
WORKDIR /opt/irisapp
COPY --chown=irisowner:irisowner src src
COPY --chown=irisowner:irisowner iris.script iris.script
RUN iris start IRIS \
    && iris session IRIS < iris.script \
    && iris stop IRIS quietly

Within the dockerfile we call this generic iris.script that would allow us to run ObjectScript code, to setup, import & compile code, etc.., in the IRIS image we're building:

iris.script

// Unexpire passwords to simplify dev mode. Comment these two lines for Production use
    zn "%SYS"
    Do ##class(Security.Users).UnExpireUserPasswords("*")

    zn "USER"
    // Load the IRIS Classes
    do $System.OBJ.LoadDir("/opt/irisapp/src","ck",,1)

   // Register CSP applications can be done in the merge.cpf
    halt

The files used for the CPF merge for nodo1 and the other IRIS cluster data nodes are:

merge_first_data-node.cpf

# Primer nodo del cluster
# 2 GB 8k / 204 MB gmpheap / 64 nodos max
[config]
globals=0,0,2048,0,0,0
gmheap=204800
MaxServerConn=64
MaxServers=64
# Define el primer nodo del sharding cluster. Crea un endpoint REST de prueba
[Actions]
ConfigShardedCluster:ClusterURL=IRIS://irishost1:1972/IRISCLUSTER,Role=node1
CreateApplication:Name=/rest/testapp,MatchRoles=:%ALL,NameSpace=USER,DispatchClass=Test.RESTserver,AutheEnabled=64

merge_data-node.cpf

# Nodo de dato adicional
# 2 GB 8k / 204 MB gmpheap / 64 nodos max
[config]
globals=0,0,2048,0,0,0
gmheap=204800
MaxServerConn=64
MaxServers=64
# Define un nodo de datos y lo añade al cluster. Crea un endpoint REST de prueba(opcional)
[Actions]
ConfigShardedCluster:ClusterURL=IRIS://irishost1:1972/IRISCLUSTER,Role=data
# CreateApplication:Name=/rest/testapp,MatchRoles=:%ALL,NameSpace=USER,DispatchClass=Test.RESTserver,AutheEnabled=64

We could have more nodes of type data as part of the cluster only adding additional services with the same definition that we have for irisnode2 (changing the name of course)

One more thing, in order to make our routing definition in the webserver to work properly, and so being able to access the System Management Portals for each instance, we have to change the parameter CSPConfigName in each one of them, and we do that with scripts in the files: configure_first_data-node.sh y configure_data-node.sh; for this example both are equal, but I've leave them as separated files because, at some point, we could want to execute different actions for each IRIS instance during start time, depending on if it's a cluster node of type nodo1 or type data.

configure_data-node.sh

cat << EOF | iris session iris
do ##class(%SYSTEM.CSP).SetConfig("CSPConfigName","$HOSTNAME")
halt
EOF

And that's basically it.

You could define the nodes using the available API in %SYSTEM.Cluster class, but the truth is that the possibility of introducing actions altogether with the CPF Merge feature simplifies the task a lot. I recomend you to take a look at here, in particular at the part referred to the section [Actions].

To build the images and deploy the cluster, we could build our image sharnode:latest and launch our docker-compose from VS Code. Alternately we could do it in our command line, from within the folder in which we have the file: docker-compose.yml, executing these commands:

docker compose build
docker compose up

It'll take a bit the first time because the instance tagged as node1 has to start before whatever other data node in the cluster starts. But everything should be ready and working in 1 minute or less.

If everything went right, you should be able to access to the SMPs of each instance with the URLs below:

SMP cluster node node1: http://localhost:7772/irishost1/csp/sys/UtilHome.csp
SMP cluster node data : http://localhost:7772/irishost2/csp/sys/UtilHome.csp
Access to WebGateway: http://localhost:7772/csp/bin/Systems/Module.cxw

And, done! From here, the limits regarding DDBB storage and tables size it's determined by your hardware. You would have and IRIS cluster with sharding enabled to define your shard and/or federated tables.

Hope it's useful for you!! See you around...

Go to the original post written by @Jose-Tomas Salvador