Environment setup with config-api

Article

Lorenzo Scalese · Apr 15, 2021 6m read

Open Exchange

#Best Practices #Deployment #DevOps #InterSystems IRIS

Hi Developers,

Writing a script for the application deployment can be very interesting to ensure rapid deployment without forgetting anything.
config-api is a library to help developers to write configuration scripts based on a JSON document.

Implemented features :

Set system settings.
Set security settings.
Enable services.
Configure namespaces, databases, mapping.
Export existing configuration.
All features are exposed with a RESTful API.

This library is focused on IRIS configuration to help applications deployment. So, config-api doesn't import\compile code feature, considering it should be the role of your application installer module or the client registry. config-api could be used with the ZPM client to configure IRIS settings on module deployment, we learn how to combine this library with ZPM in another article.

Install

zpm “install config-api”

If you aren't a ZPM user, download the latest version in XML format with dependencies release page import and compile.

First step

Let’s write a simple configuration JSON document to set a few system settings.
In this first document we :

Enable journal Freeze on error.
Set limit journal size to 256 MB.
Set SystemMode to development.
Increase the locksiz.
Increase the LockThreshold.

Set config = {
  "Journal": {                                /* Service class Api.Config.Journal */
       "FreezeOnError":1,
       "FileSizeLimit":256
   },
   "SQL": {                                    /* Service class Api.Config.SQL */
       "LockThreshold" : 2500
   },
   "config": {                                 /* Service class Api.Config.config */
       "locksiz" : 33554432
   },
   "Startup":{                                 /* Service class Api.Config.Startup */
       "SystemMode" : "DEVELOPMENT"
   }
}
Set sc = ##class(Api.Config.Services.Loader).Load(config)

configuration JSON document structure

The first level keys (Journal,SQL,config,Startup) are related to classes in %SYS namespace (using an intermediate class in Api.Config.Services package). It means Journal support all properties available in Config.Journal, SQLall properties in Config.SQL, etc...

Output :

2021-03-31 18:31:54 Start load configuration
2021-03-31 18:31:54 {
  "Journal":{
    "FreezeOnError":1,
            "FileSizeLimit":256
  },
  "SQL":{
    "LockThreshold":2500
  },
  "config":{
    "locksiz":33554432
  },
  "Startup":{
    "SystemMode":"DEVELOPMENT"
  }
}
2021-03-31 18:31:54  * Journal
2021-03-31 18:31:54    + Update Journal ... OK
2021-03-31 18:31:54  * SQL
2021-03-31 18:31:54    + Update SQL ... OK
2021-03-31 18:31:54  * config
2021-03-31 18:31:54    + Update config ... OK
2021-03-31 18:31:54  * Startup
2021-03-31 18:31:54    + Update Startup ... OK

Trick : The Load method is compatible with a string argument, in this case, the string must be a filename to a JSON configuration document (stream object is also permitted).

Create an application environment

In this section, we write a configuration document to create :

A namespace "MYAPP".
4 databases (MYAPPDATA, MYAPPCODE, MYAPPARCHIVE,MYAPPLOG)
1 CSP Web Application (/csp/zwebapp).
1 REST Web Application (/csp/zrestapp).
Setup globals mapping.

Set config = {
    "Defaults":{
        "DBDIR" : "${MGRDIR}",
        "WEBAPPDIR" : "${CSPDIR}",
        "DBDATA" : "${DBDIR}myappdata/",
        "DBARCHIVE" : "${DBDIR}myapparchive/",
        "DBCODE" : "${DBDIR}myappcode/",
        "DBLOG" : "${DBDIR}myapplog/"
    },
    "SYS.Databases":{
        "${DBDATA}" : {"ExpansionSize":128},
        "${DBARCHIVE}" : {},
        "${DBCODE}" : {},
        "${DBLOG}" : {}
    },
    "Databases":{
        "MYAPPDATA" : {
            "Directory" : "${DBDATA}"
        },
        "MYAPPCODE" : {
            "Directory" : "${DBCODE}"
        },
        "MYAPPARCHIVE" : {
            "Directory" : "${DBARCHIVE}"
        },
        "MYAPPLOG" : {
            "Directory" : "${DBLOG}"
        }
    },
    "Namespaces":{
        "MYAPP": {
            "Globals":"MYAPPDATA",
            "Routines":"MYAPPCODE"
        }
    },
    "Security.Applications": {
        "/csp/zrestapp": {
            "DispatchClas" : "my.dispatch.class",
            "Namespace" : "MYAPP",
            "Enabled" : "1",
            "AuthEnabled": "64",
            "CookiePath" : "/csp/zrestapp/"
        },
        "/csp/zwebapp": {
            "Path": "${WEBAPPDIR}zwebapp/",
            "Namespace" : "MYAPP",
            "Enabled" : "1",
            "AuthEnabled": "64",
            "CookiePath" : "/csp/zwebapp/"
        }
    },
    "MapGlobals":{
        "MYAPP": [{
            "Name" : "Archive.Data",
            "Database" : "MYAPPARCHIVE"
        },{
            "Name" : "App.Log",
            "Database" : "MYAPPLOG"
        }]
    }
}
Set sc = ##class(Api.Config.Services.Loader).Load(config)

Output :

2021-03-31 20:20:07 Start load configuration
2021-03-31 20:20:07 {
  "SYS.Databases":{
    "/usr/irissys/mgr/myappdata/":{
      "ExpansionSize":128
    },
    "/usr/irissys/mgr/myapparchive/":{
    },
    "/usr/irissys/mgr/myappcode/":{
    },
    "/usr/irissys/mgr/myapplog/":{
    }
  },
  "Databases":{
    "MYAPPDATA":{
      "Directory":"/usr/irissys/mgr/myappdata/"
    },
    "MYAPPCODE":{
      "Directory":"/usr/irissys/mgr/myappcode/"
    },
    "MYAPPARCHIVE":{
      "Directory":"/usr/irissys/mgr/myapparchive/"
    },
    "MYAPPLOG":{
      "Directory":"/usr/irissys/mgr/myapplog/"
    }
  },
  "Namespaces":{
    "MYAPP":{
      "Globals":"MYAPPDATA",
      "Routines":"MYAPPCODE"
    }
  },
  "Security.Applications":{
    "/csp/zrestapp":{
      "DispatchClas":"my.dispatch.class",
      "Namespace":"MYAPP",
      "Enabled":"1",
      "AuthEnabled":"64",
      "CookiePath":"/csp/zrestapp/"
    },
    "/csp/zwebapp":{
      "Path":"/usr/irissys/csp/zwebapp/",
      "Namespace":"MYAPP",
      "Enabled":"1",
      "AuthEnabled":"64",
      "CookiePath":"/csp/zwebapp/"
    }
  },
  "MapGlobals":{
    "MYAPP":[
      {
        "Name":"Archive.Data",
        "Database":"MYAPPARCHIVE"
      },
      {
        "Name":"App.Log",
        "Database":"MYAPPLOG"
      }
    ]
  }
}
2021-03-31 20:20:07  * SYS.Databases
2021-03-31 20:20:07    + Create /usr/irissys/mgr/myappdata/ ... OK
2021-03-31 20:20:07    + Create /usr/irissys/mgr/myapparchive/ ... OK
2021-03-31 20:20:07    + Create /usr/irissys/mgr/myappcode/ ... OK
2021-03-31 20:20:07    + Create /usr/irissys/mgr/myapplog/ ... OK
2021-03-31 20:20:07  * Databases
2021-03-31 20:20:07    + Create MYAPPDATA ... OK
2021-03-31 20:20:07    + Create MYAPPCODE ... OK
2021-03-31 20:20:07    + Create MYAPPARCHIVE ... OK
2021-03-31 20:20:07    + Create MYAPPLOG ... OK
2021-03-31 20:20:07  * Namespaces
2021-03-31 20:20:07    + Create MYAPP ... OK
2021-03-31 20:20:07  * Security.Applications
2021-03-31 20:20:07    + Create /csp/zrestapp ... OK
2021-03-31 20:20:07    + Create /csp/zwebapp ... OK
2021-03-31 20:20:07  * MapGlobals
2021-03-31 20:20:07    + Create MYAPP Archive.Data ... OK
2021-03-31 20:20:07    + Create MYAPP App.Log ... OK

It works! The configuration is successfully loaded.

In the next article, we learn how to use config-api with ZPM to deploy your application.

The app is submitted for the contest, vote for it if you like it ;-) .

Thanks for reading.

{ "Defaults":{ "DBDIR" : "${MGRDIR}", "DBDATA" : "${DBDIR}irisapp/" }, "SYS.Databases":{ "${DBDATA}" : {} }, "Databases":{ "IRISAPP" : { "Directory" : "${DBDATA}" } }, "Namespaces":{ "IRISAPP": { "Globals":"IRISAPP" } } }

Lorenzo Scalese · Apr 16, 2021

Hi @Guillaume Rongier ,

Thank you for this very interesting question.

Indeed, you can't tell directly zpm to load the classes in the freshly installed namespace.

You need to perform:

zn "irisapp"
zpm "install module-name"

I consider loading the code is the role of ZPM.
In my opinion congi-api shouldn't have an option to load the code.
We could add an option to call ZPM. Something like this :

{
    "ZPM" : {
        "IRISAPP" : {
            "install" : "module-name"
        }
    }
}

Execute zpm "install module-name" in namespace IRISAPP

Obviously this is subject to discussion. Community feedback would be appreciated.

About CPF module, I agree, my library has an overlap with all classes related to %SYS Config package.
For these operations the only difference is the REST expose. So, no added value if developers don't need REST expose.
However, config-api implement a part of %SYS Security package and also %Library SQLConnection.
You can configure users, roles, resources, ssl configuration, web application, enable services, sql connexions, set SQL Privileges and you don't with CPF module.

Hope answered to your questions.

0 0

Guillaume Rongier · Apr 15, 2021

What a nice feature.

If I understand correctly, this module can be considered as a replacement for the Manifest installer?

If so, then this module has the benefit of exposing the configuration as JSON and API rather than a compiled class.

1 0

Lorenzo Scalese · Apr 15, 2021

Exactly @Guillaume Rongier !

I am a Manifest Installer user and It works fine.

However, sometimes people which deploy applications aren't ObjectScript developers.

Manifest Installer, need to create a class, write an XML, compile.
I would like a rupture between the configuration and the code.

Moreover, everything is exposed in REST. It could be interesting to integrate IRIS configuration from a tool in another language.

Thank you!

Evgeny Shvarov · Apr 15, 2021

Wow. So now we can setup the environment using JSON and do more things than we can with %Installer, right?

Like users, roles, mappings, etc. Great feature!

Thank you @Evgeny Shvarov
So, the main goals are REST expose and the rupture between config and code.

%Installer allows to create, users, roles, mapping too, but it's not possible to configure SQL Privileges, SSL Configuration, SQL Connexion (config-api can do).

There are a few features existing in %Installer and not in config-api, but we have a good base to implement if needed.

What is the difference between SYS.Databases and Databases?

Got it. This is if I want special parameters like Expansion size, thanks.

Yes, that's it.

And it looks like that if I want to create a database I need to have two entries:

SYS.Databases and

Databases.

Here is the minimal config to create IRISAPP namespace with one database IRISAPP in /irisapp folder.

Works.

I use it in the following way to set up a docker container before loading the ZPM dev module into it.

Nice !

> I need to have two entries: SYS.Databases and Databases.

Yes, perhaps should I think to implement a short way to avoid two entries when we use default settings.

This is the correct usage.

To deploy a large application divided into several modules, using many namespaces, databases, web applications can be complex. The best way is to write a script to configure your environment and then deploy the code as you did in this simple example with zpm.

Guillaume Rongier · Apr 16, 2021

I tried your module as a replacement for the Manifest installer, see this example:

https://github.com/grongierisc/intersystems-iris-dev-template

However, I'm stuck for the part with ZPM.
The problem is, with your module, I can create a database, its rights, etc., but how to tell ZPM to load the classes in the freshly installed NameSpace?

What are the possible solutions?

Add an option to your module to load code ?
Add an option to ZPM to take into account your configurations ?
What to do with the CPF module?
- https://docs.intersystems.com/irisforhealthlatest/csp/docbook/DocBook.UI...

We have today many options to load configuration IRIS, are they in competition?

How to federate these modules who are all complementary?

Lorenzo Scalese · Apr 19, 2021

I added a feature in version 1.0.2 to call zpm directly.

The forked repository objectscript-docker-template is up to date. See commit.

So, now for a complex application, we can create multiple namespaces and execute zpm command on freshly created namespaces. I am still open to any suggestions.

Documentation up to date here

intersystems-iris-dev-template is also forked on my GitHub with config-api usage.
You can see the differences on this commit.
I'll keep this repo up to date for testing purposes if there is any change.

@Evgeny Shvarov, @Guillaume Rongier

Robert Cemper · Apr 21, 2021

This is an excellent application.
- It allows to create a base configuration and handle variants.
- And it's an excellent readable and easy-to-understand documentation.
- On top, it is JSON based and therefore well suited to source and version management

The original CachéParameterFile iris.cpf does a mimic of versioning.
But it is buried deep into the installation directory and as cryptic as the
Egyptian Book of the Dead. [reserved to the priest of IRIS cultus]

I was waiting for something similar useful for decades!
I suggest:

No one running and supporting more than 2 configurations should miss it!

3 0

Lorenzo Scalese · Apr 21, 2021

Thank you @Robert Cemper .
Glad to hear that!

The original CachéParameterFile iris.cpf does a mimic of versioning.
But it is buried deep into the installation directory and as cryptic as the
Egyptian Book of the Dead. [reserved to the priest of IRIS cultus]

;-) Yes, I guessed that there is an existing feature for cpf versionning when I see classmethod parameters in Config.CommonSingleMethods :
classmethod Get(ByRef Properties As %String, ByRef CPFFile As %String = "", Flags As %Integer = $$$CPFSave+$$$CPFWrite+$$$CPFActivate) as %Status [ Language = objectscript ]

Luca Ravazzolo · Jul 13, 2021

@Lorenzo Scalese great way of exposing the IRIS internal API! I like it and I like the way the community brings innovation and supports the needs of users. Great effort, Lorenzo!

I also wanted to draw attention to a utlity that InterSystems has been supporting for several versions. We call this feature the CPF merge feature.

Q: What is the CPF merge feature?

A: It's the capability to configure an instance dynamically from the outside. It can be used with any configuration management tool like Chef, Puppet, Ansible, Salt or simpler bash or any cloud service provider provisioning tool like AWS CloudFormation, Terraform or orchestrator like Kubernetes. A user can define the ultima state of an IRIS instance. The operation is executed idempotently and all you need is an environment variable called ISC_CPF_MERGE_FILE=the_file_that_holds_my_desired_config

The CPF merge file could have been JSON, YAML, TOML or whatever but we decided to go with the familiar format we know, for now. The CPF merge file provides a way to Create, Delete and Update instance resources.

The doc.

Some Examples - Note how the CPF merge feature does not only helps us in single instance configuration but also automates more complex cluster configurations like Mirror pairs and shard architecture topologies.

I hope this is useful to the reader who is seeking more elegant and easy ways to automate InterSystems IRIS clusters.

Lorenzo Scalese · Jul 16, 2021

Thank you @Luca Ravazzolo !

Developer Commu... · Jul 16, 2021

💡 This article is considered as InterSystems Data Platform Best Practice.