Open Exchange App Coding Dockerfile with InterSystems ObjectScript

Primary tabs

Hi Developers!

Those who use Dockerfile to work with InterSystems IRIS often need to execute several lines of ObjectScript.  For me, this was a game of "escaping this and that" every time just to shoot a few commands on ObjectScript to IRIS. Ideally, I'd prefer to code ObjectScript without any quotes and escaping.

Recently I found a nice "hack" on how this could be improved to exactly this state. I got this from @Dmitriy Maslennikov's repo and this lets you use Objectscript in a way as you would type it in IRIS terminal.

Here is what you have in dockerfile:

///
COPY irissession.sh /
SHELL ["/irissession.sh"]
RUN \
  do $SYSTEM.OBJ.Load("Installer.cls", "ck") \
  set sc = ##class(App.Installer).setup()
# bringing the standard shell back
SHELL ["/bin/bash", "-c"]
CMD [ "-l", "/usr/irissys/mgr/messages.log" ]
///

The trick is that before executing ObjectScript we call irisession.sh which does the following:

1. starts IRIS

2. Calls some "standard" ObjectScript commands and brings control to Dockerfile to let you introduce lines in ObjectScript - "$@" instruction.

3. stops IRIS

Check the content of the irissession.sh:

#!/bin/bash

iris start $ISC_PACKAGE_INSTANCENAME quietly
 
cat << EOF | iris session $ISC_PACKAGE_INSTANCENAME -U %SYS
do ##class(%SYSTEM.Process).CurrentDirectory("$PWD")
$@
if '\$Get(sc) do ##class(%SYSTEM.Process).Process.Terminate(, 1)
do ##class(SYS.Container).QuiesceForBundling()
do ##class(SYS.Container).SetMonitorStateOK("irisowner")
Do ##class(Security.Users).UnExpireUserPasswords("*")
halt
EOF

exit=$?

iris stop $ISC_PACKAGE_INSTANCENAME quietly

exit $exit

This $@ expects one RUN command in Dockerfile which will contain only ObjectScript. If you need to use some more commands in Dockerfile bring the control back with standard shell:

# bringing the standard shell back
SHELL ["/bin/bash", "-c"]
CMD [ "-l", "/usr/irissys/mgr/messages.log" ]

This code is introduced into IRIS community template to let you start using ObjectScript in your IRIS solutions with accurate and readable code.

Learn how to use Github templates with IRIS  in this video.
 

 

Comments

This looks like a convenient way to handle things. It would be good, though, if someone could explain what the various ObjectScript commands in irissession.sh are needed for. I also see no credentials being passed?

Well, here are some comments:

do ##class(%SYSTEM.Process).CurrentDirectory("$PWD")

This is to make WORKDIR /opt/irisapp current for IRIS.

$@

Here we run the arbitrary ObjectScript in Dockerfile

if '\$Get(sc) do ##class(%SYSTEM.Process).Process.Terminate(, 1)

Here we check the status of the sc variable changed with status from ObjectScript in Dockerfile if it has an error and terminate IRIS in this case and fail the build.

do ##class(SYS.Container).QuiesceForBundling()
do ##class(SYS.Container).SetMonitorStateOK("irisowner")

Two methods to prepare IRIS operate in a container mode properly. I'm pinging @Luca Ravazzolo to provide more details on it.

Do ##class(Security.Users).UnExpireUserPasswords("*")

This removes the password expiration because it's very annoying to change the password on every build. This line could be used for DEVELOPMENT MODE only. Please remove the line if you build the image for PRODUCTION.

Thanks for that, Evgeny, that does clear up things a bit. I hope Luca responds as well about the Quiesce... and SetMonitorState calls. I get the impression that things relating to creating containers are still changing rapidly. I do find the backslash-escaped multiline commands annoyingly ugly; your post above made that a bit better, compared to what was there before. (I'm hoping some of the ugly boilerplate will be delt with by ISC "inside" the container eventually.) I'm going to play around with your template a bit when I get the time!

I also see no credentials being passed?

As of 2019.3, the IRIS images we provide have OS Authentication enabled.  The net effect of this is that login to irissession is automatic for the right Unix user.  This does not change anything else, as the new user can't log in in any other way in those images.

Customers who build their own images from our kits - available on the WRC! - can choose to use this feature or not.

I have enabled and used that feature in older versions as well; it is really useful. I have a PuTTY setup with a key pair that starts the IRIS terminal in docker on a remote machine, without any password, but still safe. (I wanted to use my regular account name as well, so I had to do some additional setup in the container, but the principle remains the same.)

Anyway, that explains why there's no need for a password anymore, thanks!

By the way, I have not een an announcement that 2019.3 became an official release, did I miss something? (I also see no Studio for 2019.3, and the 2019.4 preview Studio download doesn't work.)

Hi Evgeny,

As clever as this hack is, I strongly recommend against this approach.  This is a fun tool to write but a dangerous tool to hand to someone who doesn't understand all the factors at play well enough that they could have written our own.  There are a few problems lurking here just beneath the surface, and when taken together they mean that the SHELL directive in Dockerfiles is not our friend.

Error handling

irissession is a tool intended for human, interactive use.  In containers we often make use of it to "bootstrap" some code into IRIS, but there are caveats here.  The Docker build daemon interprets exit status 0 as success, and anything else as failure.  irissession reports errors by printing them:

USER>write foo
WRITE foo
^
<UNDEFINED> *foo


This error doesn't change irissession's exit status, so it isn't caught by an image build daemon.  Nor are bad status values, though I see that @Dmitry Maslennikov 's caught that by checking "sc". :)  Additionally, while these errors end an ObjectScript routine or method when uncaught, they do not end execution in irissession, which increases the risk that these errors get silently swallowed.

No multi-line blocks

The tool here is similar to pasting your text into irissession, which takes line-oriented input.  This means that if someone looks at the format and uses blocks, they can get caught off-guard, when blocks like this get read as their component lines:

for i=1:1:10 {
  w !, "i: ", i
}

That gets executed like so:

USER>for i=1:1:10 {

FOR i=1:1:10 {
               ^
<SYNTAX>
USER>  w !, "i: ", i

i: 1
USER>}

}
^
<SYNTAX>
USER>

In simple cases, this is no problem to cram onto one line: "for i=1:1:10  w !, "i: ", i"  In complex real-world cases, this paradigm breaks down quickly. 

Backslashes

All this and we don't quite get the ability to paste in ObjectScript code from elsewhere.  Because of the Dockerfile's syntax re: line endings, each line of ObjectScript that we'd put in still needs an extra backslash, except for the last one.

It's okay, though, because there's a better way.

A better way

Every time you cross the ObjectScript/shell boundary, you have to be careful.  Since you have to be careful there, why not cross it as few times as possible?  Your existing payload is good:

  do $SYSTEM.OBJ.Load("Installer.cls", "ck")
  set sc = ##class(App.Installer).setup()

But it could be better.  If you're writing Installer.cls, why not add all the other calls there?  All of these:

if '$Get(sc) do ##class(%SYSTEM.Process).Process.Terminate(, 1)
do ##class(SYS.Container).QuiesceForBundling()
do ##class(SYS.Container).SetMonitorStateOK("irisowner")
Do ##class(Security.Users).UnExpireUserPasswords("*")

Can be added into your installer.cls.  There, you can engage in as-sophisticated-as-you-want error handling, with no need for backslashes and no need to keep a tight leash on your use of whitespace.

You'll still need to have a minimum of one pipe into irissession, until we add other features to streamline this process, but your code will be more readable, and your daily builds will be more robust.

InterSystems IRIS 2019.3 includes a lot of new features for containers, several of which are aimed at streamlining this kind of image building.  Take a look at what we've committed to https://github.com/intersystems/container-tools, and what the 2019.3 Class Reference has to say about the SYS.Container class.  These things are intended to make the ObjectScript-in-a-Dockerfile process less frustrating, and to demystify some of the things we do like "kill ^SYS("NODE")".

Hi Conor!

Thanks for really wise suggestions.  Agreed with everything. Two cents/comments:

1. I think we never want to code a lot of ObjectScript inside Dockerfile. Only a few lines to make some necessary configurations and the intention is to make the code more readable paying with some coding conditions (like sc hard-coded sc for error handling). 

2. I am very supportive about putting all the possible setup activity into Installer.cls. But the issue is that not all the actively used setup tweaks are supported by %Installer format, e.g. RESTFul Web app. If you suggest to use <Invoke> - it's better to call method directly. So, here I'm putting some wishes for %Installer format approvements.

Always open for improvements.

I would rather expect that some kind of irissession.sh was implemented by InterSystems. It is a docker build process, InterSystems already added ##class(SYS.Container).QuiesceForBundling() which in fact I would expect it should be called automatically while IRIS stopping during docker build.

I'm against putting all of those container's only lines to Installer manifest, just because of Installer manifests can be used in non-container's environment as well. And those lines need only during docker build.

PS. With 2019.4 this line

do ##class(SYS.Container).SetMonitorStateOK("irisowner")

should not be used (even will give an error), as it is already as part of 

do ##class(SYS.Container).QuiesceForBundling()

Multi-line blocks

It's actually working. And it's because of code from RUN command goes to chosen SHELL as one line without line endings, \ at the end of the line, is not line ending it is line continuation. So, you can write code in Dockerfile visually like multi-line but it is no so.

Error handling

Any error in iris session will change exit code, so, any such error will fail docker build anyway.

So, with, such lines in my Dockerfile

SHELL ["/irissession.sh"]
RUN \
  write !,"Start",! \  
  for i=1:1:10 { \
    write !,"Test Loop",i \ 
  }\
  write !,"Finish",! \
  write !, error

I will get this

Starting IRIS

Node: 24032930b1e3, Instance: IRIS
%SYS>
%SYS>
Start
Test Loop1
Test Loop2
Test Loop3
Test Loop4
Test Loop5
Test Loop6
Test Loop7
Test Loop8
Test Loop9
Test Loop10
Finish

WRITE !,"Start",!   FOR i=1:1:10 {     WRITE !,"Test Loop",i   }  WRITE !,"Finis
h",!   WRITE !, error
                ^
<UNDEFINED> *error
%SYS>
ERROR: Service 'iris' failed to build: The command '/irissession.sh write !,"Start",!   for i=1:1:10 {     write !,"Test Loop",i   }  write !,"Finish",!   write !, error' returned a non-zero code: 1

As you can see, for loop working well, and UNDEFINED error fails to build

That kind of whitespace transformation has serious tradeoffs in ObjectScript.  While they're not commonly used, ObjectScript does have cases where two spaces are significantly different than one (argumentless commands), or where a newline and a space are significantly different (legacy versions of no-block FOR, DO, etc).  The way that docker parses these things in a Dockerfile, as part of SHELL, mangles that whitespace.

I would rather expect that some kind of irissession.sh was implemented by InterSystems.

The reasons I've listed here are why we have chosen not to implement a feature like this.  It is not possible to embed exact ObjectScript syntax inside a Dockerfile.  I recommend, wherever possible, piping as few lines as you can into an irissession, and letting the routines or classmethods handle things from there.  This is part of why QuiesceForBundling exists.

See this done in imageBuildSteps.sh on GitHub: https://github.com/intersystems/container-tools/blob/master/2019.3/offic...

Unfortunately, InterSystems officially does not offer how to deal with docker at all. Where articles from, let's say from you or somebody else, with the ways how to do it the Right way. 

Comments to the repository https://github.com/intersystems/container-tools:

  • 2019.4 already exists and published, but examples still only for 2019.3
  • ZSTART.XML in XML format. Why so, when it should be MAC file
    • quite arguing the decision, to load code in this place
    • endless loop with goto in the code
  • Dockerfile, with a line longer than screen size. 
    • where QuiesceForBundling?

InterSystems say how to run InterSystems products with docker, while nothing about running customer's applications on IRIS with Docker.

What's happens here is, it is how the community works. I have a problem, I don't want to write to much in my Dockerfile. I wrote a lot of them, for many different projects. I need some universal way, for me. I did it, and I update my own template from time to time.