Package Manager for InterSystems: the Design Question to All Engineers |

Discussion

Nikita Savchenko · Dec 12, 2019

#InterSystems Package Manager (IPM) #ObjectScript #Caché #InterSystems IRIS

Hello, InterSystems community!

Lately, you have probably heard of the new InterSystems Package Manager - ZPM. If you're familiar with it or with such package managers as NPM, Dep, pip/PyPI, etc. or just know what is it all about -- this question is for you! The question I want to arise is actually a system design question, or, in other words, "how should ZPM implement it".

In short, ZPM (the new package manager) allows you to install packages/software to your InterSystems product in a very convenient, manageable way. Just open up the terminal, run ZPM routine, and type install samples-objectscript: you will have a new package/software installed and ready to use! In the same way, you can easily delete and update packages.

From the developer's point of view, quite as same as in other package managers, ZPM requires the package/software to have a package description, fairly represented as module.xml file. Here's an example of it. This file has a description of what to install, which CSP applications to create, which routines to run once installed and so on.

Now, straight to the point. You've also probably heard of InterSystems WebTerminal - one of my projects which is quite widely used (over 500 installs over the last couple of months). We try to bring WebTerminal to ZPM.

So far, anyone could install WebTerminal just by importing an XML file with its code - no more actions were needed. During the class compilation, WebTerminal runs the projection and does all required settings on its own (web application, globals, etc - see here). In addition to this, WebTerminal has its own self-updating mechanism, which allows it to self-update when the new version comes out, made exactly with the use of projections. Apart from that, I have 2 more projects (ClassExplorer, Visual Editor) that use the same import-and-install convenient installation mechanism.

But, it was decided that ZPM won't accept projections as a paradigm and everything should be described in module.xml file. Hence, to publish WebTerminal for ZPM, the team tried to remove Installer.cls class (one of WebTerminal's classes which did all the install-update magic with the use of projections) and manually replaced it with some module.xml metadata. It turned to be quite enough for WebTerminal to work but it potentially leads to unexpected incompatibilities to be 100% compatible with ZPM (see below). Thus, the source code changes are needed.

So the question is, should ZPM really avoid all projection-enabled classes for its packages? The decision of avoiding projections might be changed via the open discussion here. It's not a question of why can't I rewrite WebTerminal's code, but rather why not just accept original software code even if it uses projections?

My opinion was quite strong against avoiding projection-enabled classes in ZPM modules. For multiple reasons. But first of all, because projections are the way how the programming language works, and I see no constructive reasoning against using them for whatever the software/package is designed for. Avoiding them and cutting Installer.cls class from the release is absolutely the same as patching a working module. I agree that the packages which ship specifically for ZPM should try to use all installation features which module.xml provides, however, WebTerminal is also shipped outside of ZPM, and maintaining 2 versions of WebTerminal (at least, because of the self-update feature) makes me think that something is wrong here.

I see the next pros of keeping all projection-enabled classes in ZPM:

The package/software will still be compatible with both ZPM and a regular installation done for years (via XML/classes import)
No original package/software source code changes needed to bring it to ZPM
All designed functions work as expected and don't cause problems (for instance, WebTerminal self-updates - upon the update, it loads the XML file with the new version and imports it, including projection-enabled Installer.cls file anyway)

Cons of keeping all projection-enabled classes in ZPM:

Side effects made during the installation/uninstallation, made by projection-enabled classes won't be statically described in the module.xml file, hence they are "less auditable". There is an opinion that any side effect must be described in module.xml file.

Please indicate any other pros/cons if this isn't the full list. What do you think?

Thank you!

Sergei Shutov · Dec 12, 2019

The whole purpose of package manager is to get rid of individual installer/updater scripts written by individual developers and replace them with package management utility. So that you have a standard way of installing, removing and updating your packages. So I don't quite understand why this question is raised in this context -- of course package manager shouldn't support custom installers and updaters. It might need to support Projections eventually because as you said it's a part of language, but definitely not for installing purposes.

1 0

Exactly not for installing purposes, you're right, I agree. But what do you think about the WebTerminal case in particular?

1. It's already developed and bound to projections: installation, its own update mechanism, etc.
2. It's also shipped outside of ZPM
3. It would work as usual if only ZPM supported projections

I see you're pointing out to "It might need to support Projections eventually because as you said it's a part of language" - that's what mostly my point is about. Why not just to allow them.

0 0

Thanks! Considering the points others mention, I agree that projections should not be the way to install things but rather the acceptable exception as for WebTerminal and other complex packages.

Thanks! Exactly, I completely agree about simplicity, transparency, and installation standard. But see my answer to Sergey's answer - what to do with WebTerminal in particular?

1. Why would I need to rewrite the update mechanism I developed years ago (for example)?
2. Why would I need to maintain 2 code bases for ZPM & regular installations (or automate it in a quite crazy way, or just drop self-update feature when ZPM is detected)
3. Why all these changes to the source code are needed, after all, if it "just works" normally without ZPM complications (which is how the ObjectScript works)

I think this leads to either "make a package ZPM-compatible" or "make ZPM ObjectScript-compatible" discussion, isn't it?

True points. For sure, developers can customize it. I can do another version of WebTerminal specifically for ZPM, but it will involve additional coding and support:

1. A need to change how the self-update mechanism works or shut it down completely. Right now, the user gets a message on the UI, suggesting to update WebTerminal to the latest version. There's quite a lot of things happen under the hood.
2. Thus, create an additional pipeline (or split the codebase) for 2 WebTerminal versions: ZPM's one and a regular one with all the tests and so on.

I am wondering is it worth doing so in WebTerminal's perspective, or is it better to make WebTerminal a kind of an exception for ZPM. Because, still, inserting a couple of if (isZPMInstalled) { ... } else { ... } conditions to WebTerminal (even on front-end side) looks as anti-pattern to me.

Eduard Lebedyuk · Dec 12, 2019

I completely support inclusion of projections.

ObjectScript Language allows execution of arbitrary code at compile time through three different mechanisms:

Projections
Code generators
Macros

All these instruments are entirely unlimited in their scope, so I don't see why we need to prohibit one way of executing code at compilation.

Furthermore ZPM itself uses Projections to install itself so closing this avenue to other projects seems strange.

Evgeny Shvarov · Dec 12, 2019

The answer to all this could be "To make the world the better place").

Because if you do all 3 you get:

the same wonderful Web terminal, but with simple, transparent, and standard installing mechanism with and yet another channel for distribution, cause ZPM seems to be a very handy and popular way to install/try the staff.

Maybe yet another channel of clear and handy app distribution is enough argument to change something in the application too?

I completely agree, and to get to

standard installing mechanism

for USERS, we need to zpm-enable as many existing projects as possible. To enable these projects we need to simplify the zpm-enabling, leveraging existing code if possible (or not preventing developers from leveraging the existing code). I think allowing developers to use already existing installers (whatever form they may take) would help with this goal.

This is very wise, thanks Ed!

For zpm-enabling we plan to add some boilerplate module.xml generator for the repo, stay tuned

Timothy Leavitt · Dec 12, 2019

Another option rather than having two versions of the whole codebase could be having a wrapper module around webterminal (i.e., another module that depends on webterminal) with hooks in webterminal to allow that wrapper to turn off projection-based installation-related features.

Sergei Shutov · Dec 16, 2019

Hi Nikita,

> A need to change how the self-update mechanism works or shut it down completely.
If a package is distributed via package manager, its self-update should be completely removed. It should be a responsibility of package manager to alert the user that new version of package is available and to install it.

> Thus, create an additional pipeline (or split the codebase) for 2 WebTerminal versions: ZPM's one and a regular one with all the tests and
so on.
Some package managers allow to apply patches to software before packaging it, but I don't think it's the case for ZPM at the moment. I believe you will need to do a separate build for ZPM/non-ZPM versions of your software. You can either apply some patches during the build, or refactor the software so that it can run without auto updater if it's not installed.

Evgeny Shvarov · Dec 20, 2019

Hi Nikita!

Do you want the ZPM exception for the Webterminal only or for all your InterSystems solutions? )

Thanks for the good question!

The answer is on why module.xml vs installer.cls on projections is quite obvious IMHO.

Compare module.xml and installer.cls which does the same thing.

Examining module.xml you can clearly say what the installation does and easily maintain/support it.

In this case, the package installs:

1. classes from WebTerminal package:

<Resource Name="WebTerminal.PKG" />

2. creates one REST Web app:

3. creates another REST Web app:

I cannot say the same for Installer.cls on projections - what does it do to my system?

Simplicity, transparency, and installation standard with zpm module.xml approach vs what?

From the pros/cons, it seems the objectives are:

Maintain compatibility with normal installation (without ZPM)
Make side effects from installation/uninstallation auditable by putting them in module.xml

I'd suggest as one approach to accomplish both objectives:

Suppress the projection side effects when running in a package manager installation/uninstallation context (either by checking $STACK or using some trickier under-the-hood things with singletons from the package manager - regardless, be sure to unit test this behavior!).
Add "Resource Processor" classes (specified in module.xml with Preload="true" and not included in normal WebTerminal XML exports used for non-ZPM installation) - that is, classes extending %ZPM.PackageManager.Developer.Processor.Abstract and overriding the appropriate methods - to handle your custom installation things. You can then use these in your module manifest, provided that such inversion of control still works without bootstrapping issues following changes made in https://github.com/intersystems-community/zpm.
- Generally-useful things like creating a %All namespace should probably be pushed back to zpm itself.