Discussion
· Sep 22, 2022

Testing MergeCPF feature

For quite some time InterSystems IRIS supports such thing as Merging CPF. So, with help of this it should be possible to define only desired changes in configuration. And get them applied even with vanilla Docker image. 

And I though it could be useful when used with Dockerfile. Use this way to configure IRIS during docker build instead of using Installer manifest.

Format of the file, is the same as original IRIS.cpf, with addition of one block, with Actions, where you can define a list of actions which will be executed during merging. And this is most interesting part. It can help to create Database file, users, roles, services and so on.

Create a new Database and Namespace, is quite simple, this lines will help with it.

[Actions]
CreateDatabase:Name=TEST,Directory=test
CreateNamespace:Name=TEST,Globals=TEST

In most cases, it uses Config or Security classes, and you may define as many parameters as you need, and for the list of all available parameters you have to look at the corresponding class.

How to call the merge

There is new merge sub-command added to iris command. So, you can do Merge CPF on a running instance.

$ iris merge IRIS merge.cpf

IRIS Merge completed successfully

And it is possible to do this merge during the start of IRIS Docker Image, in this case, the path to merge.cpf have to be in the environment variable ISC_CPF_MERGE_FILE

$ docker run -e ISC_CPF_MERGE_FILE=/dur/merge.cpf containers.intersystems.com/intersystems/iris

Caveats

Unfortunately, I found some issues, which I would like to share here, so, you would know what to expect if you decide to use it

  • IRIS Merge completed successfully, it's the only result you will see in the output. And in fact, it does not mean, that anything was even done at all. Some possible situations
    • If you pass, a non-existing path, it's a success
    • It does not accept a relative path, only a full absolute path
    • Type in [Actions], it's case-sensitive, if you have [actions], nothing changes and it's a success, with no errors
    • All good? Yes, if namespace already exists, no changes, and it's a success
      [Actions]
      CreateDatabase:Name=TEST,Directory=test
      CreateNamespace:Name=USER,Globals=TEST
  • Error Code
    • Success = 3; In what world success code became 3, I have no idea, when always expected to be 0
    • Error = 7; any non-0 is usually interpreted as an error, so, it's ok at least, in some scenarios it may return it, but difficult to find that scenario
  • No output at all, if no errors are found

In the documentation, you may find this example

[Actions]
CreateUser:Name=SQLAdmin,
  PasswordHash="cec6638a357e7586fddfb15c0e7dd5719a1964e774cd37466fb0c49c05,
  323cb89148c887166dd2be61c107710539af2c01b43f07dccc8d030ac2c1a8cf7c5ace4a00d57e3780f,10000,SHA512",
  Roles="%SQL,%DB_USER"
GrantAdminPrivilege:Grantee=SQLAdmin,Namespace=USER,AdminPriv="%DB_OBJECT_DEFINITION,%BUILD_INDEX"

Keep, in mind to remove all new lines in your file, because written this way, it will fail with errors

ERROR #501: Error reading [Actions] section in file /home/irisowner/temp/merge.cpf
ERROR #508: Invalid action 'PasswordHash="cec6638a357e7586fddfb15c0e7dd5719a1964e774cd37466fb0c49c05,' at line 3
ERROR #508: Invalid action '323cb89148c887166dd2be61c107710539af2c01b43f07dccc8d030ac2c1a8cf7c5ace4a00d57e3780f,10000,SHA512",' at line 4
ERROR #508: Invalid action 'Roles="%SQL,%DB_USER"' at line 5
IRIS Merge failed%   

Note: How to get PasswordHash for a desired plain password, is not so easy to find, it is in a different section of the documentation, and only possible with special Docker images just for it, by some unknown reasons this tool you will not find in IRIS Kit.

Conclusion

I know, that I may not test all possible combinations, but even with these results, I see no reasons for recommending using this feature at all. The behavior is very unpredictable, even for development, you would have to manually check if any changes were applied correctly. 

Discussion (8)3
Log in or sign up to continue

Hi Dmitry,

Thanks for trying out the CPF merge utility. I am sorry to hear you found it not very useful because in many areas, especially cloud compositions we use it daily, it's of declarative nature and just aids in just about anything we do with containers.

The utility was created to be indempotent so if you have a database or namespace and you run it again it won't create new resources so, yes, the process run successfully it just did not bother you telling you you made a mistake by submitting the same request twice.

For the password use the following container:

containers.intersystems.com/intersystems/passwordhash:1.1

HTH

When I use docker in any project, I expect that any Docker build, will do the job as I expected, or fail-fast. We already have a bunch of templates with docker and zpm. And we have lines like this in Dockerfile

RUN --mount=type=bind,src=.,dst=. \
    iris start IRIS && \
	iris session IRIS < iris.script && \
    iris stop IRIS quietly

So, I expected that I would just add merge there, so I could get what I want

RUN --mount=type=bind,src=.,dst=. \
    iris start IRIS && \
    iris merge IRIS ./merge.cpf && \
	iris session IRIS < iris.script && \
    iris stop IRIS quietly

And what I found, is that even though I use start, session, and stop from the same iris command, always had no issues, and suddenly, merge failed my build, even though the output says, it's a success. I did not expect that the error code would be just a random magic number, while the whole Linux world uses 0 as a success, and even Windows uses 0 as success, but for unknown reasons, InterSystems uses 3 as success in this case, and did not think it's kind of important and worth to at least mention in the Documentation. 

So, I have no idea how you debug the issues, there, probably you can type with no typos, you can't do any mistakes, but people do mistakes, and it should cover these situations, and indicate that you doing something unexpected. 

As soon as iris merge does not return 0, does not have any verbose output, and ignores issues, I strictly do not recommend this in development and in production for sure at all, to anyone. 

Hi Dmitry,

Again, thanks for pointing things out. Let me comment on a couple of things as you so strongly say that you do not recommend the facility provided.

The first thing is that the %Installer although it still work it is out of sync with internal classes. It is also cumbersome to feed it through in a declarative way to an IRIS instance. The hope with the CPF merge and its actions is to provide a more modern tool that fits with modern, declarative, gitops paradigms. The third is that containers forces us to think about the distinction between build and run phases and the CPF merge is definitely into the injection of configuration at the run phase. To that point, the actual ```iris merge``` was added later for organizations working with VMs that still want to supply a CPF Merge so that they can more adequately adhere to a declarative approach with tools like Ansible, CloudFormation etc. As I said, we use it daily in the cloud to configure just about anything we need. I know we can now go down a rabbit hole and chase my-use case vs yours so I shall leave it at it and I hope you can understand the intention.

I think we can probably fix the error code returned that comes from within IRIS just like anything else we call as we perform the merge.

It would be good to hear other feedback and what other features we should implement and we hope to hear from a wider audience too.

Right %Installer is still in place, and we have it in some of the templates.

So, what would I like to see in with iris merge

  • error code 0, for success
  • --dry-run flag, to have an ability to somehow validate the merge process, what will happen if I do merge
  • --verbose|-v flag, or by default, and --no-verbose to get the current behavior. Verbose output should show every single change made, similar to %Installer output, including plain changes not covered by [Actions]
  • --fail-fast by default, fail on any error that happens, CreateNamespace on existing namespace should fail, and so on.
  • --ignore-errors for the current behavior, when some issues are ignored completely. But it still has to log in verbose output.

@Dmitry Maslennikov  - I agree with all of your points above except for one:

"CreateNamespace on existing namespace should fail, and so on."  

Per Luca's original statement - "The utility was created to be idempotent" ... this is very important to us - we need to make sure that the same command can be run multiple times and if it is creating new changes to the config that it does so, but if it being asked to create what is already there it will ensure that it is already there and will return success (https://www.infoworld.com/article/3263724/idempotence-and-the-discipline...).  With this design philosophy in mind, CreateNamespace on existing namespace should succeed and not fail (but of course, verbose output should make it clear that CreateNamespace was a no-op due to the previously existing namespace).  

An idempotent operation can be performed multiple times without changing the result beyond the initial application. Put more simply, an operation is idempotent if you can repeat it over and over again without causing any unwanted side effect or harm to the system, always producing the same result.

I agree with these, and, that's not what actually happens with MergeCPF.

We have two different options

  • CreateNamespace
  • ModifyNamespace

But, to make MergeCPF truly follow idempotent, it should be only one like CreateOrModifyNamespace or to make it declarative way just Namespace. Where in this case, I will get what I defined in any case

I have this file

[Actions]
CreateDatabase:Name=DEMO,Directory=demo
ModifyNamespace:Name=DEMO,Globals=DEMO

I have no NameSpace Demo, and after merge, I will get the new Database, But No Namespace at all. And no failure, merge said all good.

Yeah, probably Namespace not a good example, not so variants to change it on the fly, but what about User, I would probably want to change password with the next deploy, or changes Roles. Or CSP Application , and so on. 

In the current state of work, the only correct way is to always define Create and Modify, so, it will always create the desired resources, and Modify, if I would need to update already created previously. This does not look like an obvious way to go, at all.