All
Accepted answers
Rob Tweed · May 18, 2021 go to post

Yes, I think that everyone who uses Cache or IRIS should take the time to discover what lies beneath! The power and flexibility of Global Storage is way beyond anything else out there - something that has been true ever since I first started working with such databases way back in the early 1980s. In all that time, I've never come across any other database architecture that is as quick and simple to grasp and yet as devastatingly powerful.  I'm hoping our efforts in putting together these resources will help a new generation of developers discover and harness that unique magic.

Rob Tweed · Feb 3, 2021 go to post

No disrespect to what you've created, but I have to say I'm always amazed and dismayed at how complex the Java community always seem to manage to make even the simplest of tasks - Spring/Hibernate is a classic example of making a crazy mountain out of what should technically be a very simple mole-hill.

By comparison, take a look at mgweb-server: https://github.com/robtweed/mgweb-server  and its underlying mg_web interface (https://github.com/chrisemunt/mg_web) for probably the thinnest (and therefore the most performant) and simplest way possible of delivering REST services using IRIS.  

Rob Tweed · Jan 20, 2021 go to post

Awaiting my app/repo to be approved and then I'll submit it to the competition...

Rob

Rob Tweed · Jan 18, 2021 go to post

We have two Open Source products that will look after JWTs for you in the ways you are asking about (ie REST services with IRIS):

- QEWD, if you want to implement everything at the back-end in Node.js / JavaScript

- mgweb-server if you want to use ObjectScript logic for your back-end logic

For QEWD and IRIS, see:

https://github.com/robtweed/qewd-starter-kit-iris-networked

In particular for REST services, see:

https://github.com/robtweed/qewd-starter-kit-iris-networked/blob/master…

and specifically this section:

https://github.com/robtweed/qewd-starter-kit-iris-networked/blob/master…

For mgweb-server, see:

https://github.com/robtweed/mgweb-server

specifically using with IRIS:

https://github.com/robtweed/mgweb-server/blob/master/IRIS.md

and within that document, this section on JWTs:

https://github.com/robtweed/mgweb-server/blob/master/IRIS.md#using-json…

Rob

Rob Tweed · Jan 1, 2021 go to post

if you're a JavaScript/Node.js developer, you can use the QEWD-JSdb abstractions of IRIS: 

- Document database

- Persistent JavaScript Objects

- Redis-like Lists

- Redis-like key/object store

- Persistent XML DOM (with XPath querying)

See: https://github.com/robtweed/qewd-starter-kit-iris-networked

Take the introductory tutorial:

https://github.com/robtweed/qewd-starter-kit-iris-networked/blob/master…

and then delve into the other database models:

- https://github.com/robtweed/qewd-starter-kit-iris-networked/blob/master…

https://github.com/robtweed/qewd-starter-kit-iris-networked/blob/master…

https://github.com/robtweed/qewd-starter-kit-iris-networked/blob/master…

A whole world of multi-model opportunities to explore for this competition!

Rob Tweed · Dec 3, 2020 go to post

If you're interested in trying out the mgweb-server Docker Container  (aka mg_web Server Appliance) with an IRIS back-end (eg using the IRIS Community Edition Docker Container), I've put together a detailed user guide and tutorial that takes you through the entire process, step by step, showing you how to create REST APIs, along with showing you how to get it working with the pre-built mgweb-conduit back-end Demonstrator/Example on IRIS, and how to add the wc-conduit front-end to the mg_web Server Appliance.  

It's all very quick and simple to get it up and working, with lots of utility functions included in the kit that make it very simple to create your REST APIs, use JWTs, secure passwords etc.  It's a different and alternative approach to using IRIS for REST APIs that you might find interesting.

Why not take a look at:

  https://github.com/robtweed/mgweb-server/blob/master/IRIS.md

Rob Tweed · Dec 1, 2020 go to post

Notice, by the way, the hefty performance penalty incurred by the SQL/Class option compared with the low-level ObjectScript functions ($extract and $translate).  

Rob Tweed · Nov 30, 2020 go to post

Results running IRIS Container on Raspberry Pi 4 (4Gb RAM):

$e() only   1.279893

$tr()       1.281999

$e()+$zd()  1.470656

SQL/class   66.405927

I didn't include the SQL/static in my test on the RPi.

Comparison with IRIS on a Windows 10 i7 based Intel NUC machine:

$e() only   .14742
$tr()       .185998
$e()+$zd()  .182579
SQL/class   11.305143

Mind you, the NUC cost about 15 times that of the RPi :-)

Anyway, based on a combination of the performance and coolness of technique, the $tr() one gets my vote!

Rob Tweed · Nov 30, 2020 go to post

Wow, how have I not known this trick after all these years? :-) 

Rob Tweed · Nov 27, 2020 go to post

..and, of course, you could now be running both the mgweb-server container (Apache + mg_web) and the IRIS container on Raspberry Pi 4's (with 0.5 million global node sets/second performance), to create a very cool, low-cost REST platform. 

BTW did you know that Raspberry Pi (the company) run their public web site on a cloud-based  farm of Raspberry Pi 4s? These are the guys that run it for them: https://www.mythic-beasts.com/blog/2020/06/17/raspberry-pi-4-now-availa…

Rob Tweed · Nov 27, 2020 go to post

I would add that it is worth remembering that you're not limited to using the technologies built-in to IRIS (or Cache for that matter).  There are also proven Open Source alternatives available for building web applications that use IRIS/Cache as a back-end environment (or simply an ultra-high-performance database).  So, if you want to develop applications in JavaScript/Node.js and use IRIS as a persistent JavaScript Object store, QEWD should be on your list.  Alternatively, check out mg_web and mgweb-server as an alternative to the built-in REST solution.  For more info see:

https://github.com/chrisemunt?tab=repositories

https://github.com/robtweed?tab=repositories

https://github.com/robtweed/qewd-conduit/blob/master/IRIS-windows.md

Rob Tweed · Nov 26, 2020 go to post

It's pretty fast two - I do have a pretty fast micro SD in the RPi4 but it looks like you can set 0.5m global nodes/sec in a very simple test:

USER>s s=$p($zts,",",2) f i=1:1:1000000 s ^rob(i)=$r(1000) i i=1000000 w $p($zts,",",2)-s
2.072394


Not bad for a $35 computer :-)

Rob Tweed · Nov 26, 2020 go to post

Bingo - nice find, Dmitry

Here we go on Ubuntu Server 20.04 on my RPi4:

ubuntu@ubuntu:~$ sudo docker run --name my-iris -it --rm -p 9091:1972 -p 9092:52772 -p 9093:52773 -p 9094:7041 store/intersystems/iris-community-arm64:2020.3.0.221.0
[WARN] No init process detected!  This container may accumulate zombie processes if run for a long time.  Consider using "docker create --init ..." or equivalent.
[INFO] Starting InterSystems IRIS instance IRIS...
[INFO] This copy of InterSystems IRIS has been licensed for use exclusively by:
InterSystems IRIS Community
Copyright (c) 1986-2020 by InterSystems Corporation
Any other use is a violation of your license agreement
Starting IRIS

11/26/20-11:02:27:145 (361) 0 [Generic.Event] Global buffer setting requires attention.  Auto-selected 25% of total memory.
11/26/20-11:02:27:436 (361) 0 [Generic.Event] Allocated 449MB shared memory: 227MB global buffers, 80MB routine buffers
11/26/20-11:02:27:588 (361) 0 [WriteDaemon.UsingWIJFile] Using WIJ file: /usr/irissys/mgr/IRIS.WIJ
11/26/20-11:02:27:588 (361) 0 [WriteDaemon.CreatingNewWIJ] Creating a new WIJ file
11/26/20-11:02:34:614 (361) 0 [WriteDaemon.CreatedNewWIJ] New WIJ file created
11/26/20-11:02:34:629 (361) 0 [Generic.Event]
Startup of InterSystems IRIS [IRIS for UNIX (Ubuntu Server LTS for ARM64 Containers) 2020.3 (Build 221U) Thu Oct 1 2020 15:11:45 UTC]
        in /usr/irissys/bin/
        with mgr: /usr/irissys/mgr
        with wij: /usr/irissys/mgr/IRIS.WIJ
        from: /usr/irissys/mgr/
  OS=[Linux], version=[#25-Ubuntu SMP PREEMPT Thu Oct 15 13:31:49 UTC 2020], release=[5.4.0-1022-raspi], machine=[aarch64]
  nodename=[b6deee189a6f].
  numasyncwijbuf: 8, swdwrtmax: 64, wijdirectio: on, synctype: 3
  System Initialized.
11/26/20-11:02:34:642 (362) 0 [WriteDaemon.Started] Write daemon started.
11/26/20-11:02:45:083 (373) 0 [Database.MountedRW] Mounted database /usr/irissys/mgr/ (SFN 0) read-write.
11/26/20-11:02:45:216 (373) 0 [Utility.Event] Instance 'IRIS' starting on node b6deee189a6f by user irisuser
11/26/20-11:02:45:216 (373) 0 [Utility.Event] Using parameters from file '/usr/irissys/iris.cpf'
11/26/20-11:02:45:221 (373) 0 [Utility.Event] Loading DLLs
11/26/20-11:03:31:951 (373) 0 [Database.MountedRO] Mounted database /usr/irissys/mgr/irislib/ (SFN 1) read-only. Database label is marked read-only.
11/26/20-11:03:32:147 (373) 0 [Utility.Event] Switching to temporary %SYS Namespace
11/26/20-11:03:32:340 (373) 0 [Utility.Event] Loading Locale enuw (English, United States, Unicode) from objects
11/26/20-11:03:34:020 (373) 0 [Database.MountedRW] Mounted database /usr/irissys/mgr/iristemp/ (SFN 2) read-write.
11/26/20-11:03:34:073 (373) 0 [Utility.Event] /usr/irissys/mgr/iristemp/ initialized as IRISTEMP
11/26/20-11:03:34:080 (373) 0 [Utility.Event] Switching to default %SYS Namespace
11/26/20-11:03:34:199 (390) 0 [Utility.Event] Log Monitor Started
11/26/20-11:03:34:334 (391) 0 [Utility.Event] Clean Daemon Started
11/26/20-11:03:34:382 (373) 1 [Utility.Event] Configuration file /usr/irissys/iris.cpf is not the same as when last shut down
11/26/20-11:03:35:657 (373) 0 [Utility.Event] Updating configuration information from /usr/irissys/iris.cpf
11/26/20-11:03:35:759 (373) 0 [Utility.Event] Performing Journal Recovery
11/26/20-11:03:35:764 (373) 0 [Utility.Event] Graceful system shutdown, journal restore not required
11/26/20-11:03:35:765 (373) 0 [Utility.Event] Graceful system shutdown, transaction rollback not required
11/26/20-11:03:35:790 (373) 0 [Utility.Event] START: /usr/irissys/mgr/journal/20201126.001
11/26/20-11:03:35:853 (373) 0 [Generic.Event] INTERSYSTEMS IRIS JOURNALING SYSTEM MESSAGE
Journaling started to: /usr/irissys/mgr/journal/20201126.001
11/26/20-11:03:35:855 (373) 0 [Utility.Event] Journaling to /usr/irissys/mgr/journal/20201126.001 started.
11/26/20-11:03:35:979 (373) 0 [Utility.Event] Processing Startup section
11/26/20-11:03:36:001 (392) 0 [Utility.Event] Purging old application errors
11/26/20-11:03:36:246 (373) 0 [Utility.Event] Initializing Security system
11/26/20-11:03:37:325 (392) 0 [Database.MountedRW] Mounted database /usr/irissys/mgr/irislocaldata/ (SFN 3) read-write.
11/26/20-11:03:37:495 (373) 0 [Utility.Event] Processing Network section
11/26/20-11:03:37:502 (373) 0 [Utility.Event] Activating Network
11/26/20-11:03:37:756 (373) 0 [Utility.Event] Processing Databases section
11/26/20-11:03:37:959 (373) 0 [Database.MountedRW] Mounted database /usr/irissys/mgr/irisaudit/ (SFN 4) read-write.
11/26/20-11:03:37:978 (373) 0 [Utility.Event] Processing Namespaces section
11/26/20-11:03:37:979 (373) 0 [Utility.Event] Namespaces are up to date
11/26/20-11:03:37:979 (373) 0 [Utility.Event] Activating namespaces
11/26/20-11:03:37:991 (373) 0 [Utility.Event] Activating new namespace map
11/26/20-11:03:38:157 (373) 0 [Utility.Event] Namespace changes have been activated
11/26/20-11:03:38:363 (373) 0 [Utility.Event] Starting Super Server on port 1972
11/26/20-11:03:38:406 (373) 0 [Utility.Event] Network Lock Upload Phase Starting
11/26/20-11:03:38:410 (373) 0 [Utility.Event] Lock Upload Phase Complete
11/26/20-11:03:38:410 (373) 0 [Utility.Event] Processing Miscellaneous section
11/26/20-11:03:38:632 (397) 0 [Utility.Event] LMF Info:  Licensed for 5 users.
11/26/20-11:03:38:644 (398) 0 [Utility.Event] Starting Servers
11/26/20-11:03:38:644 (373) 0 [Generic.Event] init_gcr_seed:  gen_crypt_rand seeded from /dev/urandom:  64 bytes.
11/26/20-11:03:38:644 (373) 0 [Utility.Event] Processing Devices section
11/26/20-11:03:38:663 (373) 0 [Utility.Event] Processing DeviceSubTypes section
11/26/20-11:03:38:664 (373) 0 [Utility.Event] Processing MagTape section
11/26/20-11:03:38:684 (373) 0 [Utility.Event] Processing IO section
11/26/20-11:03:38:691 (373) 0 [Utility.Event] Processing SQL section
11/26/20-11:03:38:806 (373) 0 [Generic.Event] Auditing to /usr/irissys/mgr/irisaudit/
11/26/20-11:03:38:978 (373) 0 [Utility.Event] Executing ^ZSTU routine
11/26/20-11:03:38:984 (373) 0 [Utility.Event] Executing ^%ZSTART routine
11/26/20-11:03:38:985 (373) 0 [Utility.Event] Enabling logons
11/26/20-11:03:40:649 (398) 0 [Utility.Event] Private webserver started on 52773
11/26/20-11:03:40:650 (398) 0 [Utility.Event] Processing Shadows section (this system as shadow)
11/26/20-11:03:40:806 (373) 0 [Database.MountedRW] Mounted database /usr/irissys/mgr/user/ (SFN 5) read-write.
11/26/20-11:03:40:812 (398) 0 [Utility.Event] Processing Monitor section
11/26/20-11:03:41:242 (547) 0 [Utility.Event] Starting TASKMGR
11/26/20-11:03:41:404 (548) 0 [Utility.Event] [SYSTEM MONITOR] System Monitor started in %SYS
11/26/20-11:03:41:451 (398) 0 [Utility.Event] Shard license: 0
11/26/20-11:04:00:517 (373) 0 [Database.MountedRO] Mounted database /usr/irissys/mgr/enslib/ (SFN 6) read-only. Database label is marked read-only.
11/26/20-11:04:00:527 (373) 0 [Utility.Event] Initializing Interoperability during system startup
[INFO] ...started InterSystems IRIS instance IRIS

and then in another process I can shell in and run

iris terminal IRIS

and access the IRIS shell and database!

Amazing!

Rob Tweed · Nov 26, 2020 go to post

Actually I think that might be the answer - uname -m on the RPi4 I've been using says armv7l which means 32 bit - it should say aarch64.  The version of Raspbian is pretty old on that particular RPi4 and I'm not sure if they have now put out a 64-bit version of Raspbian?

Now I have another RPi4 (I did say I had loads of them, right!) running Ubuntu Server 20.04 which I believe is a 64 bit version - I'll give it a try on that one and report back

Rob

Rob Tweed · Nov 26, 2020 go to post

The latest model RPi 4 running the standard Raspian OS.  I have other Docker containers running fine on it - eg my rtweed/qewd-server-rpi and rtweed/mgweb-rpi ones (available on Docker Hub)

I've not tried the 3, but really there shouldn't be any difference as far as Docker is concerned - all my RPi containers work fine on the 3 and 4.

Rob Tweed · Nov 26, 2020 go to post

Dmitry

What I see on the Raspberry Pi is this if I try to start it (in foreground mode so I can see the error):

$ docker run --name my-iris -it --rm -p 9091:1972 -p 9092:52772 -p 9093:52773 -p 9094:7041 store/intersystems/iris-community-arm64:2020.3.0.221.0
standard_init_linux.go:211: exec user process caused "exec format error"

Rob Tweed · Nov 26, 2020 go to post

Thanks Vic. Yes that was my conclusion too - it seems to be for AWS EC2 ARM Graviton processors.  The question is whether it ought to work on other ARM chips too? 

As far as ARM support for IRIS is concerned, Raspberry Pi would be really handy to have as it's a very low cost but extremely potent platform, and out there in huge numbers (eg you should see how many I have! :-).  I know some work on an RPi/ARM port was done a while ago by an ISC intern but don't know what, if anything, became of it.

Apple M1 would be the other obvious ARM platform of course :-)

Rob Tweed · Oct 29, 2020 go to post

A quick update:  

I've modified our WebComponent-based wc-conduit RealWorld Client to optionally use the alternative WebSocket API interfaces supported by the qewd-conduit back-end. 

Furthermore, to make it easy for you to try out the wc-conduit front end and compare the REST and WebSocket APIs,  a copy of the wc-conduit front-end is integrated into the qewd-conduit repository.  I've updated the documentation to explain how to use the WebSocket version of wc-conduit  if you want to try it out.  It's very simple - just load a different URL to launch the wc-conduit front-end and it will automatically do the rest (no pun intended!).

  http:xx.xx.xx.xx:8080/conduit-wc  (REST)

  http:xx.xx.xx.xx:8080/conduit-wc/index-ws.html  (WebSockets)

See if you notice any performance difference between the WebSocket and REST APIs.  The back-end literally runs the same handler code regardless of the interface you choose, and the front-end WebComponents operate identically, so any difference in performance you see is network overheads of HTTP versus WebSockets, and the back-end interface overheads of HTTP (handled by the Node.js Express module) versus WebSockets (handled by the Node.js socket.io module).

REST or WebSocket APIs with an IRIS back-end. What's not to like?

Rob Tweed · Oct 21, 2020 go to post

So, step 2 along the road for qewd-conduit: this time focusing on the front-end.  If you've noticed, most of the available front-ends for the RealWorld Conduit application are a real pain to install and configure.  So, courtesy of the WebComponents and Module support that is now built-into modern browsers, I've created a version using my mg-webComponents framework.  No bundled JavaScript, no heavy-weight frameworks like React, Angular, Vue etc.  And a very simple configuration allowing you to change the Conduit back-end location in a matter of seconds.

Read all about it and try it out here:

https://github.com/robtweed/wc-conduit

Let me know if you have any issues / questions / comments etc.

Rob Tweed · Oct 11, 2020 go to post

I've updated the repository on Github to include files for IRIS running on Mac OS X.  Here's how to get it working:

- replace the /package.json file with /package.json.iris-osx (ie rename it to package.json)

- replace the /configuration/config.json file with /configuration/config.json.iris.osx

- Edit the /configuration/config.json file to ensure the database connection parameters are correct for your IRIS system.  In particular you'll need to change the database.path property

Then just follow the same instructions as provided for Windows in the README.

HOWEVER!!  IMPORTANT:  When you start QEWD for the second (or subsequent) time, you need to have root privileges, so you need to do this:

  sudo npm start

This is due to the root permissions needed to connect to IRIS's C interface

It should then work according to the instructions I've provided for Windows.

Rob Tweed · Oct 9, 2020 go to post

I'd need information on how you're supposed to customise the IRIS Docker container.  From past experience it's a pretty complex process, but perhaps that's changed?  To summarise what's needed for qewd-conduit, I'd need to be able to install Node.js on the running container, install a C++ compiler to its Linux environment, pull in the qewd-conduit repo files and modify the package.json and config.json files, then invoke npm install and npm start.  The listener port used by the QEWD process would also need to be mapped via the Docker container's run command -p directive. 

Rob Tweed · Oct 9, 2020 go to post

For these to be "bonuses", they imply that there is something better about using them than doing the equivalent in other ways and using other technologies.  They therefore rig the competition to favour solutions that use what's built-in to IRIS, and penalise systems that still use IRIS but use other technologies for achieving the same results just as effectively.  So, for example, since qewd-conduit's logic is all in Node.js and not inside the IRIS programming environment, things like ZPM are not relevant, but I'm penalised for not using it.  Ditto the REST API within the IRIS programming environment.

And no, QEWD doesn't use the Native API for Node.js which is provided within IRIS, but uses the Open Source mg-dbx interface and the APIs it provides, so I guess that means qewd-conduit will be further penalised and lose that bonus too! :-)

I suppose it comes down to what is trying to be demonstrated and promoted in the competition, but it doesn't seem to me to be a level playing field for demonstrating what's possible in terms of a full-stack application that makes use of IRIS for data storage.

Rob Tweed · Oct 9, 2020 go to post

qewd-conduit could easily be set up to run in the Dockerised version of IRIS.  For simplicity, the version I shipped assume you'd be running on a Windows machine on which IRIS was installed - the commonest set-up out there in my experience.  

It's somewhat mean to deny qewd-conduit a bonus for not using IRIS's built-in REST APIs - it clearly implements REST APIs against an IRIS system, just not using the proprietary IRIS-specific APIs.  I'm not sure why their use warrants a bonus if REST APIs can be implemented just as effectively by other means.

Rob Tweed · Oct 9, 2020 go to post

QEWD-Conduit is a QEWD application, which means it's a Node.js application - 100% JavaScript.  As a QEWD application, it means it uses the mg-dbx connector to attach to an instance of IRIS or Cache.  For simplicity and to make it easy and convenient for readers to try out, the version I've made available for the competition assumes it's going to be run on a Windows machine on which IRIS is installed - that's the most common set up out there.

However, as a Node.js application, qewd-conduit is capable of running on any machine on which Node.js can be installed and run - so that means Linux, Unix, Mac OS and Windows.  Since the Dockerised version of IRIS is based on a Linux OS, then yes, of course, qewd-conduit could be installed and run on it, eg by modifying the Dockerfiles setup, or simply shelling into the running container, installing Node.js, copying in the qewd-conduit repo files, running npm install and npm start.

The key to modifying how/where qewd-conduit runs is just 2 configuration files:

- the Node.js-specific package.json file.  For Linux, Unix and MacOS systems, add "mg-dbx" as a dependency and make sure that the OS has a C++ compiler installed, since mg-dbx needs to be built from source during the npm install.  For Windows, leave mg-dbx out as a dependency - QEWD will automatically download and install a pre-built Windows binary of mg-dbx when it is first run.

- the QEWD-specific /configuration/config.json file which is used by QEWD to configure itself.  In particular, the "database" section defines how the connection is to be made to IRIS or Cache, so that's where you define the path, namespace etc as appropriate for the instance of IRIS you want to connect.

Note that the config.json file I've shipped assumes that you'll run qewd-conduit on the same physical machine as your instance of IRIS.  However, you can change the settings to tell mg-dbx to connect to a separate IRIS (or Cache) instance via a network connection.

Just explore the QEWD and mg-dbx documentation and you'll discover how to run qewd-conduit in any configuration you like on anything you like.  It's a shame there's no publicly-available version of IRIS for a Raspberry Pi, because you could run qewd-conduit there too.  You *could*, of course, run qewd-conduit on a Raspberry Pi and change the config.json to make a network connection to an IRIS system on a separate machine.

QEWD really does provide the ultimate in flexibility for implementing IRIS/Cache REST services

Rob Tweed · Oct 6, 2020 go to post

Yes, it would be possible to do so - it would be a simple matter of writing a process that invoked the appropriate Conduit REST APIs to create the articles, comments and links to authors/users from your data export.  The one thing that might be tricky would be the user password property, which would be required for a user to log in and view/edit their own articles, and to follow other users and/or favourite their articles.