You should take a look at this alternative Open Source Node.js client for IRIS: mg_dbx:
https://github.com/chrisemunt/mg-dbx
It provides significantly better performance.
To use its synchronous APIs safely you need to look at either:
- QEWD: https://github.com/robtweed/qewd
- qoper8-fastify: https://github.com/robtweed/qoper8-fastify/blob/master/IRIS.md
Well it only took 13 years for the idea to be taken up.
An observation: the administrators of this forum regularly report the size of this community, currently an impressive number in excess of 12,000 developers, yet, as can be seen from the OP, numbers of views of even the most popular posts is tiny by comparison. From what I've seen, a typical post will get around about 50 views, and each comment will add a further 50 - a meagre 0.4% of the total community. This leads me to believe that there is actually only a small core (~50?) of regular contributors and readers (many of whom, I suspect, are actually ISC employees)
It does make me wonder what, if anything, most of those 12,000 Cache/IRIS developers read, and what site(s) they use instead. Perhaps they don't read anything?
My reason for raising this is that, for a third-party developer of solutions for these technologies, reaching and informing those 12,000 developers is a significant issue and problem, and it's fairly clear to me that this forum currently doesn't provide a very effective solution to that problem.
My question/challenge for 2023 is therefore what do folks here (the regular 50?) think can be done to provide a more effective way to reach that reported developer base?
Use the tried and tested mg_python:
If the applications you're providing to users are REST/web/browser-based, you can make use of connection/process pooling, eg via an implementation in another back-end language, to avoid/minimise these issues. A good example is the Node.js/JavaScript-based QEWD framework or its recent evolution - qoper8-fastify - I've described the latter in another recent posting here on Node.js.
Another alternative is to use our mg_web interface (you may want to check out our mgweb-server framework which packages up mg_web for ObjectScript-based REST APIs to save you a lot of effort): this uses modern web servers (eg Apache/NGINX) to manage connections to IRIS in a parsimonious way.
Some holiday reading:
QEWD: https://github.com/robtweed/qewd
QOper8-Fastify: https://github.com/robtweed/qoper8-fastify
Using QOper8-Fastify with IRIS: https://github.com/robtweed/qoper8-fastify/blob/master/IRIS.md
mg_web: https://github.com/chrisemunt/mg_web
mgweb-server: https://github.com/robtweed/mgweb-server
Take a look at our other language interfaces and front and back-end solutions too - see https://www.mgateway.com
Well, if it was me, I'd keep that and the rest of my logic in JavaScript/Node.js and access the Cache database using the mg-dbx interface.
... well they don't need to develop their own. They could actively support and work with third party experts in this area and simply make available suitable public interfaces.
I should point out that if you use the in-process API connection to IRIS from mg-dbx, you don't need to install anything on IRIS - just enable the call-in interface and away you go. You do, of course, need Node.js running on the same physical hardware as IRIS for such a connection, but it's significantly better performance than a networked connection.
Ideally yes, I would agree, but you're up against the publicly available interfaces for IRIS. What would you suggest?
Well of course you can use the mg-dbx interface for IRIS which is available on NPM and which provides equivalent functionality to the Native API interface.
https://github.com/chrisemunt/mg-dbx
Otherwise, in terms of complexity of getting IRIS working with Node.js for multi-user environments, the qoper8-fastify approach I've described makes it about as simple and easy as you can get, and all you need to focus on is the logic for each REST request handler and how it interacts with and uses your IRIS data.
Use glsdb's even higher-level abstraction and it becomes even simpler for a JS/Node.js developer
https://github.com/robtweed/glsdb
So the solutions are all there. It's the means to communicate at appropriate scale that's missing. I've tried relentlessly over recent years, but a single lone voice doesn't get things very far, sadly.
I wonder how many people are actually using Node.js with IRIS? My hunch is, sadly, not many, yet JavaScript is a really great modern alternative language to ObjectScript, and IRIS is potentially a really natural database for JavaScript/Node.js deveopers. How to get IRIS developers to check out and adopt Node.js, and how to get Node.js developers to check out and adopt IRIS? It's an incredibly potent combination for anyone willing to try!
Hi Ward - I too have wondered about the built-in IRIS Native API for Node.js. It does, indeed, appear to be synchronous which is good in that it makes it easy to use and understand, but very bad for the reasons you describe that relate to Node.js concurrency, and effectively unusable as it stands for production multi-user settings.
Conversely, I know from experience that asynchronous versions of the APIs can tie you in horrible knots, and if you need to use locks or transactions, the fact that every concurrent user is sharing the same physical thread of execution makes them pretty much impossible to use. So having async versions of the APIs isn't really the solution either. So does this mean the Node.js API is effectively unusable? No - there's a solution!
First some quick background: I've grappled with and specialised in Node.js integration since the earliest days of Node.js (all the way back to 2010/11!), and long ago concluded that the best solution was a queue/worker architecture, whereby access to IRIS could be limited to Worker processes, within which only a single task would execute in isolation. Remove the concurrency issues of Node.js and you'd have an ideal scenario of being able to use synchronous APIs to access a database such as IRIS without any downsides.
I even wrote an article about this:
https://medium.com/the-node-js-collection/having-your-node-js-cake-and-…
That article describes my solution in the context of our QEWD.js framework - now, to be fair, QEWD is a somewhat monolithic, all-or-nothing framework/solution which locks you into certain key building-blocks (eg Express as the web framework, mg-dbx for the Node.js APIs to IRIS etc) and particular ways of working.
Recently, however, as a spin-off from other stuff I've been working on, I decided it was time to revisit the core ideas of QEWD and not only modernise its logic to use the latest JavaScript constructs and features where relevant, but also break it down into a set of "building blocks" which would allow people to customise how they use and assemble them - a "buffet menu" instead of a "set meal" as it were.
One such building block is the queue/worker piece, now named QOper8: actually there are two variants - QOper8-cp for Child Process Worker and QOper8-wt for Node.js Worker Thread Workers. Both work identically and have almost identical APIs and configuration. You can read about them here:
https://github.com/robtweed/qoper8-cp
https://github.com/robtweed/qoper8-wt
So these provide a potential queue/worker solution for making the IRIS Native API for Node.js acceptable and workable in a multi-user Node.js environment, but how, you might ask, would you make use of them in practice? What, for example, would be a likely use-case?
Well, the most common scenario for using Node.js in such a multi-user environment is as the back-end of an interactive web application or a REST-based web service. Node.js excels at this: it's exceptionally fast, highly scalable, lightweight in terms of hardware resource demands. A real benefit is you have a single language - JavaScript - describing both the front-end and the back-end logic.
Ask any Node.js practictioner these days what web framework they use and they'll pretty much all now say the same thing - something called Fastify, which, as its name implies, provides an extremely fast web server platform. Furthermore, Fastify has been designed to be easily extensible via a standardised plug-in architecture.
I've therefore used this to create a Fastify plug-in for my QOper8 modules, which, as a result, makes it extremely quick and simple to build out a Fastify/QOper8 back-end whereby incoming HTTP/REST requests can be automatically forwarded to QOper8 which queues and dispatches each request to be handled in isolation within the first available Worker process. You can read about this plug-in, known as qoper8-fastify, here:
https://github.com/robtweed/qoper8-fastify
So then take this one step further: initialise each QOper8 Worker Process by loading the IRIS Native API for Node.js, connecting to your IRIS system and making the APIs available for your handler methods, and you have a high-performance, highly scalable back-end with direct access to your IRIS database, and allowing all your logic to be written in JavaScript (actually you can even use the Native API to invoke ObjectScript functions and procedures on your IRIS system), and safely using the synchronous IRIS Native API for Node.js in a production multi-user environment.
If you want to find out more about how you can implement such a solution, I've written it all up, complete with a simple worked example that you can easily extend to build out your own web/REST back-end solution:
https://github.com/robtweed/qoper8-fastify/blob/master/IRIS.md
Enjoy, and do provide me with feedback!
... which makes me wonder about a Python equivalent of glsdb. I notice that Python includes Proxy Objects which I assume work similarly to JavaScript's Proxy. Our mg_python interface provides all the same IRIS APIs as mg-dbx. I'm not a Python developer, so wouldn't be the right person to implement such a beast, but all the logic that's needed is in my glsdb Github repo for an appropriately-skilled Python developer to translate from JavaScript. All the features of embedded Python but using standard Python :-)
One point I should mention - this kind of direct abstraction of IRIS objects onto another language's in-memory data structures (JSON/JavaScript Objects in this case) would normally have been thought to require the language to be embedded into the IRIS (or other database) environment (and therefore requiring implementation by the vendor). glsdb demonstrates that it can actually be done without such embedding - it's all done using standard Node.js language features, and can even be achieved over a networked connection between Node.js and IRIS. And with mg-dbx providing the interface, performance isn't sacrificed either.
Yes there is if you look (I do need to add an index to the Readme):
https://github.com/robtweed/glsdb#the-iriscache-specific-api
If you mean a worked example? No - not yet but I'll add a simple one in due course.
No, as the documentation says, we use our own mg-dbx interface and its APIs:
https://github.com/chrisemunt/mg-dbx
This is designed to get the very highest performance through the Node.js API (which otherwise has serious bottlenecks). It's also compatible with most earlier versions of IRIS and Cache and with Node.js v14 - 18, as well as aligned with our other similar interfaces for use with other technologies.
mg-dbx provides all the APIs needed to access IRIS classes:
Thanks for pointing that out, John. I've changed them :-)
For anyone wanting to learn more about globals as a database storage technology, and some information on how to use them to model various kinds of database structures, I've created this repository of documents just for you! Hopefully it will provide some inspiration towards this competition.
As it happens, if you use the Node.js-based QEWD framework, any time you use the QEWD-JSdb APIs that modify a global node (eg set, kill, increment etc), an event is triggered that you can write your own customer handler for.
It's described in this set of training slides: https://www.slideshare.net/robtweed/ewd-3-training-course-part-26-event…
Glad you liked the articles, Ken. How IRIS (and Cache) handles the physical side of global storage is, of course, proprietary, but, as the article referred to by Alexander explains, it's done using a fairly classic b-tree architecture. For performance, access to the physical database is buffered via memory, the amount of which you can configure. Additionally, IRIS and Cache both have ECP networking which adds an amazing level of additional power and flexibility with Globals able to be transparently abstracted across networked machines - needless to say the technicalities of ECP are a closely-guarded proprietary secret!
As my articles explain, however, you can actually implement the basics of global storage on top of a number of other different databases, with BerkeleyDB being probably the closest example to how IRIS and Cache implement them.
Of course, for the average user, how the concept/abstraction of Global Storage is physically implemented is of less interest than how you can harness and make use of Global Storage to do the kinds of things you want to do. That, of course, was the focus of my articles, to show just some of the most common ways (and some of the lesser-known and very sophisticated ways) in which you can harness Global Storage.
I've sometimes described Global Storage as a "proto-database" - a very simple but powerful and flexible database engine on which you can model pretty much anything else and on which you can layer all the other stuff you need to create your specific database environment. As such, it's unique in the database marketplace, and it has always baffled me over the years why it's so little known about and used: it blows away everything else out there.
Anyway, a Happy New Year to all fans of IRIS and Global Storage!
Rob
If you're wanting to use IRIS with Node.js, you might want to also look at QEWD. Rather than using the IRIS Native Node.js APIs, it uses the alternative Open Source mg_dbx interface and its APIs (https://github.com/chrisemunt/mg-dbx), and QEWD creates an architecture that allows the synchronous APIs to be used safely (via a queue/dispatch architecture which gives QEWD its name). With QEWD and the mg-dbx interface, you can run Node.js on either the same machine/system as IRIS, or on separate machines/systems via a networked connection, and you can run QEWD either as a native installation or you can use a pre-built/configured Dockerised version, so it's extremely flexible.
QEWD can be used to develop both Node.js REST services and interactive web applications, the latter connecting users' browsers and your back-end over WebSockets. QEWD does all the tricky technical work for you, leaving you to just focus on your application or API functionality.
Here's a few ways to get started with QEWD:
Native IRIS/QEWD: https://github.com/robtweed/qewd-baseline/blob/master/IRIS-WINDOWS.md
Using Dockerised IRIS: https://github.com/robtweed/qewd-baseline/blob/master/IRIS.md
Networked connection between QEWD/Node.js and IRIS: https://github.com/robtweed/qewd-starter-kit-iris-networked
Although it allows use of IRIS via its standard Object and SQL architecture, and APIs QEWD can also abstract the IRIS database as persistent JavaScript Objects/JSON, which allows you to access and use an IRIS database in a completely unique and powerful way that makes it a natural fit with JavaScript. To explore this aspect of QEWD with IRIS, check out:
https://github.com/robtweed/qewd-jsdb-kit-iris
These repositories and the documentation/examples/tutorials they contain should get you started
Rob
Calling any interested Python developers: It would be interesting for someone to take a look at what I've done in JavaScript/Node.js to implement what I refer to as persistent JSON objects, essentially creating JavaScript objects/JSON that resides on disk rather than memory and which can also be manipulated, traversed and modified in-situ rather than being shuttled between disk and memory. It's a concept I've referred to as QEWD-JSdb. Given the native support for JSON that now exists in Python, it strikes me that the equivalent ought to be possible using the new embedded Python in IRIS. I'm not a Python developer so I'm not the person to implement it, but all the logic for implementing this concept is available in my JavaScript repositories and my assumption is that it should be a matter of recasting the logic in Python. The really cool part is the multi-model stuff you can then derive from it such as the persistent DOM against which a standard XPath library can then be applied for searches. For information on what I'm referring to, see:
https://github.com/robtweed/qewd-jsdb
and a more broad discussion of the underlying concepts:
https://github.com/robtweed/global_storage
If anyone is interested in such a project, all I ask is for the appropriate attribution for the original concept.
If you are using JWTs as bearer tokens, then you need to provide a logout REST API that invalidates the JWT at the back-end and returns a response without a bearer token. See the QEWD/mg_web Conduit application examples for how it's done
All this kind of stuff is built into both QEWD's REST services and mgweb-server with appropriate APIs for all the JWT/Bearer Token life-cycle. Find both on my Github repos:
https://github.com/robtweed/qewd
https://github.com/robtweed/mgweb-server
See the REST/IRIS examples based on the RealWorld/Conduit application which use JWT-based authentication carried as bearer tokens:
IT techniques and trends go in cycles - it's always been that way and still is. Watch for the latest trend that's beginning to gain traction that's known as "HTML over the wire". It's basically server-side code that generates fragments of markup that are delivered over either HTTP or WebSockets and which dynamically update the DOM - stuff we were doing years ago in WebLink Developer and then EWD (which could be compiled to either (functionally identical) WebLink or CSP code). The capabilities at the back-end really haven't changed or improved since the mid 1990s - just lots of new buzzwords to describe the same old ways of skinning the cat. The key difference is the performance you'll get from an IRIS/Cache/M back-end compared with every other database-linked back-end out there. Anyway the good news is you have loads of choice these days for how you want to engineer and architect your web technology using IRIS, via what ISC provide and what others such as ourselves provide as Open Source solutions (eg see our mg_web, mg_web server and QEWD technologies)
I have to agree with Herman. When WebLink was first acquired by ISC, and during the period 1996-2000, we were demonstrating things that no other web tech at the time could do. Chris's Event Broker technology, originally released as a Java-based technology that ran in the browser, preceded and delivered what much later became known as Ajax. I remember during that time John Bertoglio, then a non-ISC user web guy who discovered what we were doing, telling me that in his estimation, we had a lead of between 5 and 8 years over what the rest of the industry was doing. We did try to lead the horse to water and all that, but hey...
Something that isn't, I think, widely known/realised is that if you install a 64-bit version of Linux on a Raspberry Pi (eg Ubuntu 20.04 64-bit for ARM), and then, if you have an M1 Apple Mac, install Parallels on it and install a Ubuntu 20.04 VM, the two environments (Raspberry Pi and M1 Mac VM) are functionally identical. Literally you can move the exact same code from the M1 Mac VM to the Raspberry Pi and it will work. Make it all even simpler by then installing Docker on both the Raspberry Pi and the M1 Mac VM, and a container built on one will, again, work identically on the other.
The only difference, of course, is the massively faster speed of execution you'll get on the M1 Mac (and the huge difference in cost!).
I've had a lot of fun with the various QEWD-related Docker containers I've created: there's something quite magical about watching a container built on the M1 Mac working, without any change, on a Raspberry Pi! It's what, IMO, makes ARM-64 a very interesting platform for the future
Take a look at QEWD.js - it works extremely well (and fast!) with IRIS on the Raspberry Pi, and you'll be able to build REST APIs and interactive apps using it. See:
https://github.com/robtweed/qewd
but also look at the QEWD-related postings I've put in here on Open Exchange:
https://openexchange.intersystems.com/package/QEWD-js
https://openexchange.intersystems.com/package/qewd-jsdb-kit-iris
Take a look at the ObjectScript functions $NAME, $QUERY, $QSUBSCRIPT and $QLENGTH
https://docs.intersystems.com/irislatest/csp/docbook/Doc.View.cls?KEY=R…
The difference is described in full in these two documents:
https://github.com/robtweed/global_storage/blob/master/Subscripts.md
https://github.com/robtweed/global_storage/blob/master/Leaf_Nodes.md
It's not a good idea to store files in the DB that you'll simply be reading back in full. The main issue you'll suffer from if you do hold them in the database (which nobody else seems to have picked up on) is that you'll needlessly flush/replace global buffers every time you read them back (the bigger the files, the worse this will be). Global buffers are one of the keys to performance.
Save the files and files and use the database to store their filepaths as data and indices.