Challenges of real-time AI/ML computations

We will start from the examples that we faced as Data Science practice at InterSystems:

  • A “high-load” customer portal is integrated with an online recommendation system. The plan is to reconfigure promo campaigns at the level of the entire retail network (we will assume that instead of a “flat” promo campaign master there will be used a “segment-tactic” matrix). What will happen to the recommender mechanisms? What will happen to data feeds and updates into the recommender mechanisms (the volume of input data having increased 25000 times)? What will happen to recommendation rule generation setup (the need to reduce 1000 times the recommendation rule filtering threshold due to a thousandfold increase of the volume and “assortment” of the rules generated)?
  • An equipment health monitoring system uses “manual” data sample feeds. Now it is connected to a SCADA system that transmits thousands of process parameter readings each second. What will happen to the monitoring system (will it be able to handle equipment health monitoring on a second-by-second basis)? What will happen once the input data receives a new bloc of several hundreds of columns with data sensor readings recently implemented in the SCADA system (will it be necessary, and for how long, to shut down the monitoring system to integrate the new sensor data in the analysis)?
  • A complex of AI/ML mechanisms (recommendation, monitoring, forecasting) depend on each other’s results. How many man-hours will it take every month to adapt those AI/ML mechanisms’ functioning to changes in the input data? What is the overall “delay” in supporting business decision making by the AI/ML mechanisms (the refresh frequency of supporting information against the feed frequency of new input data)?

4 0
1 475
Article
· Mar 5, 2021 3m read
Using ECP across IRIS and Caché

Migration from Caché to IRIS can be quite a challenge if your code is grown over many years
and probably not so clean structured as you may like it. So you face the need to check your
migrated code against some reference data. A few samples might not be a problem,
but some hundred GB of data for testing might be.

A possible step could be to have your fresh code in IRIS but leave your huge datastore on Caché and connect both environments over ECP. I have created a demo project that gives you the opportunity to try this based on 2 Docker images with IRIS and with Caché connected over ECP.

12 2
0 473

Some changes in IRIS configuration require a restart of IRIS.
This is no big issue as long as I have access to the server command line with sufficient privileges.

In a container, this is not always given.
Stopping IRIS from the terminal/session prompt is no problem.
But the restart after is.

Note1: container start-stop is no option as it might be removed by option --rm in docker run
Note2: the target is linux (manly in docker). Windows is excluded

GitHub

9 4
0 465
Article
· Jan 25, 2021 2m read
Static Documentation Generators

Hi Community,

In the past, technical documentation of the source code and software products was generated in chm, pdf files and documentation generators of the programming languages themselves. This old approach had the following limitations:
1. Outdated documentation;
2. Non-interactive and difficult to consult documentation;
3. Layout unresponsive, unfriendly and not adherent to HTML;
4. Inability to customize the layout of the documentation;
5. Inability to have HTML 5 documentation online and offline.
6. Lack of Markdown support.

1 5
0 460

In Studio you could open a class directly via it's name, without having to traverse the package tree with multiple clicks until arriving at the desired class.

You would Ctrl + O or (File -> Open) and be able to simply type in the class name, for example:

You press Enter, and viola - the class is opened.

How do you achieve this in VSCode?

5 1
0 449

What is Distributed Artificial Intelligence (DAI)?

Attempts to find a “bullet-proof” definition have not produced result: it seems like the term is slightly “ahead of time”. Still, we can analyze semantically the term itself – deriving that distributed artificial intelligence is the same AI (see our effort to suggest an “applied” definition) though partitioned across several computers that are not clustered together (neither data-wise, nor via applications, not by providing access to particular computers in principle). I.e., ideally, distributed artificial intelligence should be arranged in such a way that none of the computers participating in that “distribution” have direct access to data nor applications of another computer: the only alternative becomes transmission of data samples and executable scripts via “transparent” messaging. Any deviations from that ideal should lead to an advent of “partially distributed artificial intelligence” – an example being distributed data with a central application server. Or its inverse. One way or the other, we obtain as a result a set of “federated” models (i.e., either models trained each on their own data sources, or each trained by their own algorithms, or “both at once”).

2 0
1 444
Article
· Mar 3, 2021 4m read
Access to IRIS from Rust

What do you think If I will say you, that very soon you will be able to connect to IRIS from the application written in Rust.

What is Rust

Rust is a multi-paradigm programming language designed for performance and safety, especially safe concurrency. Rust is syntactically similar to C++, but can guarantee memory safety by using a borrow checker to validate references. Rust achieves memory safety without garbage collection, and reference counting is optional. (c) Wikipedia

6 5
2 414

Hi folks!

Want to share a lifehack with you on ObjectScript highlighiting withing VSCode for script files.

Script files are just files with lines of ObjectScript that we feed anywhere, e.g. into IRIS during Docker baking procedure.

Typical usecase - here is the Scriptfile

here is the Dockerfile where we feed it.

And here is how it looks like usually in VSCode:

Screenshot 2021-08-05 at 09 51 06

Coudl be more beautiful, right?

1 0
0 413
Article
· Nov 28, 2021 3m read
Leveraging the Audit database

The InterSystems IRIS has a great audit system. It is responsible for auditing system events, but you can use it to audit your applications (great feature).

The audit system is based into event concept. The events can occur with IRIS or in an application. So, we have two type of events to the audit system:

1. System events: events occured into the InterSystems IRIS components (database, interoperability, analytics and core);

2 4
1 412

Purpose

Most CloudFormation articles are Linux-based (no wonder), but there seems to be a demand for automation for Windows as well. Based on this original article by Anton, I implemented an example of deploying a mirror cluster to Windows servers using CloudFormation.I also added a simple walk through.
The complete source code can be found here.

Update: 2021 March 1 I added a way to connect to Windows shell by public key authentication via a bastion host as a one-liner.

0 0
0 412

Deploying InterSystems HealthShare code, supporting lookups and artifacts like ssl certs, keys etc is relatively straight forward using Gitlab Runners. Not only does this approach enable managing the code base and deploying with git type workflows, but it also lends to a speedy recovery and repeatable environments for some implementations.

3 1
1 410

I have just created a new Global Master Topic, "IRIS Cheatsheets". IRIS has introduced a lot of new functionality, especially in scripting languages, FHIR R4 support, enhanced Interoperability Tools, and IRIS Analytics. Having spent 35 years working on Windows-based PC's and Laptops, I have surprisingly little knowledge of Linux, Docker and Git. Furthermore, I have written almost every application and Interface in ObjectScript with splatterings of SQL, .Net, and Java Gateways and the most basic knowledge of WinSCP, Putty, SSH. All that changed when I received my first Raspberry Pi.

8 3
1 407

Hi developers!

Just want to share an old but always relevant best practice on namespaces changing @Dmitry Maslennikov shared with me (again).

Consider method:

classmethod DoSomethingInSYS() as %Status

{

set sc=$$$OK

set ns=$namespace

zn "%SYS"

// try-catch in case there will be an error

try {

// do something, e.g. config change

}

catch {}

 zn ns    ; returning back to the namespace we came in the routine

return sc

}

And with new $namespace the method could be rewritten as:

classmethod DoSomethingInSYS() as %Status

{

set sc=$$$OK

new $namespace

set $namespace="%SYS"

// do something

return sc

}

So! The difference is that we don't need to change the namespace manually as it will be back automatically once we return the method.

and we don't need try-catch (at least for this purpose) too.

3 4
0 401
Article
· Sep 11, 2021 2m read
Successful Troubleshooting

During the last weeks, I was working on various issues and problems related to SW development.
I found that quite often problem analysis was mostly chasing issues just on the surface
but not really attacking the deeper reasons of the problem and follow the consequences.
It's like the doc that stops your leg bleeding but doesn't see that it is broken.

8 2
0 396