Challenges of real-time AI/ML computations

We will start from the examples that we faced as Data Science practice at InterSystems:

• A “high-load” customer portal is integrated with an online recommendation system. The plan is to reconfigure promo campaigns at the level of the entire retail network (we will assume that instead of a “flat” promo campaign master there will be used a “segment-tactic” matrix). What will happen to the recommender mechanisms? What will happen to data feeds and updates into the recommender mechanisms (the volume of input data having increased 25000 times)? What will happen to recommendation rule generation setup (the need to reduce 1000 times the recommendation rule filtering threshold due to a thousandfold increase of the volume and “assortment” of the rules generated)?
• An equipment health monitoring system uses “manual” data sample feeds. Now it is connected to a SCADA system that transmits thousands of process parameter readings each second. What will happen to the monitoring system (will it be able to handle equipment health monitoring on a second-by-second basis)? What will happen once the input data receives a new bloc of several hundreds of columns with data sensor readings recently implemented in the SCADA system (will it be necessary, and for how long, to shut down the monitoring system to integrate the new sensor data in the analysis)?
• A complex of AI/ML mechanisms (recommendation, monitoring, forecasting) depend on each other’s results. How many man-hours will it take every month to adapt those AI/ML mechanisms’ functioning to changes in the input data? What is the overall “delay” in supporting business decision making by the AI/ML mechanisms (the refresh frequency of supporting information against the feed frequency of new input data)?

Author: Sergey Lukyanchikov, InterSystems

For one major reason: to avoid progressive technical and economic performance deterioration in an AIaaS setup characterized by increasing volume, velocity and variety of data flows (the famous Big Data’s “3 Vs”).

Let us take a closer look at the variety factor: if the variety of our data increases (e.g., data comes from an ever-increasing range of ever-evolving data-generating processes), this inevitably builds up demand for more and more functionality.

* In-Platform Agent-Based Simulation of a Connected Factory Cluster

Author: Sergey Lukyanchikov, InterSystems

1. Purpose

In this paper we prototype and explore how multiple agent-based models of robotic factories connected to other robotic factories (represented by their respective models) can be orchestrated using an all-purpose data platform – thereby simulating descriptive and predictive properties of a group of factories (a factory cluster).

Fixing the terminology

A robot is not expected to be either huge or humanoid, or even material (in disagreement with Wikipedia, although the latter softens the initial definition in one paragraph and admits virtual form of a robot). A robot is an automate, from an algorithmic viewpoint, an automate for autonomous (algorithmic) execution of concrete tasks. A light detector that triggers street lights at night is a robot. An email software separating e-mails into “external” and “internal” is also a robot. Artificial intelligence (in an applied and narrow sense, Wikipedia interpreting it differently again) is algorithms for extracting dependencies from data. It will not execute any tasks on its own, for that one would need to implement it as concrete analytic processes (input data, plus models, plus output data, plus process control). The analytic process acting as an “artificial intelligence carrier” can be launched by a human or by a robot. It can be stopped by either of the two as well. And managed by any of them too.

What is Distributed Artificial Intelligence (DAI)?

Attempts to find a “bullet-proof” definition have not produced result: it seems like the term is slightly “ahead of time”. Still, we can analyze semantically the term itself – deriving that distributed artificial intelligence is the same AI (see our effort to suggest an “applied” definition) though partitioned across several computers that are not clustered together (neither data-wise, nor via applications, not by providing access to particular computers in principle). I.e., ideally, distributed artificial intelligence should be arranged in such a way that none of the computers participating in that “distribution” have direct access to data nor applications of another computer: the only alternative becomes transmission of data samples and executable scripts via “transparent” messaging. Any deviations from that ideal should lead to an advent of “partially distributed artificial intelligence” – an example being distributed data with a central application server. Or its inverse. One way or the other, we obtain as a result a set of “federated” models (i.e., either models trained each on their own data sources, or each trained by their own algorithms, or “both at once”).

Hi Community,

Enjoy watching the new session recording from the InterSystems AI+ML Summit 2021:

⏯ AI+ML Summit Convergent Analytics – Healthcare Stream

Hey Developers,

Please welcome the new session recording from the InterSystems AI+ML Summit 2021:

⏯ AI+ML Summit Convergent Analytics – Cross-Industry Stream

ML Toolkit for InterSystems IRIS, besides Python/R/Julia, allows orchestrating cloud-based advanced analytics services, such as Microsoft Azure Data Factory and Machine Learning:

Inter

Intersystems IRIS is a complete platform to get insights from SQL and NoSQL data. It is possible get data from Interoperability adapters or using a set of IRIS tables as data sources and model BI or NLP Cubes, covering all type of data (other tools are limited to SQL). There are the option to enable intensive analytics processing using Spark too. So you can model your analysis using IRIS web analyzers (many tools use desktop tools) and than visualize and produce insights using IRIS Dashboards and IRIS User Portal or your third party options, using open options like MDX and REST.

InterSystems IRIS ML Toolkit adds the power of InterSystems IntegratedMLto further extend convergent scenario coverage into the area of automated feature and model type/parameter selection. The previous "manual" pipelines now collaborate within the same analytic process with "auto" pipelines that are based on automation frameworks, such as H2O.

This series of articles would cover Python Gateway for InterSystems Data Platforms. Execute Python code and more from InterSystems IRIS. This project brings you the power of Python right into your InterSystems IRIS environment:

• Execute arbitrary Python code
• Seamlessly transfer data from InterSystems IRIS into Python
• Build intelligent Interoperability business processes with Python Interoperability Adapter
• Save, examine, modify and restore Python context from InterSystems IRIS

Other articles

The plan for the series so far (subject to change).

• Part I: Overview, Landscape, and Introduction
• Part II: Installation and Troubleshooting
• Part III: Basic functionality
• Part IV: Interoperability Adapter
• Part V: Execute function
• Part VI: Jupyter Notebook <-- you're here
• Part VII: Dynamic Gateway
• Part VIII: Proxy Gateway
• Part IX: Use cases and ML Toolkit

Intro

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

This extension allows you to browse and edit InterSystems IRIS BPL processes as jupyter notebooks.

This repository is a go-public that builds on the already existing InterSystems-internal Convergent Analytics community and InterSystems-private MLToolkit repo (remains active and contains the most recent information for external ML Toolkit users - request memebership by writing us at MLToolkit@intersystems.com). This repo embraces more than ML Toolkit, we would like to host any discussions, publications, projects that add up in what we call convergent analytics approach. Welcome!