#ECP

As applications grow, every database eventually hits scaling limits. Whether it's storage capacity, concurrent users, query throughput, or I/O bandwidth, single-server architectures have inherent constraints. This guide explains fundamental approaches to database scalability and shows how InterSystems IRIS implements these patterns to support enterprise-scale workloads.

We'll explore two complementary scaling strategies: horizontal scaling for user volume (distributing computational load) and sharding for data volume (partitioning datasets). Understanding the general principles behind these approaches will help you make informed decisions about when and how to scale your IRIS applications.

The examples in this guide use InterSystems IRIS in Docker containers.

Sometimes customers need a small IRIS instance to do something in the cloud and shut it down, or they need hundreds of containers (i.e. one per end user or one per interface) with small workloads. This exercise came about to see how small an IRIS instance could be. For this exercise we focused on what is the smallest amount of memory we can configure for an IRIS instance.Do you know all the parameters that affect the memory allocated by IRIS ?

Memory Settings

These are the different buckets that affect memory allocation by IRIS and its corresponding parameters:

• Global Buffers (32 MB) => 

Some time ago, I put together a small example to quickly deploy InterSystems IRIS instances connected via ECP using Docker.

Time passed and, like everything, it needed a bit of a refresh.

Mirror Your Database Across the Galaxy with Seeding

Hello cpf fans!  This distraction I used the "seed" capability in IRIS to provision an entire IrisCluster mirror, 4 maps wide with compute starting from an IRIS.DAT in a galaxy far far away.  This is pretty powerful if you have had a great deal of success with a solution running on a monolithic implementation and want it to scale to the outer rim with Kubernetes and the InterSystems Kubernetes Operator.  Even though my midichlorian count is admittely low, I have seen some hardcore CACHE hackers shovel around DATS, compact and shrink and update their ZROUTINES, so this same approach could also be helpful shrinking and securing your containerized workload too.  If you squint and feel all living things around you,  you can see a glimpse of in place (logical) mirroring in the future as a function of the operator and a migration path to a fully operational mirrored Death Star as the workload matures.

Hello Community,

I'm currently planning a system upgrade from Caché (2018.1.8.766) to InterSystems IRIS (2024.1). The environment consists of one database server and three application servers connected via ECP.

I’m considering two possible upgrade approaches:

1. Upgrade the application servers to IRIS first.
   Since IRIS is expected to be backward-compatible with Caché over ECP, the database server would remain on Caché temporarily.
2. Upgrade the database server first,
   followed by upgrading the application servers, as they don’t store data.

Could you please advise on the recommended approach

Are you familiar with SQL databases, but not familiar with IRIS?  Then read on...

About a year ago I joined InterSystems, and that is how IRIS got on my radar.  I've been using databases for over 40 years—much of that time for database vendors—and assumed IRIS would be largely the same as the other databases I knew.  However I was surprised to find that IRIS is in several ways quite unlike other databases, often much better.  With this, my first article in the Dev Community, I'll give a high-level overview of IRIS for people that are already familiar with the other databases such as Oracle, SQL Server, Snowflake, PostgeSQL, etc.   I hope I can make things clearer and simpler for you and save you some time getting started.

If several application servers are connected using ECP, and one of them create many locks, so many that lock table became full :

Should we expect only that application server to be impacted ? (usually, when lock table is full the system became instable). Or, are other application servers going to be impacted the same way ? (because that lock table is synchronized and maintained in sync across all servers)

I have another question (which is related) : is this the same with remote databases ?
Let's say 2 application servers access the same remote database and one application server create many l

With ECP we have the option to have a collection of Frontend instances
All Frontend servers typically have a common Master in the background
Concentrating data on the Master server is the primary goal.

As a side effect, this applies also to Classes, Routines, .. anything stored in Globals.
This is probably not the most efficient setup. But rather common anyhow.

Is embedded Python code also stored in Globals?

What is the recommended solution for a similar installation?

Hello,

I need to ensure that the task created/scheduled, by system user, is created in the routine database and not remotely in the ECP to which it is connected. How can I guarantee the creation/scheduling of this task?

Here is a suggestion for creating the routine in both environments:

https://openexchange.intersystems.com/package/Background-Jobs-ECP

Is there another option?

InterSystems has corrected two defects.

The first defect can cause an ECP application server to hang. This defect is only relevant to ARM and IBM Power processors; it is present in version 2022.1.2 and 2022.1.3 of InterSystems IRIS®, InterSystems IRIS for Health™, and HealthShare® Health Connect. The correction is identified as DP-423661. The defect is addressed in all future versions. 

The second defect can, in rare circumstances, cause dejournaling to hang. This defect is present in versions 2020.4, 2021.x, 2022.x, and 2023.1 of InterSystems IRIS®, InterSystems IRIS for Health™, and Healt

InterSystems has corrected a defect that can result in Enterprise Cache Protocol (ECP) client instability under rare conditions.

The defect exists in the following products and any InterSystems offerings based on them.

Impacted versions are 2022.1.x, 2022.2, and 2022.3:

InterSystems IRIS®

InterSystems IRIS for Health™

HealthShare® Health Connect

Impacted version is 2022.2 (only for customers deploying ECP):

InterSystems HealthShare®

The issue can only occur on ECP client systems. When this issue is triggered, processes may experience a <SYSTEM> or <DATABASE> error. Following the error,

Hey Community,

We're excited to announce that Community webinars are back!

Let us invite you all to @sween's webinar on Scaling InterSystems FHIR Server on Amazon Web Services with ECP.

Join this webinar to make a technical deep dive, see a demonstration, and benchmark horizontal scaling InterSystems FHIR Server on Amazon Web Services with Enterprise Cache Protocol (ECP).

⏱ Date & Time: Thursday, August 18, 8 AM ET | 2:00 PM CEST
👨‍🏫 Speaker: @sween, Full Stack Architect at Integration Required

Hi Community,

This article describes the small ZPM module global-archiver.
The goal is to move a part of a global from a database to another database.

Why this package?

A typical use case is read-only data sequentially added to your database that you can never delete.
For example:

• User log access to patient medical data.
• Medical documents versioning.

Depending on the intensive usage of your application, these data could highly increase your database size.
To reduce the backup time, it could be interesting to move these data to a database dedicated to the archive and make a backup of this dat

For those that, at some point, need to test what means that of ECP for horizontal escalability (computing power and/or users and processes concurrency), but they're lazy o have no much time to build the environment, configure the server nodes, etc..., I've just published in Open Exchange the app/sample OPNEx-ECP Deployment .

Testing ECP-based applications often take quite some effort for setup and preparation.
I have created a Docker-based workbench that allows you to have it quick at hands.
And if you crash it? You just give your containers a fresh start.
The whole setup runs code-based during the start-up of your instance.
In that sense, it is also a portable coding example using ZPM and the objectscript-docker-template

 see Video

Hi Community,

Please welcome the new video recorded by @Robert Cemper on InterSystems Developers YouTube:

⏯ IRIS easy ECP workbench

Migration from Caché to IRIS can be quite a challenge if your code is grown over many years
and probably not so clean structured as you may like it. So you face the need to check your
migrated code against some reference data. A few samples might not be a problem,
but some hundred GB of data for testing might be.  

A possible step could be to have your fresh code in IRIS but leave your huge datastore on Caché and connect both environments over ECP.  I have created a demo project that gives you the opportunity to try this based on 2 Docker images with IRIS and with Caché connected over ECP.

Hello,

I am working with Elastic Container Service in AWS running IRIS for Health containers 2020.1. We have an ECP Data Server. I like to be able to add a new host IP on the data server when a new client starts up. What table are the Allowed ClientIPs stored in?

Thanks,

Oliver

I was able to define ECP connection in Installer class that I run when I build docker image for my Docker IRIS application. Now I start to use Amazon ECS and Autoscaling. When a new instance is created and it runs the IRIS container, I want it to "register" as an Application server on the remote database server. What code can I execute to add an ip address as an Application Server?

Hello,

I work on deploying IRIS inside Docker container. I really like %Installer class can automate many steps. I want to establish an ECP connection to a mirror database and then define a remote database on the application server. I have already seen we can create local database and namespace in %Installer. What code is needed to establish ECP connection?

Those of you that worked already with ECP might know that it operates not only across
operating systems, processor technologies (big-little-endian), but also -with limits- across versions in C/E.
This cross-version feature also includes IRIS (verified for 2020.1)  and C/E (2018.3)

I have a Read-Only Reporting Async with a Remote Database defined to a non-mirrored database on the primary failover member. I would like to update the non-mirrored database on the primary from the async; so far all my attempts fail with *protect* errors.

I found this page 

https://community.intersystems.com/post/execute-method-failover-member-…

and tried the SqlProc method. The procedure executed, no errors, but the change to the global was never made (or least not retained) either directly or through a JOBbed process. I wonder if I'm missing something in my configuration. If not, the

Hello!

I'm interested to hear if folks have experience using Docker containers with Caché instances using ECP. Wondering if there are any special considerations when setting up a distributed application with multiple containers communicating with ECP. Any input is appreciated!

In one of the projects, when we have ECP with 10 ECP application servers, from time to time we faced the issue when our journals fail to purge, due to open transactions. While we have about 100-150 GB journal files per day, it quite quickly became a big issue, and with mirroring a very big issue. Mostly we just rebooted our ECP Data server, so it searches rollbacks any transactions, but such process is too long, may steal a few hours. I did not find any way, how to get the list of the open transactions from one place from ECP Data Server. We just migrated our Data server to 2018.1. while our A

Hi,

I'm trying to get the ECP status of 2 servers using $system.ECP.GetProperty("ServerStats") and $system.ECP.GetProperty("ClientStats"), but I'm not able to find the documentation of the values returned:

For example, on the server side I execute this:

USER>w $system.ECP.GetProperty("ServerStats")
14973692,14965389,15903851,2482140,12941,306696,0,0,613392,0,9767,7,10,0,2,0,896,0,180883896,1943934512,0,2444,0,0,12215838,8204,43,0,40,40,0,0,0,8585,82238,1,4,19825,4387,3926,3059,4813,3368,5456,37412,8204,12215838,2,8760,2,87963,0,3,2290946,180615,28945,0,18366,0,499250,128761,137048,1592639,0,

Some readers of my previous article, Caché eXTreme for .NET - direct access to globals from C#, wondered if you could access information not just from the same instance in which you’re working, but also from another instance on the same computer, or from an instance located on another computer in the same local network. Some theorized, correctly, that this would be possible using the Enterprise Cache Protocol (ECP). In this article I’m going to show how it can be done.

Running a Background Job using the JOB command is a well-known feature.
Using ECP to distribute databases to several servers is also well-known.
But using the combination of both to run a process on a different server
seems to be a rare case.

Sure there are enough other ways to start a remote job, but the special
combination with ECP where the application server starts a process on a
data server without additional networking is worth to be remembered.

The example starts a remote process and receives back a result.
The parameters sent and received are pure demo purposes and require
adaption to the i

Suppose we need to store millions of values temporarily, that means, we don't care about them if we lose them but our application use them to get realtime information. Should I use Cachetemp or whatever other DB without journaling enabled? If answer is Cachetemp, shouldn't be a problem if we decide to scale using App Server + ECP? I'm not sure what would happen with the app logic in such architecture as I guess I couldn't map and share cachetemp...

Any idea/suggestion?

Last week, we announced the InterSystems IRIS Data Platform, our new and comprehensive platform for all your data endeavours, whether transactional, analytics or both. We've included many of the features our customers know and loved from Caché and Ensemble, but in this article we'll shed a little more light on one of the new capabilities of the platform: SQL Sharding, a powerful new feature in our scalability story.

Hi All,

Background: I have my data in Globals on a Cache instance A and I SQL projected(Added SQL Storage) all my globals as Classes to do SQL Operation.

I am about to start writing my application code in Ensemble(Planning to use Business Service, process and Operation) , Now I wanted to do ECP connection between Ensemble instance and Cache instance to get data from Cache.

Questions

How to do ECP connection between two instances?

Does SQL projection of classes is really required or I can use the globals from Cache server.

Can anyone please share the link to learn the basics of ECP connec