I have a database that is 2TB in size (from a LIVE system).

I would like to defragment it but I have some questions. I took a look at official documentation but it does not help.

• Is is possible to get a measurement about how much database is fragmented ? (eg: a percentage). Most operating systems are able to give a an indication about how much a filesystem is fragmented before any operation.
• The database is on a LIVE server, the IRIS service is restarted every night (around 1AM). Because of that I can't run defragment continuously. I don't think it will be able to complete under 24 hours.

Having been inspired with Shared code execution speed question/discussion, I dare to ask another one which is annoying me and my colleagues for several weeks.

We have a routine called Lib that comprises 200 $$-functions of 1500 code lines total. It was noticed that after calling _any_ function of another rather big routine (1900 functions, 32000 lines) the next call of $$someFunction^Lib(x) is getting 10-20% slower than previous call of the same function.

Hey everyone,

I'm diving deeper into Caché ObjectScript and would love to open a discussion around the most useful tips, tricks, and best practices you’ve learned or discovered while working with it.

Whether you're an experienced developer or just getting started, ObjectScript has its own set of quirks and powerful features—some well-documented, others hidden gems. I’m looking to compile a helpful set of ideas from the community.

Hello,
I have created this script that does lot of writes to a single global. DB write performance is much slower than expected (compared to another similar systems).

set rec = "..." //fill it with something
set time = $piece($horolog,",",2)
while(($piece($horolog,",",2)-time) < 30) //30 seconds
    set ^A($System.Util.CreateGUID()) = rec
}

I have notified the following : 

• CPU usage does not reach 100% on a single core (eg: 25% of total CPU usage should be seen on a 4 cores system).

IRIS and Ensemble are designed to act as an ESB/EAI. This mean they are build to process lots of small messages.

But some times, in real life we have to use them as ETL. The down side is not that they can't do so, but it can take a long time to process millions of row at once.

To improve performance, I have created a new SQLOutboundAdaptor who only works with JDBC.

BatchSqlOutboundAdapter

Extend EnsLib.SQL.OutboundAdapter to add batch batch and fetch support on JDBC connection.

Benchmark

Benchmarks released on Postgres 11.2 with 1 000 000 rows fetched and 100 000 rows inserted on 2 columns.

Introduction

MonLBL is a tool for analyzing the performance of ObjectScript code execution line by line. codemonitor.MonLBL is a wrapper based on the %Monitor.System.LineByLine package from InterSystems IRIS, designed to collect precise metrics on the execution of routines, classes, or CSP pages.

The wrapper and all examples presented in this article are available in the following GitHub repository: iris-monlbl-example

Let's suppose two different routines use one and the same chunk of code. From the object-oriented POV, a good decision is to have this chunk of code in a separate class and have both routines call it. However, whenever you call code outside of the routine as opposed to calling code in the same routine, some execution speed is lost. For reports churning through millions of transactions this lost speed might be noticeable. Any advice how to optimize specifically speed?
P.S. Whenever someone is talking about the best choice for whatever, I am always tempted to ask: "What are we optimizing?".

Hello Community,

I have a queue that I need to traverse and perform operations on. To distribute the workload across multiple processes, I used a Work Queue Manager do process and I'm not expecting any status from it So, I skipped Sync / WaitForComplete in my implementation

Hello Community,

The subroutine ^routine is not executed while the queue is being processed in WorkMgr. However, it works when defined as a function. Is it mandatory to define subroutine^routine as a function for it to execute properly?

testwqm.mac
 set wqm = ##class(%SYSTEM.WorkMgr).%New()
 set sc=wqm.Queue("subr1^testwqm")  ; not executing the subr1 
 set sc=wqm.Queue("subr2^testwqm") ; executing the subr2 properly
 set sc=wqm.

Introduction

Database performance has become a critical success factor in a modern application environment. Therefore identifying and optimizing the most resource-intensive SQL queries is essential for guaranteeing a smooth user experience and maintaining application stability. 

This article will explore a quick approach to analyzing SQL query execution statistics on an InterSystems IRIS instance to identify areas for optimization within a macro-application.

Rather than focusing on real-time monitoring, we will set up a system that collects and analyzes statistics pre-calculated by IRIS once an hour.  This approach, while not enabling instantaneous monitoring, offers an excellent compromise between the wealth of data available and the simplicity of implementation. 

We will use Grafana for data visualization and analysis, InfluxDB for time series storage, and Telegraf for metrics collection.  These tools, recognized for their power and flexibility, will allow us to obtain a clear and exploitable view.

More specifically, we will detail the configuration of Telegraf to retrieve statistics. We will also set up the integration with InfluxDB for data storage and analysis, and create customized dashboards in Grafana. This will help us quickly identify queries requiring special attention.

To facilitate the orchestration and deployment of these various components, we will employ Docker.

InterSystems has been at the forefront of database technology since its inception, pioneering innovations that consistently outperform competitors like Oracle, IBM, and Microsoft. By focusing on an efficient kernel design and embracing a no-compromise approach to data performance, InterSystems has carved out a niche in mission-critical applications, ensuring reliability, speed, and scalability.

Hello everyone,

I’m facing issues with replicating data from my Caché 2016 database to a PostgreSQL database. I need to handle around 300 data updates per minute, and whenever certain tables are modified, those changes must be reflected in other databases.

So far, I’ve tried various approaches, including:

• Setting up an intermediary API,
• Using Azure Service Bus,
• Leveraging Caché Jobs,
• All of which rely on table triggers as the entry point.

However, each of these solutions has led to performance bottlenecks and system lockups.

There are three things most important to any SQL performance conversation:  Indices, TuneTable, and Show Plan.  The attached PDFs includes historical presentations on these topics that cover the basics of these 3 things in one place.  Our documentation provides more detail on these and other SQL Performance topics in the links below.  The eLearning options reinforces several of these topics.  In addition, there are several Developer Community articles which touch on SQL performance, and those relevant links are also listed.

There is a fair amount of repetition in the information listed below.  The most important aspects of SQL performance to consider are:

1. The types of indices available
2. Using one index type over another
3. The information TuneTable gathers for a table and what it means to the Optimizer
4. How to read a Show Plan to better understand if a query is good or bad

The rise of Big Data projects, real-time self-service analytics, online query services, and social networks, among others, have enabled scenarios for massive and high-performance data queries. In response to this challenge, MPP (massively parallel processing database) technology was created, and it quickly established itself. Among the open-source MPP options, Presto (https://prestodb.io/) is the best-known option. It originated in Facebook and was utilized for data analytics, but later became open-sourced. However, since Teradata has joined the Presto community, it offers support now.

I use the following code to calculate the SHA1 of a file :

set stream = ##class(%Stream.FileBinary).%New()
do stream.LinkToFile(filename)
write $SYSTEM.Encryption.Base64Encode($SYSTEM.Encryption.SHA1HashStream(stream))

This code is called thousands of time and performance is critical. I have tried to code same logic in another language (which is lower level) and it's almost twice as fast. It's unclear why so I started investigating.

Using Process Monitor, it shows that files are read in chunks of 1024 bytes (1K) which is suboptimal. Reading a file of 1MB while require 1024 file system calls.

We've recently made available a new version of InterSystems IRIS in the Vector Search Early Access Program, featuring a new Approximate Nearest Neighbor index based upon the Hierarchical Navigable Small World (HNSW) indexing algorithm. This addition allows for highly efficient, approximate nearest-neighbor searches over large vector datasets, dramatically improving query performance and scalability.

A customer recently asked if IRIS supported OpenTelemetry as they where seeking to measure the time that IRIS implemented SOAP Services take to complete.

Is there a difference in outcome between the two screengrabs below?

In both cases, when  certain conditions are met, a transformation is called and the output sent on to two targets. In the first case we surmise the transformation is called twice, and the output of the first run sent to the first target, the output of the second run to the second target. In the second case we surmise the transformation is called once, and the output duplicated and sent to the two targets. 

• Are there any cases where they would not be functionally the same (eg because of side-effects of the transformation?

So if you are following from the previous post or dropping in now, let's segway to the world of eBPF applications and take a look at Parca, which builds on our brief investigation of performance bottlenecks using eBPF, but puts a killer app on top of your cluster to monitor all your iris workloads, continually, cluster wide!  

Continous Profiling with Parca, IRIS Workloads Cluster Wide

I attended Cloud Native Security Con in Seattle with full intention of crushing OTEL day, then perusing the subject of security applied to Cloud Native workloads the following days leading up to CTF as a professional excercise. This was happily upended by a new understanding of eBPF, which got my screens, career, workloads, and atitude a much needed upgrade with new approaches to solving workload problems. 

So I made it to the eBPF party and have been attending clinic after clinic on the subject ever since, here I would like to "unbox" eBPF as a technical solution, mapped directly to what we do in practice (even if its a bit off), and step through eBPF through my experimentation on supporting InterSystems IRIS Workloads, particularly on Kubernetes, but not necessarily void on standalone workloads.

eBee Steps with eBPF and InterSystems IRIS Workloads

Question: can a business process with pool size > 1 invoke external programs in parallel via $ZF(-100)?

We have a production that takes HL7 (ORU_R01) messages in and produces a PDF from each one for onward transmission to a document store.

The PDF creation is implemented in a business process which transforms the HL7 to XML, saves the XML to the filesystem, then calls Apache FOP (the version distributed with Ensemble) via $ZF(-100) to produce the PDF, which is stored on the filesystem.

It's been a long time since I didn't write an update post on IoP.

So what's new since IoP command line interface was released?

Two new big features were added to IoP:

• Rebranding: the grongier.pex module was renamed to iop to reflect the new name of the project.
• Async support: IoP now supports async functions and coroutines.

Rebranding

The grongier.pex module was renamed to iop to reflect the new name of the project.

The grongier.pex module is still available for backward compatibility, but it will be removed in the future.

When there's a performance issue, whether for all users on the system or a single process, the shortest path to understanding the root cause is usually to understand what the processes in question are spending their time doing.  Are they mostly using CPU to dutifully march through their algorithm (for better or worse); or are they mostly reading database blocks from disk; or mostly waiting for something else, like LOCKs, ECP or database block collisions?

Tools to help answer the questions above have always been available in various forms.

Hi Community, 
We invite you to join our next Meetup in Cambridge, MA on August 28, 5:30-7:30 pm.
>> RSVP here <<
 

Hello Community,

As per the Build index documentation "If you use BUILD INDEX on a live system, the index is temporarily labeled as not selectable, meaning that queries cannot use the index while it is being built. Note that this will impact the performance of queries that use the index." Is this  hiding/not selectable is only applicable for BUILD INDEX or it supports class level %BuildIndices as well. as far as my analysis both syntax setting this setting  SetMapSelectability. 

Thanks!

There are often questions surrounding the ideal Apache HTTPD Web Server configuration for HealthShare.  The contents of this article will outline the initial recommended web server configuration for any HealthShare product. 

As a starting point, Apache HTTPD version 2.4.x (64-bit) is recommended.  Earlier versions such as 2.2.x are available, however version 2.2 is not recommended for performance and scalability of HealthShare.

Hello everyone,

Recently, I've been working on a Business Process that processes a large JSON FHIR message containing up to 50k requests in an array within the JSON.

Currently, the code imports the JSON as a dynamic object from the original message stream, obtains an iterator from it, and processes each request one at a time in a loop.

The performance meets the requirements, even with much larger requests than the one exposed above.

We are experimenting with IIS, as the PWS will be gone in newer versions.

The code which is executed, takes 15ms to run. If we execute it through PWS (REST), there is some overhead and the total execution time is 40ms, which is acceptable. However, if we go through IIS, it takes 150ms or sometimes even more.

Both PWS and IIS are running on the same server as IRIS in this case. No optimisations have been done on IIS.

Any suggestions on where to look/what to optimize on IIS?

[Background]

InterSystems IRIS family has a nice utility ^SystemPerformance (as known as ^pButtons in Caché and Ensemble) which outputs the database performance information into a readable HTML file. When you run ^SystemPerformance on IRIS for Windows, a HTML file is created where both our own performance log mgstat and Windows performance log are included.

You may have heard about our mg-dbx-napi interface for IRIS which provides insanely fast access from Node.js.  If you've been following recent developments in the server-side JavaScript world, you'll be excited to know that mg-dbx-napi also works with Bun.js, the latter proving to be significantly faster than Node.js for many/most purposes.

Of course, if you're a Node.js user, you'll probably wonder how mg-dbx-napi compares with the Native API for Node.js that is included with IRIS.