High-Performance Message Searching in Health Connect

The Problem

Have you ever tried to do a search in Message Viewer on a busy interface and had the query time out? This can become quite a problem as the amount of data increases. For context, the instance of Health Connect I am working with does roughly 155 million Message Headers per day with 21 day message retention. To try and help with search performance, we extended the built-in SearchTable with commonly used fields in hopes that indexing these fields would result in faster query times. Despite this, we still couldn't get some of these queries to finish at all.

Hello community,

I wanted to share my experience about working on Large Data projects. Over the years, I have had the opportunity to handle massive patient data, payor data and transactional logs while working in an hospital industry. I have had the chance to build huge reports which had to be written using advanced logics fetching data across multiple tables whose indexing was not helping me write efficient code.

Here is what I have learned about managing large data efficiently.

Choosing the right data access method.

As we all here in the community are aware of, IRIS provides multiple ways to access data. Choosing the right method, depends on the requirement.

• Direct Global Access: Fastest for bulk read/write operations. For example, if i have to traverse through indexes and fetch patient data, I can loop through the globals to process millions of records. This will save a lot of time.

Set ToDate=+H
Set FromDate=+$H-1 For  Set FromDate=$O(^PatientD("Date",FromDate)) Quit:FromDate>ToDate  Do
. Set PatId="" For  Set PatId=$Order(^PatientD("Date",FromDate,PatID)) Quit:PatId=""  Do
. . Write $Get(^PatientD("Date",FromDate,PatID)),!

• Using SQL: Useful for reporting or analytical requirements, though slower for huge data sets.

In my previous article, Using LIKE with Variables and Patterns in SQL, we explored how the LIKE predicate behaves in different scenarios, from Embedded SQL to Dynamic SQL, and what happens to performance when wildcards and variables come into play. That piece was about getting comfortable writing a working LIKE query. But writing SQL that works is only the starting point. To build applications that are reliable, scalable, and secure, you need to understand the best practices that underpin all SQL, including queries that use LIKE.

This article takes the next step. We’ll look at a few key points to help strengthen your SQL code, avoid common pitfalls, and make sure your SELECT statements run not just correctly, but also efficiently and safely. I'll use SELECT statements with LIKE predicate as an example along the way, showing how these broader principles directly affect your queries and their results.

*This is what Gemini came up with for this article, kinda cute.

Hi everyone! 👋
I’m excited to share the project I’ve submitted to the current InterSystems .Net, Java, Python, and JavaScript Contest — it’s called IRIStool and Data Manager, and you can find it on the InterSystems Open Exchange and on my GitHub page.

I joined InterSystems less than a year ago. Diving into ObjectScript and IRIS was exciting, but also full of small surprises that tripped me up at the beginning. In this article I collect the most common mistakes I, and many new colleagues, make, explain why they happen, and show concrete examples and practical fixes. My goal is to help other new developers save time and avoid the same bumps in the road.

InterSystems FAQ rubric

One way to optimize query performance is to use query parallelism on a per-query or system-wide basis (a standard feature).

This is a technique for dividing the execution of a particular query among processors on a multi-processor system. The query optimizer will execute parallel processing only if there is a possibility of benefiting from parallel processing. Parallel processing is only applicable to SELECT statements.

I have been walking through this with a few team members and as such I thought there might be others out there who could use it, especially if you work with HL7 & Ensemble/HealthConnect/HealthShare and never venture out past the Interoperability section.

Hi folks!

It is very easy to import CSV data into IRIS. But what if we want to preserve the original IDs in CSV?

Recently I came across with the situation when I needed to import two csv's into IRIS which were linked by one column referencing to another csv's col: a typical Foreign Key and Primary Key situation, where csv1 contains this column as Primary Key, and csv2 as Foreign key with id's related to csv1.

The image is generated by ChatGPT so don't blame it - it tried its best to generate countries as primary keys with countries.csv-cities.csv relationship :)

SQLTools is a Visual Studio Code (VS Code) Extension with over 3.5 million downloads that provides connections to many of the most used databases, including InterSystems IRIS, using drivers.

The August Article Bounty on the Global Masters article caught my attention, and one of the proposed topics sounded quite interesting in regard to its future use in my teaching. So, here's what I'd like to tell my students about tables in IRIS and how they correlate with the object model.

First of all, InterSystems IRIS boasts a unified data model. This means that when you work with data, you are not locked into a single paradigm. The same data can be accessed and manipulated as a traditional SQL table, as a native object, or even as a multidimensional array (a global). It means that when you create an SQL table, IRIS automatically creates a corresponding object class. When you define an object class, IRIS automatically makes it available as an SQL table. The data itself is stored only once in IRIS's efficient multidimensional storage engine. The SQL engine and the object engine are simply different "lenses" to view and work with the same data.

First, let's look at the correlation between the relational model and the object model:

It's not always a 1:1 correlation, as you may have several tables represent one class, for example. But it's a general rule of thumb.

This article is a continuation of the IRIS JSON project and features additional methods and insights.

Let's continue with the instance methods

Let's start with a simple motivating question: over the past 14 days, what are my most common errors in the Application Error Log?

InterSystems IRIS 2024 recently introduced the vector types.
This addition empowers developers to work with vector search, enabling efficient similarity searches, clustering, and a range of other applications.
In this article, we will delve into the intricacies of vector types, explore their applications, and provide practical examples to guide your implementation.

Introduction

InterSystems IRIS allows you to build REST APIs using ObjectScript classes and the %CSP.REST framework. This enables the development of modern services to expose data for web apps, mobile apps, or system integrations.

In this article, you'll learn how to create a basic REST API in InterSystems IRIS, including: