Article
· Oct 22 2m read
Tips on handling Large data

Hello community,

I wanted to share my experience about working on Large Data projects. Over the years, I have had the opportunity to handle massive patient data, payor data and transactional logs while working in an hospital industry. I have had the chance to build huge reports which had to be written using advanced logics fetching data across multiple tables whose indexing was not helping me write efficient code.

Here is what I have learned about managing large data efficiently.

Choosing the right data access method.

As we all here in the community are aware of, IRIS provides multiple ways to access data. Choosing the right method, depends on the requirement.

  • Direct Global Access: Fastest for bulk read/write operations. For example, if i have to traverse through indexes and fetch patient data, I can loop through the globals to process millions of records. This will save a lot of time.
Set ToDate=+H
Set FromDate=+$H-1 For  Set FromDate=$O(^PatientD("Date",FromDate)) Quit:FromDate>ToDate  Do
. Set PatId="" For  Set PatId=$Order(^PatientD("Date",FromDate,PatID)) Quit:PatId=""  Do
. . Write $Get(^PatientD("Date",FromDate,PatID)),!
  • Using SQL: Useful for reporting or analytical requirements, though slower for huge data sets.

3 6
1 88

In my previous article, Using LIKE with Variables and Patterns in SQL, we explored how the LIKE predicate behaves in different scenarios, from Embedded SQL to Dynamic SQL, and what happens to performance when wildcards and variables come into play. That piece was about getting comfortable writing a working LIKE query. But writing SQL that works is only the starting point. To build applications that are reliable, scalable, and secure, you need to understand the best practices that underpin all SQL, including queries that use LIKE.

This article takes the next step. We’ll look at a few key points to help strengthen your SQL code, avoid common pitfalls, and make sure your SELECT statements run not just correctly, but also efficiently and safely. I'll use SELECT statements with LIKE predicate as an example along the way, showing how these broader principles directly affect your queries and their results.

*This is what Gemini came up with for this article, kinda cute.

10 0
1 131

Hi Team,

Can I please check if anyone has built a simple web interface for maintaining custom SQL lookup class.

We have a simple persistent class in HealthShare which is used for storing Pathology test codes. Test codes in this lookup class is used for message filtering and applying additional logic when processing pathology results/orders.

0 2
0 67

Hi Community,

I’m trying to execute a directory query in InterSystems IRIS using %SQL.Statement, but encountering an unexpected error.

Details:
The following command confirms that the directory exists:

Set dirPath="\\MYNETWORK_DRIVE\DFS-Shared_Product\GXM"
Write ##class(%File).DirectoryExists(dirPath)

It returns 1, meaning the path is valid and accessible.

However, when I try to execute this SQL query:

1 3
0 54

The 2025.1.2 and 2024.1.5 maintenance releases of InterSystems IRIS® data platform, InterSystems IRIS® for HealthTM, and HealthShare® Health Connect are now Generally Available (GA). These releases include the fixes for a number of recently issued alerts and advisories, including the following:

2 0
0 49

Introduction

In a previous article, I presented the IRIStool module, which seamlessly integrates the pandas Python library with the IRIS database. Now, I'm explaining how we can use IRIStool to leverage InterSystems IRIS as a foundation for intelligent, semantic search over healthcare data in FHIR format.

This article covers what I did to create the database for another of my projects, the FHIR Data Explorer. Both projects are candidates in the current InterSystems contest, so please vote for them if you find them useful.

You can find them at the Open Exchange:

In this article we'll cover:

  • Connecting to InterSystems IRIS database through Python
  • Creating a FHIR-ready database schema
  • Importing FHIR data with vector embeddings for semantic search

1 0
0 43

Hey Community!

We're happy to share a new video from our InterSystems Developers YouTube:

Advanced SQL join table cardinality estimates @ Ready 2025

https://www.youtube.com/embed/8FE9v7xkEj8
[This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]

0 0
0 23