New post

#Data Import and Export

1

This topic unites posts which describe approaches, tools, and solutions to import and export data into InterSystems IRIS and other InterSystems Data platform products: CSV, JSON, SQL, flat files, binary files, globals, streams, etc.

Article
· Feb 24, 2024 2m read
The bewitched line terminator

I want to address the nasty problems about reading a flat text in ASCII, UTF*
explicitly excluding HTML, EBCDIC, and other encoding.
According to Wikipedia there are at least 8 variations of control characters.

  • CR+LF is typical for Windows
  • LF is typical for the Linux/UNIX world
  • CR is Mac's favorite

As you can deduct from the names the inspiration comes from mechanical typewriters.

25 0
1 464

Hello Developers!

Have you ever had to convert HL7v2 messages to FHIR (Fast Healthcare Interoperability Resources) and found the process complicated and confusing? InterSystems is rolling out a new cloud based SaaS offering called InterSystems FHIR Transformation Service, which makes the process easy. We are excited to announce an Early Access Preview Program for our new offering, and we would love to have you kick the tires and let us know what you think! All you need is a free AWS account, with an S3 bucket to drop in your HL7v2 messages, and another S3 bucket to get your FHIR output.

25 5
0 857
Article
· Jan 2, 2022 3m read
DB Migration using SQLgateway

Thanks to @Yuri Marx we have seen a very nice example for DB migration from Postgres to IRIS.
My personal problem is the use of DBeaver as a migration tool.
Especially as one of the strengths of IRIS ( and also Caché) before is the availability of the
SQLgateways that allow access to any external Db as long as for them an access usinig
JDBC or ODBC is available. So I extended the package to demonstrate this.

11 3
2 763

In recent versions of IRIS, a powerful new data loading command has been introduced to SQL: LOAD DATA. This feature has been highly optimized to import data into IRIS extremely fast, allowing hundreds of gigabytes of data to be inserted in seconds instead of hours or days.

This is a very exciting improvement. However, a big problem in the data loading experience still exists. Namely, the time and hassle it takes to:

Open Exchange app
10 6
0 890

Hi folks!

Sometimes we need to import data into InterSystems IRIS from CSV. It can be done e.g. via csvgen tool that generates a class and imports all the data into it.

But what if you already have your own class and want to import data from CSV into your existing table?

There are numerous ways to do that but you can use csvgen (or csvgen-ui) again! I prepared and and example and happy to share. Here we go!

8 2
0 1.2K
Article
· Feb 13 4m read
Bulk FHIR Step by Step

FHIR repositories, applications and servers typically serve clinical data in small quantities, whether to return data about a patient, their medications, vaccines, allergies, among other information. However, it is common for a large amount of data in FHIR/JSON format to be requested to be used to load into Data Lakes, identifying study cohorts, population health, or transferring data from one EHR to another. To meet these business scenarios that require large extractions and loads of data, it is recommended to use the FHIR Bulk Data Access feature provided by HL7 institution.

7 0
3 184

This article is a follow-up to the previous one on how to migrate from popular databases (like PostgreSQL and MySQL) to IRIS.
We will use the same procedures utilized to migrate from PostgreSQL. However, you will see that it is even easier since the data types in MySQL are very similar to IRIS. That is why we will not need to create transformation rules in the columns.

Get the sample data to the migration process

In GitHub it is possible to download a docker-compose project to build and run 2 databases:

6 0
0 704
Article
· Jan 13, 2022 4m read
How to find the dataset you need?

Hey community! How are you doing?

I hope to find everyone well, and a happy 2022 to all of you!

Over the years, I've been working on a lot of different projects, and I've been able to find a lot of interesting data.

But, most of the time, the dataset that I used to work with was the customer data. When I started to join the contest in the past couple of years, I began to look for specific web datasets.

I've curated a few data by myself, but I was thinking, "This dataset is enough to help others?"

Open Exchange app
5 4
0 391

Hi Community,

We are pleased to invite all the developers to the upcoming InterSystems Datasets Contest Kick-off Webinar! The topic of this webinar is dedicated to the Datasets contest.

In this webinar, we’ll do a quick tour of the new LOAD DATA feature, also chime in on packaging global data or file data with ZPM, and run a data-generation script as part of a method in the zpm install.

As always, our experts will answer the questions on how to develop, build, and deploy datasets using InterSystems IRIS.

Date & Time: Tuesday, December 28 – 10:00 AM EDT

Speakers:
🗣 @Benjamin De Boe, Product Manager, InterSystems
🗣 @Evgeny Shvarov, InterSystems Developer Ecosystem Manager

https://www.youtube.com/embed/jm7bDK0FoiI
[This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]

5 3
0 413

Hi Community,

Join the upcoming InterSystems Interoperability Contest Kick-off Webinar dedicated to the Interoperability Contest.

In this webinar, you'll get an overview of the interoperability capabilities of InterSystems IRIS, and we'll show you how to use the PEX. Also, we’ll discuss and answer your questions on how to build interoperability solutions using InterSystems IRIS and IRIS for Health.

Date & Time: Monday, October 4 — 12:00 AM EDT

Speakers:
🗣 @Stefan Wittmann, InterSystems Product Manager
🗣 @Bob Kuszewski, InterSystems Product Manager
🗣 @Evgeny Shvarov, InterSystems Developer Ecosystem Manager

5 5
0 367

I recently had the need to monitor from HealthConnect the records present in a NoSQL database in the Cloud, more specifically Cloud Firestore, deployed in Firebase. With a quick glance I could see how easy it would be to create an ad-hoc Adapter to make the connection taking advantage of the capabilities of Embedded Python, so I got to work.

Open Exchange app
4 2
1 430
Article
· Feb 13, 2023 3m read
MTConnect ObjectScript Implementation

Logo

In today's fast-paced and highly competitive manufacturing industry,
efficient machine communication and data exchange is essential to maximize
productivity and minimize downtime. That's where MTConnect comes in.
MTConnect is an open, royalty-free standard that provides a common language
for communication between machines, devices, and software applications in a
manufacturing environment.

Open Exchange app
4 0
0 342
Article
· Oct 19, 2022 3m read
Ingestion and Querying Speed Test

The capacity of taking numerous records every second while also facilitating real-time queries simultaneously in real time is called Hybrid Transactional Analytical Processing (HTAP). It is also called Transactional analytics or Transanalytics or Translytics and is a very useful element in scenarios where there is constant flow of real time data coming from IIOT sensors or data on fluctuations in stock market, and supporting the need for querying these data sets in real-time or near real-time.

4 4
0 846

This is the fourth part of the series of articles on migrating from the main databases on the market to InterSystems IRIS. In this part, the procedures for migrating from Microsoft SQL Server will be detailed. As described in previous articles, there are currently a few options to do the migration, but the two most popular options include the use of DBeaver (https://openexchange.intersystems.com/package/DBeaver) or SQLGateway.

4 0
2 784

Profiling CCD Documents with LEAD North’s CCD Data Profiler
Ever opened a CCD and been greeted by a wall of tangled XML? You’re not alone. Despite being a core format for clinical data exchange, CCD's are notoriously dense, verbose, and unfriendly to the human eye. For developers and analysts trying to validate their structure or extract meaningful insights, navigating these documents can feel more like archaeology than engineering.

Open Exchange app
3 0
0 48

The latest "Bringing Ideas to Reality" InterSystems competition saw me trawling through the ideas portal for UI problems to have a go at.

https://www.youtube.com/embed/zw51X1JQhQ0
[This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]

Open Exchange app
3 0
0 148

WIN SQL is the normal editor used by most of the users .But we can't download large amount of data using winsql . So I have written a tutorial how to connect with a new Java based editor called Squirrel SQL which can easily download or export data in excel or any other formats. Also I included a Java JCBC connection program to connect with the IRIS database especially a mirroring/failover server.

3 5
0 596

In this GitHub we gather information from a csv, use a DataTransformation to make it into a FHIR object and then, save that information to a FHIR server all that using only Python.

The objective is to show how easy it is to manipulate data into the output we want, here a FHIR Bundle, in the IRIS full Python framework.

Open Exchange app
3 3
0 574

Prefer not to read? Check out the demo video I created:

https://www.youtube.com/embed/-OwOAHC5b3s
[This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]

Open Exchange app
2 3
1 144