New post

#Data Import and Export

1

This topic unites posts which describe approaches, tools, and solutions to import and export data into InterSystems IRIS and other InterSystems Data platform products: CSV, JSON, SQL, flat files, binary files, globals, streams, etc.

Hi folks!

Sometimes we need to import data into InterSystems IRIS from CSV. It can be done e.g. via csvgen tool that generates a class and imports all the data into it.

But what if you already have your own class and want to import data from CSV into your existing table?

There are numerous ways to do that but you can use csvgen (or csvgen-ui) again! I prepared and and example and happy to share. Here we go!

8 2
0 1.1K

In recent versions of IRIS, a powerful new data loading command has been introduced to SQL: LOAD DATA. This feature has been highly optimized to import data into IRIS extremely fast, allowing hundreds of gigabytes of data to be inserted in seconds instead of hours or days.

This is a very exciting improvement. However, a big problem in the data loading experience still exists. Namely, the time and hassle it takes to:

Open Exchange app
10 6
0 853
Article
· Oct 19, 2022 3m read
Ingestion and Querying Speed Test

The capacity of taking numerous records every second while also facilitating real-time queries simultaneously in real time is called Hybrid Transactional Analytical Processing (HTAP). It is also called Transactional analytics or Transanalytics or Translytics and is a very useful element in scenarios where there is constant flow of real time data coming from IIOT sensors or data on fluctuations in stock market, and supporting the need for querying these data sets in real-time or near real-time.

4 4
0 826

This is the fourth part of the series of articles on migrating from the main databases on the market to InterSystems IRIS. In this part, the procedures for migrating from Microsoft SQL Server will be detailed. As described in previous articles, there are currently a few options to do the migration, but the two most popular options include the use of DBeaver (https://openexchange.intersystems.com/package/DBeaver) or SQLGateway.

4 0
2 751
Article
· Jan 2, 2022 3m read
DB Migration using SQLgateway

Thanks to @Yuri Marx Pereira Gomes we have seen a very nice example for DB migration from Postgres to IRIS.
My personal problem is the use of DBeaver as a migration tool.
Especially as one of the strengths of IRIS ( and also Caché) before is the availability of the
SQLgateways that allow access to any external Db as long as for them an access usinig
JDBC or ODBC is available. So I extended the package to demonstrate this.

10 3
2 733
Article
· Jun 4, 2020 1m read
Easy data import into IRIS

Sometimes you need quickly and easily import data into IRIS. For this, an IRIS import manager has been developed.

This application allows you to import JSON data and also provides a really simple interface for transferring data from MongoDB collections to IRIS globals. It has never been so easy.

Let's look at examples.

Import JSON

Open Exchange app
2 5
0 700

This article is a follow-up to the previous one on how to migrate from popular databases (like PostgreSQL and MySQL) to IRIS.
We will use the same procedures utilized to migrate from PostgreSQL. However, you will see that it is even easier since the data types in MySQL are very similar to IRIS. That is why we will not need to create transformation rules in the columns.

Get the sample data to the migration process

In GitHub it is possible to download a docker-compose project to build and run 2 databases:

6 0
0 683

This is the third part of the series of articles on migrating from the main databases on the market to InterSystems IRIS. In this part, the procedures for migrating from DB2 will be detailed.As described in previous articles, there are currently a few options to do the migration. However, the two most popular options include the usage of DBeaver (https://openexchange.intersystems.com/package/DBeaver) or SQLGateway.

2 0
1 573

WIN SQL is the normal editor used by most of the users .But we can't download large amount of data using winsql . So I have written a tutorial how to connect with a new Java based editor called Squirrel SQL which can easily download or export data in excel or any other formats. Also I included a Java JCBC connection program to connect with the IRIS database especially a mirroring/failover server.

3 5
0 559

In this GitHub we gather information from a csv, use a DataTransformation to make it into a FHIR object and then, save that information to a FHIR server all that using only Python.

The objective is to show how easy it is to manipulate data into the output we want, here a FHIR Bundle, in the IRIS full Python framework.

Open Exchange app
3 3
0 554
Article
· May 13, 2016 1m read
mySQL data importer tool

Hi,

If you want to import data from a mySQL export file (exported with mysqldump), you will find here a little script that could help.

Only the INSERT commands in the sql file are executed into Caché. Indices are not computed for better performance.
%NOINDEX, %NOCHECK and %NOLOCK are generated on each INSERT line.

Currently, the file can not contain a "),(" pattern inside the values part of the INSERT command. If this is the case, the line is skipped. This feature may be implemented in the extractValuesList method.

2 3
0 554

Now Sapphire enable you load CSV to IRIS. See the steps:

1) Create a sample CSV file using Excel (save file as CSV):

2) Follow these instructions to install Sapphire into your enviroment: https://openexchange.intersystems.com/package/SAPPHIRE

3) Access Sapphire web page. Go to top menu Import > Load CSV

Open Exchange app
1 1
2 536
Article
· Jun 30, 2020 3m read
Replicating Audit Log Near Real Time
Many organisations implement centralised log management systems to separate and centralise the log data in order to e.g. automate threat detection (and response) and to comply with regulatory requirements. The primary systems of interest are the various user facing applications, but increasingly also other kinds of systems including integration platforms.
1 2
2 442
Article
· Feb 24, 2024 2m read
The bewitched line terminator

I want to address the nasty problems about reading a flat text in ASCII, UTF*
explicitly excluding HTML, EBCDIC, and other encoding.
According to Wikipedia there are at least 8 variations of control characters.

  • CR+LF is typical for Windows
  • LF is typical for the Linux/UNIX world
  • CR is Mac's favorite

As you can deduct from the names the inspiration comes from mechanical typewriters.

25 0
1 442

Hi developers!

Often we need to deploy some data along with code pieces of the application.

And for InterSystems IRIS developers the question could sound: "How can I deploy the data I have in globals?"

InterSystems IRIS Globals Model QuickStart | InterSystems

Here I want to suggest to you one of the approaches - deploying global data using the ZPM package manager.

Open Exchange app
1 0
0 419

I recently had the need to monitor from HealthConnect the records present in a NoSQL database in the Cloud, more specifically Cloud Firestore, deployed in Firebase. With a quick glance I could see how easy it would be to create an ad-hoc Adapter to make the connection taking advantage of the capabilities of Embedded Python, so I got to work.

Open Exchange app
4 2
1 407
Article
· Jan 13, 2022 4m read
How to find the dataset you need?

Hey community! How are you doing?

I hope to find everyone well, and a happy 2022 to all of you!

Over the years, I've been working on a lot of different projects, and I've been able to find a lot of interesting data.

But, most of the time, the dataset that I used to work with was the customer data. When I started to join the contest in the past couple of years, I began to look for specific web datasets.

I've curated a few data by myself, but I was thinking, "This dataset is enough to help others?"

Open Exchange app
5 4
0 379
Article
· Feb 13, 2023 3m read
MTConnect ObjectScript Implementation

Logo

In today's fast-paced and highly competitive manufacturing industry,
efficient machine communication and data exchange is essential to maximize
productivity and minimize downtime. That's where MTConnect comes in.
MTConnect is an open, royalty-free standard that provides a common language
for communication between machines, devices, and software applications in a
manufacturing environment.

Open Exchange app
4 0
0 327

The latest "Bringing Ideas to Reality" InterSystems competition saw me trawling through the ideas portal for UI problems to have a go at.

https://www.youtube.com/embed/zw51X1JQhQ0
[This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]

Open Exchange app
3 0
0 123

Prefer not to read? Check out the demo video I created:

https://www.youtube.com/embed/-OwOAHC5b3s
[This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]

Open Exchange app
3 2
1 119
Article
· Feb 13 4m read
Bulk FHIR Step by Step

FHIR repositories, applications and servers typically serve clinical data in small quantities, whether to return data about a patient, their medications, vaccines, allergies, among other information. However, it is common for a large amount of data in FHIR/JSON format to be requested to be used to load into Data Lakes, identifying study cohorts, population health, or transferring data from one EHR to another. To meet these business scenarios that require large extractions and loads of data, it is recommended to use the FHIR Bulk Data Access feature provided by HL7 institution.

7 0
3 111