This topic unites posts which describe approaches, tools, and solutions to import and export data into InterSystems IRIS and other InterSystems Data platform products: CSV, JSON, SQL, flat files, binary files, globals, streams, etc.
I want to address the nasty problems about reading a flat text in ASCII, UTF* explicitly excluding HTML, EBCDIC, and other encoding. According to Wikipedia there are at least 8 variations of control characters.
CR+LF is typical for Windows
LF is typical for the Linux/UNIX world
CR is Mac's favorite
As you can deduct from the names the inspiration comes from mechanical typewriters.
Thanks to @Yuri Marx Pereira Gomes we have seen a very nice example for DB migration from Postgres to IRIS. My personal problem is the use of DBeaver as a migration tool. Especially as one of the strengths of IRIS ( and also Caché) before is the availability of the SQLgateways that allow access to any external Db as long as for them an access usinig JDBC or ODBC is available. So I extended the package to demonstrate this.
In recent versions of IRIS, a powerful new data loading command has been introduced to SQL: LOAD DATA. This feature has been highly optimized to import data into IRIS extremely fast, allowing hundreds of gigabytes of data to be inserted in seconds instead of hours or days.
This is a very exciting improvement. However, a big problem in the data loading experience still exists. Namely, the time and hassle it takes to:
Sometimes we need to import data into InterSystems IRIS from CSV. It can be done e.g. via csvgen tool that generates a class and imports all the data into it.
But what if you already have your own class and want to import data from CSV into your existing table?
There are numerous ways to do that but you can use csvgen (or csvgen-ui) again! I prepared and and example and happy to share. Here we go!
FHIR repositories, applications and servers typically serve clinical data in small quantities, whether to return data about a patient, their medications, vaccines, allergies, among other information. However, it is common for a large amount of data in FHIR/JSON format to be requested to be used to load into Data Lakes, identifying study cohorts, population health, or transferring data from one EHR to another. To meet these business scenarios that require large extractions and loads of data, it is recommended to use the FHIR Bulk Data Access feature provided by HL7 institution.
Sometimes it is necessary to transfer or migrate data and data schema from Postgres to IRIS. There are currently a few options for doing this, but the two most popular options are using DBeaver (https://openexchange.intersystems.com/package/DBeaver) or SQLGateway.
This article is a follow-up to the previous one on how to migrate from popular databases (like PostgreSQL and MySQL) to IRIS. We will use the same procedures utilized to migrate from PostgreSQL. However, you will see that it is even easier since the data types in MySQL are very similar to IRIS. That is why we will not need to create transformation rules in the columns.
Get the sample data to the migration process
In GitHub it is possible to download a docker-compose project to build and run 2 databases:
We continue with the series of articles related to the QuinielaML application. In this article we are going to cover how to prepare the raw data that we have captured using the Embedded Python functionality.
I hope to find everyone well, and a happy 2022 to all of you!
Over the years, I've been working on a lot of different projects, and I've been able to find a lot of interesting data.
But, most of the time, the dataset that I used to work with was the customer data. When I started to join the contest in the past couple of years, I began to look for specific web datasets.
I've curated a few data by myself, but I was thinking, "This dataset is enough to help others?"
The capacity of taking numerous records every second while also facilitating real-time queries simultaneously in real time is called Hybrid Transactional Analytical Processing (HTAP). It is also called Transactional analytics or Transanalytics or Translytics and is a very useful element in scenarios where there is constant flow of real time data coming from IIOT sensors or data on fluctuations in stock market, and supporting the need for querying these data sets in real-time or near real-time.
In today's fast-paced and highly competitive manufacturing industry,
efficient machine communication and data exchange is essential to maximize
productivity and minimize downtime. That's where MTConnect comes in.
MTConnect is an open, royalty-free standard that provides a common language
for communication between machines, devices, and software applications in a
manufacturing environment.
I recently had the need to monitor from HealthConnect the records present in a NoSQL database in the Cloud, more specifically Cloud Firestore, deployed in Firebase. With a quick glance I could see how easy it would be to create an ad-hoc Adapter to make the connection taking advantage of the capabilities of Embedded Python, so I got to work.
This is the fourth part of the series of articles on migrating from the main databases on the market to InterSystems IRIS. In this part, the procedures for migrating from Microsoft SQL Server will be detailed. As described in previous articles, there are currently a few options to do the migration, but the two most popular options include the use of DBeaver (https://openexchange.intersystems.com/package/DBeaver) or SQLGateway.
In this GitHub we gather information from a csv, use a DataTransformation to make it into a FHIR object and then, save that information to a FHIR server all that using only Python.
The objective is to show how easy it is to manipulate data into the output we want, here a FHIR Bundle, in the IRIS full Python framework.
WIN SQL is the normal editor used by most of the users .But we can't download large amount of data using winsql . So I have written a tutorial how to connect with a new Java based editor called Squirrel SQL which can easily download or export data in excel or any other formats. Also I included a Java JCBC connection program to connect with the IRIS database especially a mirroring/failover server.
The latest "Bringing Ideas to Reality" InterSystems competition saw me trawling through the ideas portal for UI problems to have a go at.
https://www.youtube.com/embed/zw51X1JQhQ0 [This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]
If you want to import data from a mySQL export file (exported with mysqldump), you will find here a little script that could help.
Only the INSERT commands in the sql file are executed into Caché. Indices are not computed for better performance. %NOINDEX, %NOCHECK and %NOLOCK are generated on each INSERT line.
Currently, the file can not contain a "),(" pattern inside the values part of the INSERT command. If this is the case, the line is skipped. This feature may be implemented in the extractValuesList method.
Sometimes you need quickly and easily import data into IRIS. For this, an IRIS import manager has been developed.
This application allows you to import JSON data and also provides a really simple interface for transferring data from MongoDB collections to IRIS globals. It has never been so easy.
https://www.youtube.com/embed/-OwOAHC5b3s [This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]
This is the third part of the series of articles on migrating from the main databases on the market to InterSystems IRIS. In this part, the procedures for migrating from DB2 will be detailed.As described in previous articles, there are currently a few options to do the migration. However, the two most popular options include the usage of DBeaver (https://openexchange.intersystems.com/package/DBeaver) or SQLGateway.
Many organisations implement centralised log management systems to separate and centralise the log data in order to e.g. automate threat detection (and response) and to comply with regulatory requirements. The primary systems of interest are the various user facing applications, but increasingly also other kinds of systems including integration platforms.
By likes
Open Exchange app
.png)
.png)
