This topic unites posts which describe approaches, tools, and solutions to import and export data into InterSystems IRIS and other InterSystems Data platform products: CSV, JSON, SQL, flat files, binary files, globals, streams, etc.
In the previous article, we saw different modules in IRIS AI Studio and how it could help explore GenAI capabilities out of IRIS DB seamlessly, even for a non-technical stakeholder. In this article, we will deep dive into "Connectors" module, the one that enables users to seamlessly load data from local or cloud sources (AWS S3, Airtable, Azure Blob) into IRIS DB as vector embeddings, by also configuring embedding settings like model and dimensions.
In recent versions of IRIS, a powerful new data loading command has been introduced to SQL: LOAD DATA. This feature has been highly optimized to import data into IRIS extremely fast, allowing hundreds of gigabytes of data to be inserted in seconds instead of hours or days.
This is a very exciting improvement. However, a big problem in the data loading experience still exists. Namely, the time and hassle it takes to:
I recently had the need to monitor from HealthConnect the records present in a NoSQL database in the Cloud, more specifically Cloud Firestore, deployed in Firebase. With a quick glance I could see how easy it would be to create an ad-hoc Adapter to make the connection taking advantage of the capabilities of Embedded Python, so I got to work.
I want to address the nasty problems about reading a flat text in ASCII, UTF* explicitly excluding HTML, EBCDIC, and other encoding. According to Wikipedia there are at least 8 variations of control characters.
CR+LF is typical for Windows
LF is typical for the Linux/UNIX world
CR is Mac's favorite
As you can deduct from the names the inspiration comes from mechanical typewriters.
I am exporting data from Cache using the SQL export wizard. This is on a docker image of Cache hosted on a Linux server.
I select my database, schema, and table, make sure that all columns are being exported, check other settings, and then finish the export.
All server settings look correct and permissions are rwe on the directory. I have tried various directories, including a system level directory. I tried creating the file before I do the export in case there was an issue with creating the files, and I also made sure the created files had rwe permisisons.
Many organisations implement centralised log management systems to separate and centralise the log data in order to e.g. automate threat detection (and response) and to comply with regulatory requirements. The primary systems of interest are the various user facing applications, but increasingly also other kinds of systems including integration platforms.
Is it planned that LOAD DATA takes into account several DATE/DATETIME formats with, for example, a parameter indicating the format used in the source data?
example :
LOAD DATA .../...
USING
{
"from": {
"file": {
"dateformat": "DD/MM/YYYY"
}
}
}
I'm trying exported a project from version '2012.5' to '2018.1.4' but is returning ERROR #7602. Please, could you help me. How can I do it? Below is steps that I did:
We are retiring a hosted application for an electronic health care records (EHR) system which stored the data on Cache for UNIX (Red Hat Enterprise Linux for x86-64) 2017.2.2 (Build 867_4_20245) Thu Oct 8 2020 16:58:40 EDT. The hosting company is providing me with a single CBK file. I need to install a database system to restore the database and provide occasional SQL access for reports when necessary. I'll need to maintain access to the data for an approximately 10 year retention period. Not sure how to approach restoring this old of a database and eventually upgrading it to a newer re
We continue with the series of articles related to the QuinielaML application. In this article we are going to cover how to prepare the raw data that we have captured using the Embedded Python functionality.
I want to move a development IRIS for Health database to another server. I will do this manually for specific reasons. If I simply copy the /mgr folder along with all the files (.DAT, .GBK, etc.) and configure it in the new server, will it work?
I am trying to load all the data tables from one iris server to a client server but some of the tables data failing to load all the time. But I can load around 100 tables successfully but 8 to 10 tables are failing all the time. I made an IRIS odbc connection using odbc driver to load the data from tables.
Also I can see read server loop error message on the iris server side as the same time the table loading fails.
Please find the screen shot attached which shows the error on client server.
It can sometimes be useful to list or export all of the subclasses that are derived, directly or indirectly, from a given class. In Studio, the Class -> Derived Classes menu option will show such a list, but I'm not aware of a built-in API for programmatically exporting their source code.
https://www.youtube.com/embed/FnSPT1U9u9A [This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]
https://www.youtube.com/embed/aE743TEBgmw [This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]
Exported contents of a namespace on one server (classes, include files and lookup tables). Importing that code into a newly created namespace on another server. Both servers Ensemble 2018.1, same build. Export was via InterSystems Studio. Export is around 18Mb in total (XML file sizes).
When importing and compiling on the new server, getting errors as below - with #6301: SAX XML Parser error prominent - on a number of the imported files, all containing data transformations or business processes.
WIN SQL is the normal editor used by most of the users .But we can't download large amount of data using winsql . So I have written a tutorial how to connect with a new Java based editor called Squirrel SQL which can easily download or export data in excel or any other formats. Also I included a Java JCBC connection program to connect with the IRIS database especially a mirroring/failover server.
Is there a command that will loop through the flat files of a given Linux/Unix folder? I can write the code to open and read each file. But the file names are unknown. I am looking for a way to access each file given a named Linux folder. The files have differing structures so a record map will not work.
Thank you for reading and thank you even more for answering!
In today's fast-paced and highly competitive manufacturing industry,
efficient machine communication and data exchange is essential to maximize
productivity and minimize downtime. That's where MTConnect comes in.
MTConnect is an open, royalty-free standard that provides a common language
for communication between machines, devices, and software applications in a
manufacturing environment.
Is there a way to read JSON and transform it (in DTL) by using a VDOC of this JSON (without transform it to internal message) like I can do with HL7 or XML?
If it possible, I guess that I should have a schema of the JSON so the second question is how to build a schema for JSON and load it to the IRIS?
The capacity of taking numerous records every second while also facilitating real-time queries simultaneously in real time is called Hybrid Transactional Analytical Processing (HTAP). It is also called Transactional analytics or Transanalytics or Translytics and is a very useful element in scenarios where there is constant flow of real time data coming from IIOT sensors or data on fluctuations in stock market, and supporting the need for querying these data sets in real-time or near real-time.