New post

#CSV

1

A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. The use of the comma as a field separator is the source of the name for this file format. A CSV file typically stores tabular data (numbers and text) in plain text, in which case each line will have the same number of fields.

Hi colleagues!

Every day Johns Hopkins University publishes new data on coronavirus COVID-19 pandemic status.

I built a simple InterSystems IRIS Analytics dashboard using InterSystems IRIS Community Edition in docker deployed on GCP Kubernetes which shows key measures of the disease outbreak.

This dashboard is an example of how information from CSV could be analyzed with IRIS Analytics and deployed to GCP Kubernetes in a form of InterSystems IRIS Community Edition.

Added the interactive map of the USA:

Open Exchange app
19 13
3 947

We have a yummy dataset with recipes written by multiple Reddit users, however most of the information is free text as the title or description of a post. Let's find out how we can very easily load the dataset, extract some features and analyze it using features from OpenAI large language model within Embedded Python and the Langchain framework.

9 2
2 165

In recent versions of IRIS, a powerful new data loading command has been introduced to SQL: LOAD DATA. This feature has been highly optimized to import data into IRIS extremely fast, allowing hundreds of gigabytes of data to be inserted in seconds instead of hours or days.

This is a very exciting improvement. However, a big problem in the data loading experience still exists. Namely, the time and hassle it takes to:

Open Exchange app
9 4
0 304

Hi folks!

Sometimes we need to import data into InterSystems IRIS from CSV. It can be done e.g. via csvgen tool that generates a class and imports all the data into it.

But what if you already have your own class and want to import data from CSV into your existing table?

There are numerous ways to do that but you can use csvgen (or csvgen-ui) again! I prepared and and example and happy to share. Here we go!

8 2
0 968

A short post for now to answer a question that came up. In post two of this series I included graphs of performance data extracted from pButtons. I was asked off-line if there is a quicker way than cut/paste to extract metrics for mgstat etc from a pButtons .html file for easy charting in Excel.

See: - Part 2 - Looking at the metrics we collected

7 2
0 1.4K

Hi Developers!

Sometimes we need to import CSV data programmatically to InterSystems IRIS either from CSV or from URL. And we expect the class with proper datatypes to be created and the data to be imported.

I published a module csvgen on Open Exchange which does exactly that.

If you just need the CSV file be imported into IRIS you can do the following:

USER>do ##class(community.csvgen).Generate("/usr/data/titanic.csv",,"Data.Titanic")

Class name: Data.Titanic
Header: PassengerId INTEGER,Survived INTEGER,Pclass INTEGER,Name VARCHAR(250),Sex VARCHAR(250),Age INTEGER,SibSp INTEGER,Parch INTEGER,Ticket VARCHAR(250),Fare MONEY,Cabin VARCHAR(250),Embarked VARCHAR(250)
Records imported: 891
USER>

Or if you have the CSV on the internet, e.g. COVID-19 Data on Github you can get the data in the following way:

USER>d ##class(community.csvgen).GenerateFromURL("https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/05-29-2020.csv",",","Data.Covid19")

Class name: Data.Covid19
Header: FIPS INTEGER,Admin2 VARCHAR(250),Province_State VARCHAR(250),Country_Region VARCHAR(250),Last_Update DATE,Lat MONEY,Long_ DOUBLE,Confirmed INTEGER,Deaths INTEGER,Recovered INTEGER,Active INTEGER,Combined_Key VARCHAR(250),Incidence_Rate DOUBLE,Case-Fatality_Ratio DOUBLE
Records imported: 3522
USER>

Open Exchange app
7 16
1 1.2K

Hey Developers,

Enjoy watching this brief explanation of a new feature in IRIS RAD Studio – a new project by @José Pereira and @Henrique.GonçalvesDias:

⏯ Automated user management in IRIS RAD Studio

https://www.youtube.com/embed/-Y2kaKES8yg
[This is an embedded link, but you cannot view embedded content directly on the site because you have declined the cookies necessary to access it. To view embedded content, you would need to accept all cookies in your Cookies Settings]

Open Exchange app
6 0
0 248

The last days I've work with the great new feature: LOAD DATA With this post I would like to share my first experiences with you. The following points do not contain any order or other evaluation. These are only things that I noticed when using the LOAD DATA command. It should also be noted that these points are based on the IRIS Version 2021.2.0.617 which is a preview release. So it may be that my observations do not apply to newer IRIS versions.

Open Exchange app
6 5
2 949

Hi Devs!

Last weekend I had been testing the newborn csvgen module and was looking for a CSV file to test - thus I came across an interesting datafile on Data.World with Game of Throne episodes statistics. Death statistics. These folks documented all the murders through all the 8 seasons and noted where, who, from what clan with what weapon had killed another one.

So I imported it and made an IRIS Analytics dashboard.

You Know Nothing, Jon Snow | You Know Nothing, Jon Snow | Know ...

Don't worry, Jon, with this dashboard we can figure out something ). See the details below.

Open Exchange app
6 0
2 567
Announcement
· Oct 2, 2020
Csvgen-UI

Introduction

This is a csvgen UI frontend in Angular 8.
The aim of this project is to easily import csv file to Iris from a web ui.

Demo

Demo

Install with ZPM

It will automatically install the dependency of csvgen and sslclient

zpm "install csvgen-ui"

This will create two new endpoints :

  • One for the Rest API for uploading files

Open Exchange app
4 2
0 323

Hi!

I believe the simplest is (to work with csv delimited by ";"):


set file = ##class(%File).%New( "data.csv" )
    set sc = file.Open( "R" ) 
    if $$$ISERR(sc) quit    ; or do smth

    while 'file.AtEnd {
        set str=file.ReadLine() 
        for i=1:1:$length( str, ";" ) {
            set id=$piece( str, ";" ,i ) 
            write !, id  // or do smth
        }
    }
    do file.Close()

Possible options:

different variants of error handling with sc code.

Embrace while loop into try/catch block.

And what's yours?

3 16
0 6.5K

In this GitHub we gather information from a csv, use a DataTransformation to make it into a FHIR object and then, save that information to a FHIR server all that using only Python.

The objective is to show how easy it is to manipulate data into the output we want, here a FHIR Bundle, in the IRIS full Python framework.

Open Exchange app
3 3
0 459

Tiny web application that allows you to anonymize CSV files. Based on InterSystems IRIS®

Version: store/intersystems/iris-community:2020.3.0.221.0

Application

Sample CSV where Date of Birth and Sex will remain untouched, while the rest should be anonymized.

before

The application recognizes the header columns and allows the user to chose which ones to ignore.

Open Exchange app
2 0
0 267

Hi Guys,

have a file located in \\servername\Myfiles\pull.csv but for some reason my cache routine can't file, but can when using the local drive C:\servername\Myfiles\pull.csv.

FYI I can access \\servername\Myfiles\pull.csv via Win explorer with no problems.

anyway how I can fix this?

Thanks Guys

0 4
0 489
Question
· Sep 17, 2020
Bulk load csv into global

Is there a way to bulk load csv into a global? I have a csv with 283 lines in the following pattern

123, first text

234, second text

456, third text

I want to load them into a global ^loader(code, text). Is there a way to upload them using code?

0 4
0 316

Hi folks!

Have a question for those who are masters of interoperability.

I have a basic task of having one CSV with some data. I need to transform one column in the initial dataset and get the new csv with the same form.

What's the best approach with Interoperability?

Should I user record mapper?

Should I use streams, objects?

What is the best practice?

0 12
0 511