Process CSV, Txt files from a filesystem/folder and load to Iris DB

Question

Question

Mark Retry · Nov 16, 2020

I am looking for any pointers on how Intersystems IRIS Health can monitor a filesystem/Folder that user/s /applications can drop in CSV files via FTP and load the file to the IRIS DB . I understand that I will need create a record map for the CSV files, I am looking for any configuration references on how how to process files using file inbound adapters with the intent to pick up the CSV file as they are dropped in the target location and pass it to a Business process and ingest into the IRIS database

Any help would be greatly appreciated ...

Discussion (10)1

Log in or sign up to continue

Oliver Wilms · Nov 16, 2020

Mark,

I cloned the csv repository, added an interoperability production with a file service and a process that imports the file using csvgen code. Please take a look at this repo and let me know if you have any comments or questions:

https://github.com/oliverwilms/csvgen

0 0

Evgeny Shvarov · Nov 17, 2020

Hi Mark!

CSVGEN creates a class and imports data into it vs a given CSV file or URL which could be done with one line. E.g. here is how to import fameous titanic data into IRIS:

zw ##class(community.csvgen).GenerateFromURL("https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv",",","Data.Titanic)

Or from file:

USER>zw ##class(community.csvgen).Generate("/folder/filename.csv")

And of course, this task could be automated - this is what the Interoperability feature of IRIS for. Check the sample Interoperability production template which periodically loads news from Reddit and filters it upon criteria and writes into files or sends via email. I bet we have a ton of examples of how to process folders with files, calling @Stefan Wittmann

0 0

Guillaume Rongier · Nov 17, 2020

If you need an how-to de develop an interoperability production check this git.

https://github.com/grongierisc/formation-template

It's in french but you will have a lot of gif to get throw the tuto and build a CSV parser and drop the file in the database with transformations.

0 0

score 0 · Answer 1 · 2020-11-16T16:36:38-05:00

Hi Mark,

Maybe this project can help you to achieve what you looking for.

CSV Gen
https://openexchange.intersystems.com/package/csvgen

CSV Gen import your csv and create a persistent class based in your csv content.

score 0 · Answer 2 · 2020-11-16T18:03:16-05:00

Mark Retry · Nov 16, 2020

We will need a process that's automated , users will be sending CSV file via FTP Can csvgen can be automated ?

0 0

score 1 · Answer 3 · 2020-11-17T05:56:40-05:00

Hi Mark and Evgeny,

there are plenty of options available to achieve what you are asking for. You can certainly pick one of the templates that others have pointed out in the individual replies here. Starting from scratch I would probably generate a Record Map using the CSV Wizard as described in the documentation here. Afterwards you can simply leverage the prebuilt FTPService or BatchFTPService that come with the Record Map utility. The Interoperability quick start includes a sample of the Record Map for importing a CSV file and might be a good reference as well.

score 0 · Answer 4 · 2020-11-17T08:04:20-05:00

Just adding a quick note that there is also a FileService and a BatchFileService in addition to the FTP services Stefan mentioned.

score 0 · Answer 5 · 2020-11-16T17:14:05-05:00

Hello Mark, I think you want to setup Interoperability Production. Where are you located? Are you available for phone call or online meeting?

score 0 · Answer 6 · 2020-11-17T01:56:51-05:00

Eduard Lebedyuk · Nov 17, 2020

Check:

0 0

score 0 · Answer 7 · 2020-11-30T12:35:49-05:00

So the process would be:

1) Configure the File inbound adapter, a designated filesystem (/target ) where users/application will drop in the CSV files throughout the day.

2) After the file adapter picks up the files , the CSV would be processed with a defined Record Mapper. ??

3) Once the Record Mapper processed the CSV's files, does it then need to go through a business service to get ingested into the IRIS DB ?