Question
Mark Retry · Nov 16, 2020

Process CSV, Txt files from a filesystem/folder and load to Iris DB

 I am looking for any pointers on how Intersystems IRIS Health can monitor a filesystem/Folder that user/s /applications can drop in CSV files via FTP and load the file  to the IRIS DB . I understand that I will need create a record map for the CSV files, I am looking for any configuration references on how how to process files using file inbound adapters  with the  intent to pick up the CSV  file as they are dropped in the  target location and pass it to a Business process and ingest into the IRIS database 

Any help would be greatly appreciated ... 

#InterSystems IRIS for Health
00
3 0 10 180
Log in or sign up to continue

Replies

Mark Retry · Nov 16, 2020

We will need a process that's automated , users will be sending CSV file via FTP  Can csvgen can be automated ?

00
Oliver Wilms · Nov 16, 2020

Mark,

I cloned the csv repository, added an interoperability production with a file service and a process that imports the file using csvgen code. Please take a look at this repo and let me know if you have any comments or questions:

https://github.com/oliverwilms/csvgen

00
Evgeny Shvarov · Nov 17, 2020

Hi Mark!

CSVGEN creates a class and imports data into it vs a given CSV file or URL which could be done with one line. E.g. here is how to import fameous titanic data into IRIS: 

zw ##class(community.csvgen).GenerateFromURL("https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv",",","Data.Titanic)

Or from file:

USER>zw ##class(community.csvgen).Generate("/folder/filename.csv")

And of course, this task could be automated - this is what the Interoperability feature of IRIS for. Check the sample Interoperability production template which periodically loads news from Reddit and filters it upon criteria and writes into files or sends via email.  I bet we have a ton of examples of how to process folders with files, calling @Stefan Wittmann 

00
Stefan Wittmann · Nov 17, 2020

Hi Mark and Evgeny,

there are plenty of options available to achieve what you are asking for. You can certainly pick one of the templates that others have pointed out in the individual replies here. Starting from scratch I would probably generate a Record Map using the CSV Wizard as described in the documentation here.  Afterwards you can simply leverage the prebuilt FTPService or BatchFTPService that come with the Record Map utility. The Interoperability quick start includes a sample of the Record Map for importing a CSV file and might be a good reference as well.

10
Oliver Wilms · Nov 16, 2020

Hello Mark, I think you want to setup Interoperability Production. Where are you located? Are you available for phone call or online meeting?

00
Mark Retry · Nov 30, 2020

So the process would be:

1) Configure the File inbound adapter,  a designated filesystem (/target ) where users/application will drop in the CSV files throughout the day.

2) After the file adapter picks up the files , the CSV would be processed with a defined Record Mapper. ??

3) Once the Record Mapper processed  the CSV's files, does it then need to go through a business service to get ingested into the IRIS DB ?

00