Article
Yuri Marx · Oct 17 10m read

Using FTP Adapter to produce and consume FTP messages

FTP (File Transfer Protocol) is a network protocol for transmitting files over TCP/IP connections in a network (including the Internet) configured to transfer files via this protocol. In an FTP transaction, a file sender is called a local host. A file receiver involved in FTP is a remote host, and it is usually a server. Although many file transfers can be conducted using Hypertext Transfer Protocol (HTTP), FTP is still commonly used to transfer files behind the scenes for other applications, such as banking services. Consequently, we can say that FTP is an excellent option for promoting file-based interoperability between systems located in the same local network or on the Internet (an FTP server can be accessed via the Internet). This interoperability can be enriched, processed, and orchestrated by IRIS Interoperability. To understand how IRIS can help, it is crucial to understand the functionality of the IRIS Interoperability Architecture for FTP, centered on the FTP Inbound and Outbound Adapters.

FTP Inbound Adapter

According to InterSystems documentation (https://docs.intersystems.com/iris20221/csp/docbook/DocBook.UI.Page.cls?...), the EnsLib.FTP.InboundAdapter enables InterSystems IRIS to receive files via the FTP protocol. The adapter receives FTP input from the configured location, reads it, and sends it as a stream to the associated business service. The business service that you create and configure uses this stream and communicates with the rest of the production. The following figure shows the overall flow of inbound messages (excluding responses):


Diagram showing an FTP file flowing from outside a productiion through and FTP inbound adapter and business service

In more detail:

  1. Each time the adapter encounters an input from its configured data source, it calls the internal ProcessInput() method of the business service class, passing the stream as an input argument.
  2. The internal ProcessInput() method of the business service class executes it. This method performs basic production tasks related to maintaining internal information in a way required by all business services. You do not need to customize or override the method that your business service class inherits.
  3. The ProcessInput() method then calls your custom OnProcessInput() method, passing the stream object as an input. Now, with the Stream, it is possible to develop the rules required by the business for the received files.
  4. The response message follows the same path but in reverse.

FTP Outbound Adapter

According to InterSystems documentation (https://docs.intersystems.com/iris20221/csp/docbook/DocBook.UI.Page.cls?...), the EnsLib.FTP.OutboundAdapter enables production to send files via the FTP protocol.To use this adapter, you need to create and configure a business operation specifically designed for it. This business operation will then receive a message from within the production, look up the message type, and execute the appropriate method. This method usually executes rules to generate files with the content and format required by the business to send to an application.

Most common FTP usage scenarios for IRIS interoperability

The following table summarizes the main FTP usage scenarios for IRIS interoperability:

FTP Adapter Type Scenario
Inbound

Receive business batch data inside CSV, XML, TXT, and JSON files

Outbound Send the results of business batch data processing into CSV, XML, TXT, and JSON from data sent previously
Inbound ETL (Extract, Transform and Load) process of big files like parquet, ORC, and avro for Data Lakes and Data Marts
Outbound Generate parquet, ORC, and avro files from IRIS databases to send to external Data Lakes and Data Marts
Inbound Asynchronous messages and files consumption from external systems
Outbound Asynchronous messages and files production to external systems
Outbound Send/publish generated files (PDF reports, Scanned Files, Images, etc.) to organizational remote folders

Sample Application

The IRIS-FTP-sample (https://openexchange.intersystems.com/package/iris-ftp-sample) is a sample of how to use the FTP Adapter (Inbound and Outbound).

The production

This sample has an interoperability production to:
Receive CSV data using FTP Inbound Adapter into a LoadCSVFTPBusinessService send its content to a LoadCSVFTPBusinessOperation Business Operation to persist the content in a Persistent class using CSVgen.
Do CDC (Change Data Capture) on the Payment table using SQL Inbound Adapter into a PaymentSQLBusinessService Business Operation and send table data (inside a JSON file) to a SendSQLDataToFTPBusinessOperation Business for putting the JSON file into and FTP server.

Here you can see this production:

FTP Sample Production

 

Install the Sample

To install the sample using ZPM, follow these steps:

  1. Open the IRIS Namespace with Interoperability Enabled.
  2. Open Terminal and call: USER>zpm "install iris-ftp-sample"

To install the sample using Docker, do the next:

1. Clone/git pull the repo into any local directory:

$ git clone https://github.com/yurimarx/iris-ftp-sample.git

2. Open the terminal in this directory and run:

$ docker-compose build

3. Run the IRIS container with your project:

$ docker-compose up -d

Run the Sample

To run the sample, complete these procedures:

1. Create the credentials to access the FTP server (Go to Interoperability > Configure > Credentials):

  • ID: FTPCredentials
  • User Name: irisuser
  • Password: sys

Here is the result:

Credentials for acessing the FTP server

2. Open the production and start it, as shown in the picture below:

Credentials for acessing the FTP server

Send a File using the FTP client and see the data loaded in an SQL Table

1. Open an FTP client (I usually use Filezilla to do that) and put the input/countries.csv file into the root FTP folder:

  • Host: localhost
  • User name: irisuser
  • Password: sys

The image beneath indicates how to put the file in the FTP server root folder:

FTP Client

2. Go to the System Explorer > SQL and type SELECT country, latitude, longitude, name FROM dc_irisftpsample.Country. Consult the picture below to see how the CSV data loaded into the SQL table should look like:

Select csv results

Insert data in a table and a file with the content inserted in a file on the FTP Server

1. Go to the System Explorer > SQL and type insert into dc_irisftpsample.Payment(amount payer, receiver, transactiondate) values(100.0,'Yuri','Fabiana', CURRENT_TIMESTAMP). Here you can see how the JSON file with the data inserted into the FTP server should appear on the screen:


Select csv results

 

Behind the scenes: the source code

Main files and folders

Inside the folder iris-ftp-sample we have:

File

Description

Readme.md

Information about the sample and installation/usage instructions

Module.xml

ZPM installation manifest

Dockerfile

Docker commands to create an InterSystems IRIS container

Docker-compose.yml

Commands to create a composition of 2 docker instances: 1 docker instance of FTP server and 1 docker instance of InterSystems IRIS in the same Docker network

Input/countries.csv

Sample file to be used to test data load using FTP and csvgen with a sample FTP production

src/dc/irisftpsample

The source code folder in the package dc.irisftpsample

FTPSampleProduction.cls

The source code for the FTP Production (container to run business services and operations using FTP inbound and outbound adapters)

LoadCSVFTPBusinessService.cls

ObjectScript class which uses an Inbound FTP Adapter to receive CSV files and send their content to the LoadCSVFTPBusinessOperation.cls

LoadCSVFTPBusinessOperation.cls

Receives CSV file content and loads its data to an SQL table using CSVgen

SendSQLDataToFTPBusinessOperation.cls

Receives a JSON file with the content of the table Payment from PaymentSQLBusinessService and sends this JSON file to the FTP server

Payment.cls

Persistent Class (Payment SQL table) for persistent payment data to be sent to the FTP server

Sample use case 1: Receive a CSV file from the FTP server and load CSV data in an SQL Table

For this use case we have the following sequence:


Texto Descrição gerada automaticamente

 

1. FTP Inbound Adapter reads the CSV file and deletes it from the FTP server using these settings:

  • File Path is where the FTP Adapter will connect to find files
  • Delete From Server will delete the file from the Server after reading it
  • File Spec is the pattern to match the files that will be read
  • Call Interval is the interval of the pulling (check for new files)
  • FTP server is the name of the IP of the Server in the network or Internet
  • FTP Port is the port to connect
  • Credentials are the Credentials reference to username/password configuration

Interface gráfica do usuário, Texto, Aplicativo Descrição gerada automaticamente

Interface gráfica do usuário, Aplicativo Descrição gerada automaticamente

2. Send the Stream of a CSV file to the OnProcessInput method of the LoadCSVFTPBusinessService

3. Send the Stream of a CSV file into an Ens.StreamContainer message in the LoadCSVFTPBusinessOperation

Class dc.irisftpsample.LoadCSVFTPBusinessService Extends Ens.BusinessService
{
 
Parameter ADAPTER = "EnsLib.FTP.InboundAdapter";
Method OnProcessInput(pInput As %Stream.Object, pOutput As %RegisteredObject) As %Status
{
    Set tSC = $$$OK, 
    tSource = pInput.Attributes("Filename"),
    tFileLocation = pInput.Attributes("FTPDir"),
    pInput=##class(Ens.StreamContainer).%New(pInput)
 
    Set tSC = ..SendRequestSync("LoadCSVFTPBusinessOperation", pInput, .pOutput, -1)
    
    Quit tSC
}
 
}

4. Send the Stream of a CSV file into an Ens.StreamContainer message to the method LoadCSVIntoTable. The adapter uses the XData mapping to know the method which should be called for each type of message inside the class dc.irisftpsample.LoadCSVFTPBusinessOperation:

XData MessageMap
{
<MapItems>
    <MapItem MessageType="Ens.StreamContainer">
        <Method>LoadCSVIntoTable</Method>
    </MapItem>
</MapItems>
}

5. Send the Stream as a temp CSV file and call Generate method to place the CSV file to the SQL Table dc.irisftpsample.Country

Class dc.irisftpsample.LoadCSVFTPBusinessOperation Extends Ens.BusinessOperation
{
 
Parameter ADAPTER = "Ens.OutboundAdapter";
Parameter INVOCATION = "Queue";
Property Header As %Boolean [ InitialExpression = 1 ];
Property Delimiter As %String [ InitialExpression = "," ];
Property Classname As %String(MAXLEN = "");
Property Prowtype As %String(MAXLEN = "");
Property GuessTypes As %Boolean [ InitialExpression = 1 ];
Property Append As %Boolean [ InitialExpression = 1 ];
Property UseLoadData As %Boolean [ InitialExpression = 1 ];
Parameter SETTINGS = "Header:Basic,Delimiter:Basic,Classname:Basic,Prowtype:Basic,GuessTypes:Basic,Append:Basic,UseLoadData:Basic";
Method LoadCSVIntoTable(pInput As Ens.StreamContainer, Output pOutput As Ens.StringResponse) As %Status
{
    
    Set tSC = $$$OK
    Set pOutput = ##class(Ens.StringResponse).%New()
 
    Try {
        Set tSC = ##class(community.csvgen).StreamToFile(pInput.Stream, .pFile)
        Set tSC = ##class(community.csvgen).Generate(pFile, ..Delimiter, ..Classname, ..Prowtype, ..GuessTypes, ..Append, ..UseLoadData, ..Header)
        Set pOutput.StringValue = "OK"
    } Catch err {
        Set pOutput.StringValue = $System.Status.GetOneErrorText(tSC, 1)
    }
 
    Quit tSC
}
 
XData MessageMap
{
<MapItems>
    <MapItem MessageType="Ens.StreamContainer">
        <Method>LoadCSVIntoTable</Method>
    </MapItem>
</MapItems>
}
 
}
  • Note 1: The Parameter Adapter is used to choose the Adapter to be used by the Business Operation.
  • Note 2: Such parameters as Header, Delimiter, Classname, Prowtype, GuessTypes, Append, and UseLoadData are required by csvgen and configured by the Production user because these parameters are referenced in SETTINGS parameter in the section Basic:

Interface gráfica do usuário, Aplicativo Descrição gerada automaticamente

Uma imagem contendo Interface gráfica do usuário Descrição gerada automaticamente

Check out the details about CSVgen required information on https://github.com/evshvarov/csvgen.

 

Sample use case 2: Applying CDC (Change Data Capture) on the Payment table using SQL Inbound Adapter and sending table data to a Business Operation for putting into the FTP server

For this use case we have the following sequence:

Linha do tempo Descrição gerada automaticamente

1. IRIS does a CDC on the Payment table and generates a JSON file with the table content, deleting the data read. The adapter uses these parameters to know the table and data to be read:

  • Call Interval: seconds needed to check new data
  • DSN: the namespace where the table exists
  • Target Config Names: the component of the production that will receive the data (in this case SendSQLDataToFTPBusinessOperation)
  • Query: the select command to get the data to be read (in this case it is SELECT ID, amount, payer, receiver, transactiondate FROM dc_irisftpsample.Payment)
  • Delete Query: the delete command used to delete the data read, if needed (in this case DELETE FROM dc_irisftpsample.Payment). 
  • Key Field Name: the primary key of the table.

Interface gráfica do usuário, Aplicativo Descrição gerada automaticamente

Interface gráfica do usuário, Texto, Aplicativo Descrição gerada automaticamente

2. Send the Stream of data in JSON format to the PaymentSQLBusinessService

3. Send the Stream of JSON file into an Ens.StreamContainer message in the SendSQLDataToFTPBusinessOperation

4. Send the stream of JSON file to the FTP Outbound Adapter:

Class dc.irisftpsample.SendSQLDataToFTPBusinessOperation Extends Ens.BusinessOperation
{
 
Parameter ADAPTER = "EnsLib.FTP.OutboundAdapter";
Property Filename As %String(MAXLEN = 1000);
Parameter SETTINGS As %String = "Filename";
Method OnMessage(pRequest As Ens.StreamContainer, Output pResponse As %Persistent) As %Status
{
  Set tFilename=
  ..Adapter.CreateTimestamp(##class(%File).GetFilename("/tmp/sqltojson"),..Filename)
  Quit ..Adapter.PutStream(tFilename, pRequest.Stream)
}
 
}
  • Note 1: The parameter ADAPTER defines that the Business Operation will use FTP Outbound Adapter.
  • Note 2: The parameter Filename defines the name of the file located on the FTP server.

5. Put the JSON file into the root FTP folder with the name SQLTOJSON_YYYY-MM-DD_HH.mm.ss.zzz.json using these parameters:

  • FTP server is the name of the IP of the Server in the network or Internet
  • FTP Port is the port to connect
  • Credentials are the Credentials reference to username/password configuration
  • File Path is where the FTP Adapter will connect to find files
  • File Name is the pattern for the name of the file on the FTP Server
  • Protocol: FTP or SFTP (FTP with SSL certificate) 
  • Use PASV: passive connections with the FTP server

Interface gráfica do usuário, Aplicativo Descrição gerada automaticamente

Interface gráfica do usuário, Texto, Aplicativo Descrição gerada automaticamente

To know more about

  1. Official documentation for FTP adapters: https://docs.intersystems.com/iris20221/csp/docbook/DocBook.UI.Page.cls?KEY=EFTP
  2. Official online learning about production, business services, and operations: https://learning.intersystems.com/course/view.php?id=1437
  3. Git repository for CSVGEN: https://github.com/evshvarov/csvgen
  4. About FTP: https://www.techtarget.com/searchnetworking/definition/File-Transfer-Protocol-FTP
4
1 133
Discussion (0)1
Log in or sign up to continue