Ensemble and file outbound adapters - a small hint

This short article was motivated by a problem of one of my customers. They use Ensemble to integrate many systems, some of them use just plain files.

So they naturally selected File Outbound Adapter to write into target file. Things were running smoothly for years, until recently, when the volume of data being written to the file reached large size of tens of megabytes. The operation took around half an hour to complete, causing timing problems where subsequent operations within the process had to wait, and third party system was not happy to wait so long.

The customer's code looked somehow like this pseudo-code snippet.

 set tResultSet=SQLStatement.Execute()

 // compose header based on resultset columns Describe()

 set tSC= ..Adapter.PutLine(file,header)

while tResultSet.%Next() {

 set line=... compose line of the resultset row data 

  set tSC=..Adapter.PutLine(file,line)

}

Having looked at the source code of the PutLine() method of the adapter, I found that the method is opening a file , writing a line and closing the file. For every single line!!!

So, checking whether we provide a better adapter API I found PutStream() method. Bingo!, that's the right one, it doesn't open the stream every time, just writes to it.

I rephrased the code in following manner:

set tTmpStream=##class(%FileCharacterStream).%New()

set tResultSet=SQLStatement.Execute()

 // compose header based on resultset columns Describe()

 set tSC= ..tTmpStream.WriteLine(header)

while tResultSet.%Next() {

 set line=... compose line of the resultset row data 

 set tSC=tTmpStream.WriteLine(line)

}

set tSC=..Adapter.PutStream(file,tTmpStream)

Customer used the modified code and re-ran his operation. The operation processing now takes seconds!

Outcome: Whilst PutLine() might be your first (and easiest) choice when working with files, it is not necessarily good method for production environment, specially when dealing with large files.

Hope this helps you in your future development projects.

Dan Kutac