New post
Question
· Oct 4, 2017

FTP2FTP transfer without storing information into database?

#Ensemble #FTP

Hello!
We want an Integration that should be moving files from a FTP server in a DMZ zone into another FTP server on our local network.
I tried using EnsLib.FTP.PassthroughService(EnsLib.FTP.InboundAdapter, EnsLib.FTP.OutboundAdapter)
Using this approach ensamble write data to the database, causing the CACHE.DAT to grow for every file that is moved.
Looks like the entire file is written to the database, is this the case?
We are not really interested in storing any file content information in ensamble in this particular case.
Does anyone has a good suggestion on how to accomplish the file transfer using ensamble with another approach/configuration or arguments on why this would be a bad idea in the first place?
Best regards,
Magnus

Discussion (6)0
Log in or sign up to continue
Jeff Morgan · Oct 4, 2017

Clark is more or less correct.

 

In order for a Business Operation FTP Passthrough component to send a file downstream, the file has to have been saved somewhere local to Ensemble/HealthShare.  This can be one of two default locations.

 

1. The Caché database using GlobalCharacterStreams(initial default)

2. The local file system using FileCharacterStreams

 

This is controlled by the setting in the Business Service FTP Passthrough, "Use FileStream".  There is no default way of preventing the Business Service from saving the file. 

 

If space is your primary concern, then I would suggest an aggressive Message Purge strategy (using BodiesToo and KeepIntegrity set to true) as this can be a standard solution for keeping your persistent data  low even if you have large data flow in your Passthrough interface.

1 0
Alice Shrestha · Oct 4, 2017

If the second FTP server is in your local network, is it possible to map the drive to a local drive? Once you do this, in theory you should be able to directly download the file to the mapped drive without relying on ftp to download and then upload. 

If this process still writes the file to CACHE.dat then I would simply create a batch file in windows that can download the file into the mapped drive or local drive and upload if necessary, then run the batch file, either using windows task manager or task scheduler in Ensemble (RunLegacyTask) and use Do $ZF(-1,(batch file location)) or Do ##class(%Net.Remote.Utility).RunCommandViaZF((batch file location),,.rt). 

If the growing CACHE.Dat file is an issue then you can use Compacting and Truncating feature described here:  http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...

0 0
Mauri Kaatrasalo · Oct 4, 2017

Set Service's UseFileStream=True, then content is stored on files on server, stream folder of database defined for namespace. 

It doesn't grow then CACHE.DAT anymore, of course it grows still disk usage. But You can add purge task once in day and keep messages for 0 days so everything is cleaned up once in day.

Cleaning grows some journal but not much in this case because file contents are not in CACHE.DAT

And if Your files are so big, that You want use different temporary directory during transfer You can change this for whole system:
^%SYS("TempDir")="/enssys/ensprod/mgr/Temp"

or just for one namespace:
^%SYS("TempDir","YOURNAMESPACE")="/some/temp/dir"

Or build some streaming adapter, it might be good for not so important data, which needed just on demand, not later. Like streaming live video via Ensemble :) 

Eduard's link to InMemory capabilities is interesting, thanks!

0 0
Clark Matthews · Oct 4, 2017

Magnus,

   I am no Ensemble expert, however I understood that logging what it does is part of what Ensemble does.  So I would expect it to log what it is transferring.

   You could write your own class outside of Ensemble using the %Net.FtpSession class.  Open up two connections and copy the stream you read with .Retrieve() to the stream you write with .Store().  Would this work?

HTH,
   Clark

0 0
Güvenal · Oct 5, 2017

Hello,

First thanks to everyone who contributed,  Mauri Kaatrasalo was the one who led me in the direction to my final solution but I learned something new from all of you.
So using FileStream setting Ensamble creates 2 files, one under the Ensamble installation and one in the stream folder of the namespace. Ensamble will delete the one under the ensamble installation after being done with it’s job, so the one left to care about is the one in the stream folder of the namespace.

What I ended up doing in short is… 
#Turned on the service’s UseFileStream propterty.
#Created a subclass of  EnsLib.FTP.InboundAdapter, where I added Property for “StreamedFileName”
#During processFile I save the filename ensamble gave the tempfile it is creating in “StreamedFileName” property.
#In the service after sending the file to the EnsLib.FTP.PassthroughOperation I delete the resulting stream file using the value in the ..Adapter.StreamedFileName property.

Best regards,

Magnus

0 0