A few questions:

• You didn't mention what format these files are in -- XML? flat-file/CSV? Something else?
• Which InterSystems product are you working with and what kind of application is this (interoperability production, web service, etc.)?

If these are flat-files or CSV, and if you're working with a Health Connect or IRIS interoperability production, you can look at using the Record Mapper, which will read a flat file using a format you define and allow you to work with records from the file as objects:
https://docs.intersystems.com/healthconnect20191/csp/docbook/DocBook.UI....

If the files are XML you can do something similar by importing an XSD and using XML virtual documents, or use %XML.Adaptor methods:
https://docs.intersystems.com/healthconnect20191/csp/docbook/DocBook.UI....

How about:

&sql(select count(*) into :tDefined from ENS_Config.Item where Name=:tHostName and Production=:tProductionName)

Ok, EnsLib.REST.GenericOperation also expects to receive an EnsLib.HTTP.GenericMessage.

Are you using EnsLib.File.PassthroughService to pick up the files? If so, it is sending an Ens.StreamContainer message to the target component (your business operation). You'll need to create a data transformation that creates a new EnsLib.HTTP.GenericMessage and populates it with the stream content from the Ens.StreamContainer. Then you'll need a router in the middle to run the data transformation and send the resulting EnsLib.HTTP.GenericMessage to the business operation.

Neerav, I'll just note that if you use Jenna's suggestion your code doesn't need to be a part of a business service -- it can be any normal ObjectScript code and can be called from anywhere (CSP, scheduled task, etc.). And your code can make as many calls as it needs to and can receive response messages.

It will appear in message traces as if a business service sent requests to the process or operation, but in reality it's just your code sending the requests.

Jenna's approach is a good one, and it's the standard way to achieve what you describe. Having said that, if you can provide more details on your use case and/or why this approach doesn't fit your need we can help you explore alternatives.

Which business service are you using to read the file? Are you using EnsLib.RecordMap.Service.FileService and specifying the record map type in the configuration?

It sounds like you might be using EnsLib.File.PassthroughService

You could use Apache's mod_rewrite to take all requests that fall under a certain sub-path and transform them on the fly to point to your CSP page. It could add the information about which specific page/resource was requested as a URL parameter that could be accessed by your CSP page.

For example, if a client makes a request for:
http://myserver/reporting/Dashboard1/resource2.js

mod_rewrite could change it to:
http://myserver/csp/dashboards/proxy.csp?targetResource=Dashboard1/resou...

After mod_rewrite changes the URL in the request, Apache continues processing it as usual using the new URL. Since the new URL refers to a CSP page Apache will pass it to CSP Gateway as we want.

Here's a stored procedure that accepts a setting name and returns the setting value for all components that have it. It's not SQL, but can be executed from SQL :)

You can call it this way -- this example returns the port setting for all components that have it:

call Sample.Util_SettingsByName('Port')

Here's the source code as XML export format. Copy this into a file and then import it using Studio, terminal, or the System Management Portal.

<?xml version="1.0" encoding="UTF-8"?>
<Export generator="Cache" version="25" zv="Cache for Windows (x86-64) 2016.1.1 (Build 108U)" ts="2016-10-12 16:15:39">
<Class name="Sample.Util">
<Super>%RegisteredObject</Super>
<TimeChanged>64203,58380.929948</TimeChanged>
<TimeCreated>64202,44682.614801</TimeCreated>

<UDLText name="T">
<Content><![CDATA[
/*
         *****************************************************
         *                 ** N O T I C E **                 *
         *               - TEST/DEMO SOFTWARE -              *
         * This and related items are not supported by       *
         * InterSystems as part of any released product.     *
         * It is supplied by InterSystems as a demo/test     *
         * tool for a specific product and version.          *
         * The user or customer is fully responsible for     *
         * the maintenance of this software after delivery,  *
         * and InterSystems shall bear no responsibility nor *
         * liabilities for errors or misuse of this item.    *
         *                                                   *
         *****************************************************
*/
]]></Content>
</UDLText>

<Query name="SettingsByName">
<Type>%Query</Type>
<FormalSpec>SettingName:%String</FormalSpec>
<SqlProc>1</SqlProc>
<Parameter name="ROWSPEC" value="BusinessHost:%String,SettingName:%String,SettingValue:%String"/>
</Query>

<Method name="SettingsByNameExecute">
<ClassMethod>1</ClassMethod>
<FormalSpec><![CDATA[&qHandle:%Binary,SettingNames:%String=""]]></FormalSpec>
<ReturnType>%Status</ReturnType>
<Implementation><![CDATA[
    s qHandle=##class(%ArrayOfObjects).%New()

    &sql(select %DLIST(id) into :tHostIDs from ENS_Config.Item order by Name desc)
    s tHostIDList=##class(%Library.ListOfDataTypes).%New()
    s tSC=tHostIDList.InsertList(tHostIDs)
    s tSC=qHandle.SetAt(tHostIDList,"HostIDs")

    s tSC=qHandle.SetAt(##class(%ArrayOfDataTypes).%New(),"Counters")
    s tSC=qHandle.GetAt("Counters").SetAt(0,"CurrHost")
    s tSC=qHandle.GetAt("Counters").SetAt(0,"CurrSetting")
    
    if ($L(SettingNames)>1) {
        s SettingNames=$ZCONVERT(SettingNames,"U")
        s tFilterList=##class(%Library.ListOfDataTypes).%New()
        s tSC=tFilterList.InsertList($LISTFROMSTRING(SettingNames))
         s tSC=qHandle.SetAt(tFilterList,"FilterList")
    }

    Quit $$$OK
]]></Implementation>
</Method>

<Method name="SettingsByNameClose">
<ClassMethod>1</ClassMethod>
<FormalSpec><![CDATA[&qHandle:%Binary]]></FormalSpec>
<PlaceAfter>SettingsByNameExecute</PlaceAfter>
<ReturnType>%Status</ReturnType>
<Implementation><![CDATA[    Quit $$$OK
]]></Implementation>
</Method>

<Method name="SettingsByNameFetch">
<ClassMethod>1</ClassMethod>
<FormalSpec><![CDATA[&qHandle:%Binary,&Row:%List,&AtEnd:%Integer=0]]></FormalSpec>
<PlaceAfter>SettingsByNameExecute</PlaceAfter>
<ReturnType>%Status</ReturnType>
<Implementation><![CDATA[
    s tCurrHost=qHandle.GetAt("Counters").GetAt("CurrHost")
    s tCurrSetting=qHandle.GetAt("Counters").GetAt("CurrSetting")
    s tHostIDs=qHandle.GetAt("HostIDs")
    s tFilterList=qHandle.GetAt("FilterList")
    s oHost=qHandle.GetAt("Host")

    do {
        if ('$IsObject(oHost)||(oHost.VirtualSettings.Count()<tCurrSetting)) {
            if (tCurrHost=tHostIDs.Count()) {
                s AtEnd=1
                q
            }

            s tCurrHost=tCurrHost+1
            s tCurrSetting=1
            
            s tHostID=tHostIDs.GetAt(tCurrHost)
            s oHost=##class(Ens.Config.Item).%OpenId(tHostID,0)
            
            s tSC=oHost.PopulateVirtualSettings()

            s tSC=qHandle.SetAt(oHost,"Host")                
            s tSC=qHandle.GetAt("Counters").SetAt(tCurrHost,"CurrHost")
        }

        s tSettings=oHost.VirtualSettings
        s tSetting=tSettings.GetAt(tCurrSetting)
        s tStngName=$LISTGET(tSetting,2)
        s tStngValue=$LISTGET(tSetting,3)
        
        s tCurrSetting=tCurrSetting+1
    } while ($IsObject(tFilterList)&&('tFilterList.Find($ZCONVERT(tStngName,"U"))))
        
    if ('AtEnd) {
        s Row=$LB(oHost.Name,tStngName,tStngValue)
    }
    
    s tSC=qHandle.GetAt("Counters").SetAt(tCurrSetting,"CurrSetting")
        
    Quit $$$OK
]]></Implementation>
</Method>
</Class>
</Export>

Ens.StreamContainer's %New() method expects a string as the first parameter rather than a stream object.

Something like this should work:

set tRequest=##class(Ens.StreamContainer).%New()
set tSC=tRequest.StreamSet(pInput)

Or if you're trying to send one Ens.StreamContainer for each line from the input file you could do this:

while 'pInput.AtEnd {
    set tReadLength=32000
    set tLine=pInput.ReadLine(.tReadLength,.tSC)
    set tRequest=##class(Ens.StreamContainer).%New(tLine)
    //... do other stuff
}

One other thing you should be aware of. The following will not work if TargetConfigNames has more than one target selected:

set tSC = ..SendRequestAsync(..TargetConfigNames,tRequest)

You should add a loop using $LENGTH and $PIECE and do a SendRequestAsync for each item in TargetConfigNames' comma-separated string.

The resulting XML files can be imported again using %System.OBJ.Load().

If you prefer GOF format you can use %Global.Export() instead, however it doesn't accept wildcards so you would need to first put together a list of which globals you want to export.

For automation you can execute these methods from your own custom class or routine. If you want to schedule it to run automatically, you can create your custom class as a %SYS.Task.Definition and schedule it to run using Task Manager.

You're correct -- you can't restore specific globals from an Online Backup (.cbk) file.

Have a look at %System.OBJ.Export():
https://irisdocs.intersystems.com/irisforhealth20192/csp/documatic/%25CSP.Documatic.cls

TESTING>s itm="*.GBL"
TESTING>s file="C:\Projects\export_mm.xml"
TESTING>d $System.OBJ.Export(.itm,.file,,.errors)

Having said that, since this is just for backups you might also consider just using the standard Online Backup utility which will backup all globals in the selected database(s) into a file:

https://irisdocs.intersystems.com/irisforhealth20192/csp/docbook/DocBook.UI.Page.cls?KEY=GCDI_backup#GCDI_backup_methods_online

Here's a (very simple) working example of uploading to an AWS S3 bucket. I used essentially the same approach that Cliff originally used to generate the hex. I didn't experience the same problem, but I may have just been lucky and not had any zeroes in the value I'm converting.

<?xml version="1.0" encoding="UTF-8"?>
<Export generator="Cache" version="25" zv="Cache for Windows (x86-64) 2017.2.1 (Build 801_3U)" ts="2018-10-04 08:14:32">
<Class name="Demo.Cloud.Storage.Util">
<Super>%RegisteredObject</Super>
<TimeChanged>64925,29407.276303</TimeChanged>
<TimeCreated>64924,56047.115234</TimeCreated>

<Method name="TestS3Upload">
<ClassMethod>1</ClassMethod>
<Implementation><![CDATA[
    set tAccessKeyId="[Shh... it's a secret]"
    set tSecretAccessKey="[Shh... it's a secret]"
    set tAWSRegion="us-east-1"
    set tAWSService="s3"
    set tHost="s3.us-east-1.amazonaws.com"
    set tPort="443"
    set tBucket="MyBucket12345"
    set tContentType="text/plain; charset=UTF-8"

    set tSSLConfig="AWS"
    set tUseHTTPS=1
    
    set tContentStream=##class(%Stream.TmpCharacter).%New()
    do tContentStream.Write("this is some test content oh yeah!")
    do tContentStream.Rewind()

    set tHorolog=$h
    set tDateKey=$System.Encryption.HMACSHA(256,$ZD(tHorolog,8),"AWS4"_tSecretAccessKey)
    set tDateRegionKey=$System.Encryption.HMACSHA(256,tAWSRegion,tDateKey)
    set tDateRegionServiceKey=$System.Encryption.HMACSHA(256,tAWSService,tDateRegionKey)
    set tSigningKey=$System.Encryption.HMACSHA(256,"aws4_request",tDateRegionServiceKey)
        
    set tHashedPayload=..BytesToHex($System.Encryption.SHAHashStream(256,tContentStream,.tSC))
    
    set tTmpTime=$ZDT(tHorolog,2,7)
    set tHeaderDate=$ZD(tHorolog,11)_", "_$E(tTmpTime,1,11)_" "_$E(tTmpTime,13,*-1)_" GMT"
    set tAmzDate=$TR($zdt(tHorolog,8,7),":","")

    set tKeyName="test_"_tHorolog_".txt"

    set tCanonicalURI="/"_tBucket_"/"_tKeyName
    set tCanonicalURI=$zconvert(tCanonicalURI,"O","URL")

    set tHttpVerb="PUT"
    set tCanonicalHeaders="content-length:"_tContentStream.Size_$c(10)_"content-type:"_tContentType_$c(10)_"host:"_tHost_$c(10)_"x-amz-content-sha256:"_tHashedPayload_$c(10)_"x-amz-date:"_tAmzDate_$c(10)
    set tSignedHeaders="content-length;content-type;host;x-amz-content-sha256;x-amz-date"
    set tCanonicalRequest=tHttpVerb_$c(10)_tCanonicalURI_$c(10,10)_tCanonicalHeaders_$c(10)_tSignedHeaders_$c(10)_tHashedPayload
    set tStringToSign="AWS4-HMAC-SHA256"_$c(10)_$TR($zdt(tHorolog,8,7),":","")_$c(10)_$zd(tHorolog,8)_"/"_tAWSRegion_"/"_tAWSService_"/aws4_request"_$c(10)_..BytesToHex($System.Encryption.SHAHash(256,tCanonicalRequest))
    set tSignature=..BytesToHex($System.Encryption.HMACSHA(256,tStringToSign,tSigningKey))    
    set tAuthHeader="AWS4-HMAC-SHA256 Credential="_tAccessKeyId_"/"_$zd(tHorolog,8)_"/"_tAWSRegion_"/"_tAWSService_"/aws4_request,SignedHeaders="_tSignedHeaders_",Signature="_tSignature

    set tHReq=##class(%Net.HttpRequest).%New()
    set tHReq.Server=tHost
    set tHReq.Port=tPort
    set tHReq.SSLConfiguration=tSSLConfig
    set tHReq.Https=tUseHTTPS

    do tHReq.SetHeader("Authorization",tAuthHeader)
    do tHReq.SetHeader("Content-Type","text/plain")
    do tHReq.SetHeader("Content-Length",tContentStream.Size)
    do tHReq.SetHeader("Host",tHost)
    do tHReq.SetHeader("x-amz-content-sha256",tHashedPayload)
    do tHReq.SetHeader("x-amz-date",tAmzDate)

    do tHReq.EntityBody.CopyFrom(tContentStream)

    //do tHReq.OutputHeaders()

    s tSC=tHReq.Put(tCanonicalURI)
    do $System.OBJ.DisplayError(tSC)
    
    write "Status Code:",tHReq.HttpResponse.StatusCode,!
    
    //Do tHReq.HttpResponse.OutputToDevice()
    
    if $isobject(tHReq.HttpResponse.Data) {
        s ^zresp=tHReq.HttpResponse.Data.Read(1000000)
    } else {
        s ^zresp=tHReq.HttpResponse.Data
    }
]]></Implementation>
</Method>

<Method name="BytesList">
<ClassMethod>1</ClassMethod>
<FormalSpec>pBytes:%String</FormalSpec>
<ReturnType>%String</ReturnType>
<Implementation><![CDATA[
    s l=$LISTFROMSTRING(pBytes," ")
    f i=1:1:$LL(l)  s st=$G(st)_$CHAR($ZHEX($LISTGET(l,i)))
    return st
]]></Implementation>
</Method>

<Method name="BytesToHex">
<ClassMethod>1</ClassMethod>
<FormalSpec>pBytes:%String</FormalSpec>
<ReturnType>%String</ReturnType>
<Implementation><![CDATA[
    set tHex=""
    
    for i=1:1:$L(pBytes) {
        set tHexByte=$ZHEX($ASCII($E(pBytes,i)))
        if $L(tHexByte)=1 s tHexByte="0"_tHexByte
        set tHex=tHex_tHexByte
    }
    
    return $ZCONVERT(tHex,"L")
]]></Implementation>
</Method>
</Class>
</Export>

Yes, you can loop through items in your source document and generate one HL7 message for each. 

In this case the looping would be done in a business process (BPL), and you would call the transformation once for each outbound HL7 message.

Have you tried setting "DefCharEncoding" to UTF-8 in the settings for EnsLib.HL7.Operation.HTTPOperation?

Here are the details from the popup-help. Especially note the highlighted line. Using "!UTF-8" will cause the operation to ignore what's in MSH:18 and always use UTF-8.

Default Character Encoding to use when reading or writing HL7 messages.
If MSH field 18 (Character Set) is empty, this encoding will be used. Choices you can use for this setting include:

Native: Use the default character encoding of the installed locale of the IRIS server.
latin1: The ISO Latin1 8-bit encoding. This is the default.
ISO-8859-1: The ISO Latin1 8-bit encoding.
UTF-8: The Unicode 8-bit encoding.
Unicode: The Unicode 16-bit encoding (Little-Endian).
UnicodeBig: The Unicode 16-bit encoding (Big-Endian).
Any other NLS definitions installed on this IRIS server.
@<ttable>: <ttable> means a raw InterSystems character translation table name. A prefix of '@' means to use the named table.
Putting ! before the encoding name will force the use of the named encoding and will ignore any value found in MSH:18.

Hi Joseph,

I developed something like this for a POC last year. It includes wizards to generate Business Services and Business Operations and related message objects based on a source query or stored procedure.

It's not currently in a state where I can share it, but I'm planning to eventually clean it up and post it on Open Exchange. I'll let you know when I do.

-Marc

What about setting "Reply Code Actions" to "E=D" and enabling "Alert on Error"?

From the reply code actions help text:
"D - Disable the Process, log an error and restore the original incoming message to the front of the Process's queue."

For current versions, the approach I've seen customers use is to create a business operation using the HL7 or XML outbound file adapter. In Message Viewer you can then "resend" the messages you want to the new business operation.

I'm happy to say that we've added a way to download multiple messages directly from Message Viewer. This hasn't been released yet but will be soon. For more details on this and other upcoming features, have a look at this presentation.

Hi Guillaume,

Here's some rough code showing how to use executeParametersBatch. I put this together from a few other working examples I had but I haven't tested this actual code and it may have bugs.

As always, this is provided as sample code only and is not meant for production use :)

-Marc
 

set pSQLStatement="INSERT INTO Test.Table (Field1,Field2) VALUES (?,?)"

// JDBCGwy is an instance of the JDBC Gateway object. EnsLib.SQL.OutboundAdapter instantiates this automatically and stores a reference to it in ..%Connection.%JGProxy

// Prepare the SQL statement
set pHS=JDBCGwy.prepareStatement(ConnHandle,pSQLStatement)

// executeParametersBatch expects tArgs to be a $LIST, with the following format:
//     ParamCount, ParamSets, Type1, Param1, Type2, Param2, Type3, Param3, Type11,Param11… TypeNN,ParamNN
// 
// ParamCount is the number of parameters the query expects (in this example 2) 
// ParamSets is the number of rows we will be inserting in this batch
// Type1, Type2, ..., TypeN is an integer indicating the JDBC data type for the corresponding Param value (e.g. Param1, Param2, ..., ParamN)
// Param1, Param2, ..., ParamN is the value for the query parameter

set $LIST(tArgs,1)=2  // The query has two parameters ("?") in it
set $LIST(tArgs,2)=3  // We will insert 3 rows in this batch

// Row 1 -------------------------------------------------------------

//    Set the value for the first column (Field1VarChar)
set $LIST(tArgs,3)=12 // The JDBC data type for varchar is 12
set $LIST(tArgs,4)="String value 1" // Value for column Field1VarChar

//    Set the value for the second column (Field2Integer)
set $LIST(tArgs,5)=4 // The JDBC data type for integer is 4
set $LIST(tArgs,6)=7 // Value for column Field2Integer

// Row 2 -------------------------------------------------------------

//    Set the value for the first column (Field1VarChar)
set $LIST(tArgs,7)=12 // The JDBC data type for varchar is 12
set $LIST(tArgs,8)="String value 2" // Value for column Field1VarChar

//    Set the value for the second column (Field2Integer)
set $LIST(tArgs,9)=4 // The JDBC data type for integer is 4
set $LIST(tArgs,10)=123 // Value for column Field2Integer

// Row 3 -------------------------------------------------------------

//    Set the value for the first column (Field1VarChar)
set $LIST(tArgs,11)=12 // The JDBC data type for varchar is 12
set $LIST(tArgs,12)="String value 3" // Value for column Field1VarChar

//    Set the value for the second column (Field2Integer)
set $LIST(tArgs,13)=4 // The JDBC data type for integer is 4
set $LIST(tArgs,14)=54 // Value for column Field2Integer


// Perform the batch insert
// tResultCodes is a $LIST of integers indicating success/failure for each row in the batch
set tResultCodes=JDBCGwy.executeParametersBatch(pHS,tArgs)
 

You can do this with the $CLASSMETHOD function in ObjectScript:

https://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY...

ClassMethod someMethod(className As %String)
{
	set classObject = $CLASSMETHOD(className,"%New")

...
}

This won't help you in the short term, but upcoming versions will include a nice feature to export messages from the message search page. For more details, have a look at this presentation from this year's InterSystems Global Summit.

https://learning.intersystems.com/course/view.php?id=1015

For the short term, most people use the suggestion from @Julian Matthews of adding a new file out business operation and resending the messages to it.

Some business operations that use EnsLib.File.OutboundAdapter will pull the value for %f from a property in the request object. This would allow you to set that property in a DTL before sending it to the business operation.

But, it looks like you're using EnsLib.XML.Object.Operation.FileOperation, which doesn't do this. It uses the class name of the inbound object as the value for %f.

To create a filename programmatically with EnsLib.XML.Object.Operation.FileOperation, you can create a custom class that extends EnsLib.XML.Object.Operation.FileOperation and override the OnMessage() method. 

The relevant line in the standard OnMessage method is this. You can replace this with your custom logic:

    // Create output filename using the class name of the persistent class as the base.
    Set tFilename=..Adapter.CreateFilename($classname(pRequest),..Filename)