Hi,

I am not sure I know what the issue is exactly. Is it speed, is it DB writes, or memory (FRAMESTACK) errors.

In the meantime, some things we've done to help with large files.

We have BPL's that process CSV files with 100k rows, and these things helped.

Using the proposed approach, there will not be a port listening for REST messages on the Production.

All WS requesst will have to go through the CSP gateway.
If there are other Business Operations utilising that REST service, you will have to connect via the web-server and set up credentials to use.

Also, as suggested by @Dmitry Maslennikov , you can split the services into smaller services if control is required at that level.

Then for local services consumed by the same or other productions pn the same server, you can use Ens RESTService, and control access to the port on the OS firewall to only allow only for traffic from localhost.

Also have a look at the class documentation.
You can disable the local port, so that it will not listen from the production and enable it to go via CSP. You then need to set the CSP application authentication options.
From the class documentation.
property EnableStandardRequests as %Boolean [ InitialExpression = 1 ];

Listen via the CSP WebServer in addition to listening on the HTTP.InboundAdapter's custom local port, if the Adapter is defined/

Note that SSLConfig only applies to the custom local port. To use SSL via the CSP WebServer, you must configure the WebServer separately.

If the Service is invoked via the CSP WebServer, the ?CfgItem= URL parameter may be used to distinguish between multiple configured same-class Services but the standard csp/namespace/classname URL must be used.

Which Business Service class are you using?

How is the produciton receiving the message?

Usually for REST services on a production, I do the following:

  • Create a "dispatch" class with the basic which extends from %CSP.REST
  • This REST class will receive the messages, then invoke a business service in the production.
  • You set up a CSP application that uses this class as the dispatcher.
    • Set it as authenticated, which will use basic authentication.

Something like the below. This is in one class, but you can put it into two. You then use this class as the dispatch class on the CSP application:

Class MyPackage.RESTService Extends (%CSP.REST, Ens.BusinessService)
{

Parameter UseSession = 0;

/// Name of Ensemble Business Service. Set in implementation
Parameter BUSINESSSERVICENAME = "Production Clever Methods";

XData UrlMap [ XMLNamespace = "http://www.intersystems.com/urlmap" ]
{
<Routes>
	<Route Url="/somemethod Method="POST" Call="HandleSomeFancyMethod"/>
</Routes>
}



ClassMethod HandleSomeFancyMethod() As %Status
{
	set sc = $$$OK
	try {
		// Check Content Type
		if (%request.ContentType '= "application/json") {
			// Throw some error here or repond
		}
		set %response.ContentType = "application/json"

		// Check Data Recieved
		if (%request.Content = "") {
			// Empty data error or "bad request or something"
		}

		// Parse the input into a Dynamic Object or whatever and validate as you'd like. You can also just send the stream to the service as indicated further down

		// Create a business service
		set sc = ##class(Ens.Director).CreateBusinessService(..#BUSINESSSERVICENAME, .tService)
		if $$$ISERR(sc) {
			// throw some error
		}

		// Create input for Service
		set tEnsRequest = ##class(Ens.StreamContainer).%New()
		do tInput.Rewind()
		set sc = tEnsRequest.StreamSet(tInput)
		if $$$ISERR(sc) {
			// trhow some error
		}

		Set tAttrs=##class(%ArrayOfDataTypes).%New()
		do tAttrs.SetAt(%response.ContentType,"ContentType")
		do tEnsRequest.SetAttributes(.tAttrs)

		// Process the input
		set sc = tService.ProcessInput(tEnsRequest, .tEnsOutput)
		// handle the sc however you see fit
		set sc = tEnsOutput.StreamGet().OutputToDevice()
		// handle the sc however you see fit

	} catch tEx {
		// error 500
	}

	quit sc
}

Method OnProcessInput(pInput As Ens.StreamContainer, Output pOutput As Ens.StreamContainer) As %Status
{
	set sc = $$$OK
	try {
		// do whatever you want to do
		// You can send to other business hosts and so forth


		// Set the response object into the stream
		set tStream = ##class(%Stream.GlobalCharacter).%New()
		// tDynamicObj in this case is the reponse object
		set sc = tStream.Write(tDynamicObj.%ToJSON())
		set sc = pOutput.StreamSet(tStream)
	} catch ex {
		set sc = ex.AsStatus()
	}

	quit sc
}

}

Hi there,

Are you using VSCode?

If so, you can convert the EOL for new files you create and ones you edit. On VSCode you can use LF on WIndows too without issues.

Otherwise, after exporting the classes, do the following in terminal

Set tOldFile = ##class(%Stream.FileCharacter).%New()
w tOldFile.LinkToFile("C:\whereever\code-with-crlf.xml")
Set tNewFile = ##class(%Stream.FileCharacter).%New()
w tNewFile.LinkToFile("C:\whereever\code-with-lf.xml")
Do tOldFile.Rewind()
While ('tOldFile.AtEnd) {  set tTempStr = tOldFile.ReadLine()  Do tNewFile.Write($ZSTRIP(tTempStr,"*",""_$CHAR(13)) _ $CHAR(10))  }
w tNewFile.%Save()
do tOldFile.%Close()
do tNewFile.%Close()

Then import that file and see if this solution broke your code. :)

So this code runs on Server A?

Server A handles the passthrough and then needs to check that the file landed on Server B, correct?

If that is the case, the solution is not going to be this simple. You will have to have another operation called by the BP to check if the file is on Server B. In short

  • A custom operation that uses the FTP Outbound Adaptor to FTP to Server B and pull a list of files.
  • The operation will need the filename and should be provided by the Ensemble message from the BP.
  • The operation should then check if the filename received in the Ensemble message is in the File List on Server B. You might be able to filter for the filename directly with FTP, I will have to confirm that.
  • The operation should then return a message to the BP that contains a "result" of the check. The BP can then act on that by creating an Alert Message and sending it to Ens.Alert.

I hope this helps. Let me know if I am misunderstanding the requirement.

It does not seem like the XData block parameters for the JSON mappings supports something like that.
I did the following to get a different output value on the JSON. It is not the prettiest solution.

Class Test.JSONThings Extends (%Persistent, %XML.Adaptor, %JSON.Adaptor)
{

Property Name As %String;

Property Surname As %String;

Property Gender As %String(DISPLAYLIST = ",Male,Female", JSONLISTPARAMETER = "DISPLAYLIST", VALUELIST = ",M,F");

Property SomeOtherGender As %String(DISPLAYLIST = ",Man,Vrou", JSONLISTPARAMETER = "DISPLAYLIST", VALUELIST = ",M,F") [ Calculated, SqlComputeCode = { set {*}={Gender} }, SqlComputed, Transient ];

Property Age As %Integer; Property Notes As %String(MAXLEN = ""); Property Code As Test.JSONRef(%JSONREFERENCE = "ID");

XData NSMapping
{
<Mapping xmlns="http://www.intersystems.com/jsonmapping">
        <Property Name="Name" FieldName="Name"/>
        <Property Name="Surname" FieldName="Surname"/>
        <Property Name="SomeOtherGender" FieldName="Gender"/>
</Mapping>
}

}

Hiya,

Should it be a WebSocket, or can it be a TCP counted socket connection?

If TCP counted, there are adapters available in Ensemble/IRIS, you  just need to create the business operation for the server-side, and the business service for the client-side.
You can use the EnsLib.TCP.CountedXMLOutboundAdapter  adapter, and the related inbound adapter.
In the operation you will have to serialise the Ensemble message to XML, and vice versa on the service.
Both these adapters have built-in functionality to ensure connections are up and running, and you can set up alerting controls on it.

I have never sent Ensemble messages directly like this to another instance, so I can't comment on if it will work or not.

Resolved this by adding a Rule Function and using it in the Rule Set.

Rule Class:

Class Test.TestRuleFunction Extends Ens.Rule.FunctionSet
{

/// GetAt implementation for rulesets
ClassMethod CollectionGetAt(ByRef pCollection, pKey As %String) As %Integer [ CodeMode = expression, Final ]
{
pCollection.GetAt(pKey)
}

}

Rule set expression

CollectionGetAt(ReturnValues,"CalculatedValue")

This is just an idea, and I may be corrected on this.
You get an array of artists. I can't find anything on storing arrays of the same key as you have in this scenario.

I would first do the web-service call.

  • Then create a DynamicObject using %FromJSON and the http response data as the source.
  • Get the array of artist from the DynamicObject using %Get("artists") on the DynamicObject instance.
  • Create an iterator on the object returned by the previous command using %GetIterator
  • Iterate through the collection - for each
    • Create a docdb entry and the properties or open an existing one. I recommend creating a separate method for this to keep the code concise.
    • Create a JSON stream of the current object on the iterator using %ToJSON(.tMyStream)
    • DO db.%FromJSON(.tMyStream)