New post
AllAccepted answers
Sean Connelly · Jul 13, 2017

Hi Russel,

I would solve the problem along the following lines...

set k2=$s($d(^G("ABC","A")):"A",1:$o(^G("ABC","A")))
while $e(k2,1)="A" {
   set k3=$o(^G("ABC",k2,""))
   while k3'="" {
     write !,k2," ",k3
     set k3=$o(^G("ABC",k2,k3))
   }
   set k2=$o(^G("ABC",k2))
}

Explanation...

//set k2 to either "A" if that key exists in the data, or the next key following "A".
set k2=$s($d(^G("ABC","A")):"A",1:$o(^G("ABC","A")))

//only process k2 when it starts with an "A", this is the wildcard functionality you are looking for
//when k2 does not start with an "A" the logic will drop through
while $e(k2,1)="A" {

    //get first child key of k2
    set k3=$o(^G("ABC",k2,""))

    //loop on all child keys found
    while k3'="" {

        write !,k2," ",k3

        //get next child key of k2
        set k3=$o(^G("ABC",k2,k3))
    }

    //get the next k2 key
    set k2=$o(^G("ABC",k2))

}
Sean Connelly · Jul 6, 2017

I've knocked up a quick example using dual ack's.

I've put the source code in a gist here...

https://gist.github.com/SeanConnelly/19b79c790daad530a754461923f9f2f1

Save the code to a file and import into a test namespace.

Either create the "in" and "archive" folders are per the inbound test file feeder, or change to suit your environment.

Drop an HL7 message into the "in" folder and this is what you will see in the trace...

The ACK message is sent into the service and is automatically forwarded to the sending operation where it is returned to the calling process as if it was original messages ACK.

Make sure to follow the instructions here when configuring the service and operation, they specifically need to be implemented using  EnsLib.HL7.Operation.TCPAckOutOperation and  EnsLib.HL7.Service.TCPAckInService

http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...

Also note, that you need to set the reply code actions so that the AE ACK is returned to the process, otherwise it will stop at the operation, I have set the actions to...

:?R=RF,:?E=W,:~=S,:?A=C,:*=S,:T?=C

Where a match on ?E will just warn and continue as if the message was ok.

Take a look at the custom class  Examples.DeferredHL7.CustomProcess

Which contains the following OnResponse method...

Method OnResponse(request As EnsLib.HL7.Message, ByRef response As EnsLib.HL7.Message, callrequest As EnsLib.HL7.Message, callresponse As EnsLib.HL7.Message, pCompletionKey As %String) As %Status
{
    $$$TRACE("request contains the inbound request "_request.RawContent)
    $$$TRACE("callrequest contains the sent request "_callrequest.RawContent)
    $$$TRACE("callresponse contains the deferred ACK "_callresponse.RawContent)
    quit $$$OK
}

This is where you will have both the original request messages and the ACK in the same scope. From here you can construct a new message from the data in both messages.

This is just one approach but should fit your needs.

Sean.

Sean Connelly · Jun 27, 2017

FOO>set msg=##class(EnsLib.HL7.Message).%OpenId(15)
 
FOO>w msg.RawContent
PID|2|2161348462|20809880170|1614614|20809880170^TESTPAT||19760924|M|||^^^^00000
OBR|1|8642753100012^LIS|20809880170^LCS|008342^UPPER RESPIRATORYCULTURE^L|||19980727175800||||||SS#634748641 CH14885 SRC:THROASRC:PENI|19980727000000||||||20809
OBX|1|ST|008342^UPPER RESPIRATORY||POSITIVE~~~~~~~|

FOO>w !,msg.SetValueAt("Positive","PIDgrpgrp(1).ORCgrp(1).OBXgrp(1).OBX:5")
 
0 0<Ens>ErrGeneralObject is immutable
 
FOO>set msg2=msg.%ConstructClone()
 
FOO>w !,msg2.SetValueAt(msg.GetValueAt("PIDgrpgrp(1).ORCgrp(1).OBXgrp(1).OBX:5.1"),"PIDgrpgrp(1).ORCgrp(1).OBXgrp(1).OBX:5")
 
1
 
FOO>w msg2.RawContent                                                           

PID|2|2161348462|20809880170|1614614|20809880170^TESTPAT||19760924|M|||^^^^00000
OBR|1|8642753100012^LIS|20809880170^LCS|008342^UPPER RESPIRATORYCULTURE^L|||19980727175800||||||SS#634748641 CH14885 SRC:THROASRC:PENI|19980727000000||||||20809
OBX|1|ST|008342^UPPER RESPIRATORY||POSITIVE|

Sean Connelly · Jun 6, 2017

Hi Javier,

COS does not have a Generics implementation, mainly as its a loosely/duck typed language.

You can however write generic code without needing Generics.

Make your property a base class of your Info classes, this can be %RegisteredObject...

Class Response Extends %RegisteredObject  {
​    Property Code As %String;
    Property Info As %RegisteredObject;
}


You can now assign any valid object to that property at run time.

You won't be able to assign a string to this property, so create a class with a single property of type string that you can assign it to.

Try that and if you get stuck with the JSON serialisation then post back the code that is not working.

Sean.

Sean Connelly · May 30, 2017

Hi Ranjith,

This is a very good question, and the opposite of one asked a week ago...

https://community.intersystems.com/post/xml-json-ensemble

Caché has varying degrees of support for JSON which will depend on the version of Caché that you have.

Firstly, you will not find a one step solution to your problem inside of Caché.

It's important to note that there is an impedance mismatch between JSON and XML that can produce different results in a one step solution. If you really don't care about this, or the exact format of the XML then I can point you towards...

http://www.newtonsoft.com/json/help/html/ConvertingJSONandXML.htm

which is a .NET solution. You could create a simple .NET object that wraps and calls this conversion utility. You can then bind to that .NET object using these instructions...

http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...

making the utility feel as if it was local Cache object / function.

There are Java alternatives which you can google for, for which you would use the Java binding...

http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...

A two step conversion which will require a little more coding, but will enable you to control your XML output exactly as you want it.

First you will need to convert the JSON into an internal object. If you are on 2016.1 or greater then please take a look at this article...

https://community.intersystems.com/post/introducing-new-json-capabilitie...

You can use the %Object to ingest JSON into a generic object.

From here you will need a concrete class that will be used to generate your XML. Make sure it extends %XML.Adapter. It will then be a process of mapping each property from the generic object to the concrete object. Finally call its XML to string / stream method and you will have well formed and consistent XML.

If you are on an older version of Cache then take a look at the %ZEN.Auxiliary.jsonProvider class which has a %ConvertJSONToObject, apparently its much slower than the newer object which might factor in your solution. I've never used this method myself, but would think you will end up with a very similar solution to the newer %Object.

Sean.

Sean Connelly · May 30, 2017

Hi Shobha,

Ensemble logs a great deal of metrics that you can use to determine all sorts of timings.

If you look at the header of any message you will see the time it was created and the time it was processed. This will give you the individual times taken for that message in its service or process to complete.

To get an end to end time you will need to know when the operation completed its task.

In your instance you can enable "Archive IO" which you will find under "Development and Debugging" in your file out operation settings. This will record the time received and responded. If you take one of these times and subtract the created time of the service message then you will have an overall benchmark.

Note that in dev you will probably see very little or no lag between each message stage. However when you get to a live environment this lag can increase from milliseconds to seconds depending on load. Therefore its best to take any benchmark on dev as just a best scenario.

Sean.
 

Sean Connelly · May 18, 2017

Hi Tom,

The clue is in the error message...

 <text>ERROR #5002: Cache error: &lt;UNDEFINED>zTestOperation+1^Test.WebService.1 *sc</text>

You have an undefined variable, and its name is sc, denoted by the *

If you look at your code...

if $$$ISERR(sc) do ..ReturnMethodStatusFault(sc)

You can see that you are checking the value of sc which has not yet been set.

Sean.

Sean Connelly · May 12, 2017

> What is your definition of a few k? Each line is about 25000 KB.

Do you mean 25,000 characters (25K)?

> Previous comment recommended processing the file as a single message stream. That ended up slowing the message viewer so much for these large messages that that it is impossible to view the message at all.

You can override the content display method on your Ens.Request class so that it doesn't display the entire message. You can replace this with a small summary about the file, size, no of records etc.

Creating 500,000 Ensemble messages is going to generate a lot of IO that you probably don't need.

I would still recommend processing them as one file. 

Sean Connelly · May 12, 2017

Please see previous comment to same question...

https://community.intersystems.com/post/multi-threading-improve-performance

You previously mentioned your file contains 500,000 records. How big are the records?

If each record is just a few k in size then reading and writing each record from the file to a global will take under 5 minutes.

In which case you have a serious bottleneck going on such as a poor referential integrity check. If this is not indexed or tuned then you will have some serious IO thrashing going on, and no matter how many cores you throw at the problem, you will not get any overall performance gains.

Sean Connelly · May 12, 2017

OK, something a little more sophisticated, I've bashed out a class method that should do this for you...
 

ClassMethod ObjectToArray(pObject, Output pArray)
{
    kill pArray
    if '$IsObject(pObject) Quit ""
    set rs=##class(%ResultSet).%New("%DynamicQuery:SQL")
    set sc=rs.Prepare("select Name, RuntimeType, Type from %Dictionary.CompiledProperty where parent=? and NOT Name [ '%'")
    set sc=rs.Execute(pObject.%ClassName(1))
    while rs.Next()
    {
        set property=rs.Data("Name")
        try {
            set pArray(property)=$PROPERTY(pObject,rs.Data("Name"))
        } catch duff { }
    }
    if pObject.%ClassName(1)="%CSP.Request" Merge pArray("CgiEnvs")=pObject.CgiEnvs Merge pArray("Cookies")=pObject.Cookies
}

Just call it for each object you wanted converted to an array, merge that into a global or dump it out to pre tags as before, e.g.

    write "<h2>Request</h2><pre>"
    do ..ObjectToArray(%request,.array)
    zw array
    write "</pre><br><h2>Session</h2><pre>"
    do ..ObjectToArray(%session,.array)
    zw array
    write "</pre><br><h2>Response</h2><pre>"
    do ..ObjectToArray(%response,.array)
    zw array
    write "</pre>"
    Quit $$$OK

And you should get this...

Request

array("AppData")=$lb("",8224,0,"","/csp/healthshare/","Ensemble Management Portal",1,"","",0,1,"","","/csp/healthshare/r","R","","H:\intersystems\CSP\healthshare\r",1,"%Ens_Portal","",2,"",900,2,2,"",3600,0,1,1,"%ISCMgtPortal",1,"","")
array("AppMatch")="/csp/healthshare/r/"
array("Application")="/csp/healthshare/r/"
array("CSPGatewayRequest")=1
array("CgiEnvs","COMSPEC")="E:\WINDOWS\system32\cmd.exe"
array("CgiEnvs","CONTENT_LENGTH")=0
array("CgiEnvs","CONTEXT_DOCUMENT_ROOT")="H:/InterSystems/CSP/"
array("CgiEnvs","CONTEXT_PREFIX")="/csp/"
array("CgiEnvs","CSP_ORIGINAL_FILE")=""
array("CgiEnvs","ComSpec")="E:\WINDOWS\system32\cmd.exe"
array("CgiEnvs","DOCUMENT_ROOT")="H:/InterSystems/CSP"
array("CgiEnvs","GATEWAY_INTERFACE")="CGI/1.1"
array("CgiEnvs","HTTP_ACCEPT")="text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
array("CgiEnvs","HTTP_ACCEPT_ENCODING")="gzip, deflate, sdch, br"
array("CgiEnvs","HTTP_ACCEPT_LANGUAGE")="en-US,en;q=0.8"
array("CgiEnvs","HTTP_CONNECTION")="keep-alive"
array("CgiEnvs","HTTP_COOKIE")="CSPSESSIONID-SP-13333-UP-csp-healthshare-=000000010000nKXcB0rA4r0000Z7RqCT1zYzo6RsrYydAinQ--; CacheBrowserId=zN0TSZMio2VoHmBqhiKm1A--; state-B3B82FCB-2C53-4D41-8B47-CC389A5C7606=SYSEXP%3A1; Username=sean; CSPWSERVERID=fce6d9ad0b3878fbbaef4ad2ad576c9e14c732d1"
array("CgiEnvs","HTTP_HOST")="localhost:13333"
array("CgiEnvs","HTTP_UPGRADE_INSECURE_REQUESTS")=1
array("CgiEnvs","HTTP_USER_AGENT")="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36"
array("CgiEnvs","PATH")="E:\ProgramData\Oracle\Java\javapath;E:\Program Files (x86)\Intel\iCLS Client\;E:\Program Files\Intel\iCLS Client\;E:\WINDOWS\system32;E:\WINDOWS;E:\WINDOWS\System32\Wbem;E:\WINDOWS\System32\WindowsPowerShell\v1.0\;E:\Program Files\Intel\Intel(R) Management Engine Components\DAL;E:\Program Files\Intel\Intel(R) Management Engine Components\IPT;E:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;E:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;E:\Program Files (x86)\QuickTime\QTSystem\;E:\Program Files (x86)\Git\cmd;E:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;E:\WINDOWS\system32;E:\WINDOWS;E:\WINDOWS\System32\Wbem;E:\WINDOWS\System32\WindowsPowerShell\v1.0\;E:\Program Files (x86)\Pandoc\;E:\Program Files (x86)\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn\;E:\Program Files (x86)\Microsoft SQL Server\130\Tools\Binn\;E:\Program Files (x86)\Microsoft SQL Server\130\DTS\Binn\;E:\Program Files (x86)\Microsoft SQL Server\130\Tools\Binn\ManagementStudio\;E:\Program Files (x86)\Skype\Phone\;E:\Program Files\Microsoft SQL Server\Client SDK\ODBC\110\Tools\Binn\;E:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn\;E:\Program Files\Microsoft SQL Server\120\Tools\Binn\;E:\Program Files\Microsoft SQL Server\120\DTS\Binn\;E:\Program Files\nodejs\;E:\WINDOWS\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps"
array("CgiEnvs","PATHEXT")=".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC"
array("CgiEnvs","PATH_TRANSLATED")="H:/InterSystems/CSP/csp/healthshare/r/Foo.Session.cls"
array("CgiEnvs","QUERY_STRING")=""
array("CgiEnvs","REMOTE_ADDR")="::1"
array("CgiEnvs","REMOTE_PORT")=53822
array("CgiEnvs","REQUEST_METHOD")="GET"
array("CgiEnvs","REQUEST_SCHEME")="http"
array("CgiEnvs","REQUEST_URI")="/csp/healthshare/r/Foo.Session.cls"
array("CgiEnvs","SCRIPT_FILENAME")="H:/InterSystems/CSP/healthshare/r/Foo.Session.cls"
array("CgiEnvs","SCRIPT_NAME")="/csp/healthshare/r/Foo.Session.cls"
array("CgiEnvs","SERVER_ADDR")="::1"
array("CgiEnvs","SERVER_ADMIN")="[no address given]"
array("CgiEnvs","SERVER_NAME")="localhost"
array("CgiEnvs","SERVER_PORT")=13333
array("CgiEnvs","SERVER_PORT_SECURE")=0
array("CgiEnvs","SERVER_PROTOCOL")="HTTP/1.1"
array("CgiEnvs","SERVER_SIGNATURE")=""
array("CgiEnvs","SERVER_SOFTWARE")="Apache Cache_Server_Pages-Apache_Module/2014.1.1.702.0-1401.1419b"
array("CgiEnvs","SystemRoot")="E:\WINDOWS"
array("CgiEnvs","WINDIR")="E:\WINDOWS"
array("CgiEnvs","windir")="E:\WINDOWS"
array("CharSet")="utf-8"
array("Content")=""
array("ContentType")=""
array("Cookies","CSPWSERVERID",1)="fce6d9ad0b3878fbbaef4ad2ad576c9e14c732d1"
array("Cookies","CacheBrowserId",1)="zN0TSZMio2VoHmBqhiKm1A--"
array("Cookies","Username",1)="sean"
array("Cookies","state-B3B82FCB-2C53-4D41-8B47-CC389A5C7606",1)="SYSEXP%3A1"
array("GatewayApplication")="/csp"
array("GatewayBuild")="661.1401.1419b"
array("GatewayConnectionName")="LOCAL"
array("GatewayError")=""
array("GatewayFunctions")=7
array("GatewayInstanceName")="alien:13333"
array("GatewayNewId")=0
array("GatewaySessionCookie")="CSPSESSIONID-SP-13333-UP-csp-healthshare-"
array("GatewayTimeout")=60
array("Method")="GET"
array("PageName")="Foo.Session.cls"
array("Protocol")="HTTP/1.1"
array("RequestId")="2E98"
array("Secure")=0
array("URL")="/csp/healthshare/r/Foo.Session.cls"
array("URLPrefix")=""
array("UserAgent")="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36"

Session

array("AppTimeout")=900
array("Application")="/csp/healthshare/r/"
array("ApplicationLicenses")=""
array("BrowserName")="Safari"
array("BrowserPlatform")="Windows"
array("BrowserVersion")=537.36
array("ByIdGroups")=$lb("%iscmgtportal:zN0TSZMio2VoHmBqhiKm1A--")
array("CSPSessionCookie")="000000010000nKXcB0rA4r0000Z7RqCT1zYzo6RsrYydAinQ--"
array("CookiePath")="/csp/healthshare/"
array("CreateTime")="2017-05-12 13:51:19"
array("Debug")=0
array("EndSession")=0
array("ErrorPage")=""
array("EventClass")=""
array("GetNewId")=0
array("GroupId")="%iscmgtportal:zN0TSZMio2VoHmBqhiKm1A--"
array("HttpAuthorization")=""
array("KeepAlive")=1
array("Key")="è$½Wò"_$c(31)_"qÎ3ñ3fy"_$c(132,30,3)
array("Language")="en-us"
array("LastModified")="2017-05-12 14:06:57"
array("LicenseId")="sean@127.0.0.1"
array("LogoutCleanup")=0
array("MessageNumber")=2
array("Namespace")="R"
array("NewSession")=0
array("OldTimeout")=5565507717
array("Preserve")=0
array("ProcessId")=""
array("RunNamespace")=""
array("SOAPRequestCount")=0
array("SecureSessionCookie")=0
array("SecurityContext")=$lb("sean","%All","%All",32,-559038737)
array("SessionId")="nKXcB0rA4r"
array("StickyLogin")=""
array("UseSessionCookie")=2
array("UserAgent")="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36"
array("Username")="sean"
array("nosave")=0

Response

array("CharSet")="utf-8"
array("ContentLength")=""
array("ContentType")="text/html"
array("CookiePath")="/csp/healthshare/"
array("Domain")=""
array("Expires")="Thu, 29 Oct 1998 17:04:19 GMT"
array("GzipOutput")=""
array("HTTPVersion")=""
array("HeaderCharSet")=""
array("InProgress")=1
array("Language")="en-us"
array("NoCharSetConvert")=0
array("OutputSessionToken")=1
array("Redirect")=""
array("ServerSideRedirect")=""
array("Status")="200 OK"
array("Timeout")=""
array("TraceDump")=0
array("UseASPredirect")=0
array("UseHttpOnly")=1
array("VaryByParam")=""
Sean Connelly · May 10, 2017

(thanks John)

I'm now looking at how to implement this with JavaScript files.

In my JavaScript I will have lines of code like this...

popToast("danger","Unable to open document");


Since $$$Text can't be used in this context I wonder what else other developers are doing. I could fetch the JavaScript via a CSP page and replace the strings, but that feels kludgy.

My initial thoughts would be to create a $$$Text JavaScript function for continuity and use it in a similar way...

popToast("danger",$$$Text("MyApp","Unable to open document"));


The localisation file would be loaded on application start (or when the language is changed), something along the lines of...

/csp/myapp/localisation.cls?domain=MyApp&lang=default


Spin the global out into JSON...

"MyApp" : {
  "en" : {
    "1243066710" : "Hello World"
  }
}


$$$Text would be something like...

var $$$Text = function(domain,phrase) {
  var hash = someHashFunction(phrase)  // ??
  return locals[domain][defaultLanguage][hash]
}


The next question is, what hashing algorithm is Cache using to create the hash?

I will need to replicate that in JavaScript, or come up with some other hash approach.

I've also noticed that the samples does not use this hash, instead it uses a man made key, but using that key with $$$Text in COS just creates a hash which seems odd. Any explanations for that?