New post
AllAccepted answers
Sean Connelly · May 10, 2018

You won't need a package name for the global name, this is just a naming convention for globals that are auto named by the %Persistent class.

If you provide your own global name then it can be anything you want, so for the transitory data just call it something like... 

^CacheTempDedup

I think there is probably an easier way to solve the dedup problem by passing the log file twice and recording the line numbers of interest into a temporary global, you can then skip all the early duplication and minimise the transaction writes.

This is a 2 min bash but it should give you a general idea of how to do that...

kill ^CacheTempDedupSeen
kill ^CacheTempDedupLine
set lineNo=1
set file=##class(%File).%New("file.txt")
do file.Open("R")
set data=file.ReadLine
while file.AtEnd=0 {
    set code=$piece(data,",",1) //e.g
    if $data(^CacheTempDedupSeen(code)) kill ^CacheTempDedupLine(^CacheTempDedupSeen(code))
    set ^CacheTempDedupSeen(code)=lineNo
    set ^CacheTempDedupLine(lineNo)=code
    set data=file.ReadLine
    set lineNo=lineNo+1
}
do file.Rewind()
set lineNo=1
set data=file.ReadLine
while file.AtEnd=0 {
    if $data(^CacheTempDedupLine(lineNo)) {
        //this data is the last in the log for its code type, update the persistent class directly
    }
    set data=file.ReadLine
    set lineNo=lineNo+1
}
Sean Connelly · May 10, 2018

It will be the name of the global... ^CacheTempGlobal

Global export import seems like a good strategy.

I was just looking at your SQL again, do you have all of the lookup values in one single global node? If this is the case then that would explain why the transaction log was getting full up. 

What you should really do is split the data out into many global entries...

E.g.

^Lookup("UK")="United Kingdom"
^Lookup("USA")="United States"
^Lookup("ESP")="Spain"
^Lookup("ITA")="Italy"

Making the key (or keys depending on your data) unique and easy to check for duplicates on those key(s).

Then, when you make updates to the global, only the node that changes will be committed to the transaction log, which will have a tiny overall impact.

Sean Connelly · May 10, 2018

Hi Antonio,

Sounds like you are almost there.

The temp global name is case sensitive, so make sure it starts with ^CacheTemp and not ^CACHETEMP.

You will lose this temp global on a restart, so you will need to then make it permanent once you have dealt with all the duplicates. You can do this with the merge command...

http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...

You will need to kill the original global first otherwise you will merge the temp global into it.

Alternatively you could have keyed your original global in a way that duplicates are easily identified, e.g. ^foo(uniqueKeyValue)=data, you can then check if the entry exists first before deciding to write anything to the database.

Sean.

Sean Connelly · May 9, 2018

This is a bit of an experimental (not released) error dump utility that I developed not so long back to explore the idea of dumping all variables into a JSON string.

It uses undocumented $zu functions which are listed on this site...

http://www.cachewiki.org/index.php/Undocumented_Syntax

Essentially you use $zu(42 to $order over the variables at a specific stack level.

In this instance I build an exception object with all of the primitives and objects at the stack level in error.

I then use the Cogs JSON library to covert the entire error object into a serialised JSON string that is embedded into a Status string. The good thing here is that it will do a deep serialisation of objects which often get clipped in other solutions.

From here I can pass the JSON part of the Status string to an error log viewing tool that formats the JSON into an HTML view.

It doesn't look like I fully got the stack levels into the object, so the code would need some further tweaking.

Class Cogs.Lib.Status Extends Cogs.JsonClass
{

Property Error As %String(MAXLEN = 50000);

Property Stack As array Of %String(MAXLEN = 50000);

Property Objects As array Of Cogs.Lib.Status.Object;

Property Primatives As array Of %String(MAXLEN = 50000);

ClassMethod AsStatus(pException As %Exception.AbstractException) As %Status
{
    set sc=pException.AsStatus()
    try {
        do pException.Log()
        set status=##class(Cogs.Lib.Status).%New()
        set status.Error=$zerror
        for i=1:1:$stack-1 do status.Stack.SetAt($stack(i,"PLACE")_" : "_$zstrip($stack(i, "MCODE"),"<W"),i)
        set level=$ZU(41)-2
        set var=$zu(42,level,"~")
        while var'="" {
            set name=$p(var,"~",2)
            set item=$zu(43,level,var)
            if $data(item),item'["%Exception.SystemException" {
                if $IsObject(item) {
                    set object=##class(Cogs.Lib.Status.Object).%New()
                    set object.Reference=item
                    set object.Properties=##class(Cogs.JsonObject).toJSON(item)
                    do status.Objects.SetAt(object,name)
                } else {
                    do status.Primatives.SetAt(item,name)
                }
            }
            set var=$zu(42,level,var)
        }
        set sc=$$$ERROR($p(##class(%SYSTEM.Status).GetErrorCodes(sc),","),status.toJSON())
    } catch err {}
    quit sc
}

}

Example in use...

Class Foo.Exception Extends %RegisteredObject
{

ClassMethod Foo()
{
    try {
        set customer=##class(Northwind.Customers).%OpenId("ALFKI")
        set foo="Hello, World"
        set sc=$$$ERROR($$$GeneralError,"A general error type")
        $$$ThrowOnError(sc)
    } catch exception {
        set errorString=##class(Cogs.Lib.Status).AsStatus(exception)
        write !,errorString
    }
}

}

The error object...

{
   "Error":"",
   "Objects":{
      "customer":{
         "Properties":{
            "Address":"Obere Str. 57",
            "City":"Berlin",
            "CompanyName":"Alfreds Futterkiste",
            "ContactName":"Maria Anders",
            "ContactTitle":"Sales Representative",
            "Country":"Germany",
            "CustomerID":"ALFKI",
            "Fax":"030-0076545",
            "Phone":"030-0074321",
            "PostalCode":"12209",
            "Region":""
         },
         "Reference":"1@Northwind.Customers"
      },
      "exception":{
         "Properties":{
            "NextException":""
         },
         "Reference":"2@%Exception.StatusException"
      }
   },
   "Primatives":{
      "foo":"Hello, World",
      "sc":"0 $\u0001\u0004\u0004‰\u0013\u0016\u0001A general error type\u0001\u0001\u0001\u0001\u0001\u0001\u0001\u0001"
   },
   "Stack":{
      "1":"zFoo+7^Foo.Exception.1 +1 : set errorString=##class(Cogs.Lib.Status).AsStatus(exception)"
   }
}
Sean Connelly · May 8, 2018

Hi Stephen,

I think the short (and potentially dangerous) answer that you are looking for is to use xecute to run COS provided as an argument, such that you have one generic method, e.g.

var h = foo.get_execute("$h");

where execute implements xecute on the passed argument(s).

It is better to try and understand JavaScripts built in functions then call out to Caché which could end up being expensive.

However, if you wanted to replicate functions to make context switching to JavaScript less of a cognitive load then you could write your own mini function library...

let $piece = (string,delimiter,from,to=from) => {
  return string.split(delimiter).slice(from-1,to).join("~");
}

let p1 = $piece("hello~world","~",1);

let $extract = (string,from,to=from) => {
    return string.substring(from-1,to);
}

let sub = $extract("Hello, World",2,5)

let $length = (string,delimiter) => {
  if (delimiter !== undefined) return string.split(delimiter).length;
  return string.length;
}

let len = $length("Hello, World")
>12

let count = $length("Hello~World","~")
>2

Probably the function of most use is $horolog, I've built a couple of solutions that uses horolog everywhere, but tbh I just use W3C dates and auto convert between the two these days, if your using JSON then the Cogs library will automatically do this for you.

I can't find my original horolog everywhere source code, but here is a quick bash...

let $h = function() {
  let now = new Date();
  return Math.floor(((now.getTime() + 4070908800000) / 86400000)) + "," + ((now.getHours() * 60 * 60) + (now.getMinutes() * 60) + now.getSeconds());
}

$h()
> "64776,60593"

let horologToJSDate = function(h) {
    let parts = h.split(",");
    return new Date(((parts[0] * 86400000) - 4070908800000) + (parts[1] === undefined ? 0 : seconds * 1000))
}

horologToJSDate("64776,61081")
> Tue May 08 2018 17:58:01 GMT+0100 (GMT Summer Time)

horologToJSDate("64776")
> Tue May 08 2018 01:00:00 GMT+0100 (GMT Summer Time)

Sean.

Sean Connelly · May 5, 2018

Hi David,

There are a couple of options.

The first is ZEN reports, you can read more about them here...

http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=GRPT_reports

The solution is essentially built on top of Apache FOP.

Zen reports abstract some of the complexity of FOP, but you still need to learn ZEN's DSL syntax to get up and running.

Essentially you will create an XML data object in a format specified by the ZEN documentation, an XSLT stylesheet is applied to this, which produces Apache FOP XML. This is then passed into the FOP engine which outputs a PDF document.

It's not a straight forward solution, and if your not writing ZEN reports every day it might seem like a bit of a chore.

Personally I didn't want to learn and remember a new DSL when I am fluent in HTML and CSS, which is why I roll my own solutions on top of wkhtmltopdf...

https://wkhtmltopdf.org/

I have deployments built on top of this that have been generating millions of medical documents a year without a single problem.

Its easy enough to roll your own, just create an HTML string from within COS using data extracted from the HL7 message, include some CSS to format the text, tables etc, save this to a file, and then use the $ZF function to call out to the wkhtmltopdf exe.

You can pass in extra arguments on the call to add things like headers and footers, for instance this would add a page number

--footer-center [page]/[topage]

There is also another option that I am starting to lean towards and this is using the chrome browser in headless mode which enables support for more modern CSS features and allows for more creativity.

>"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --headless --disable-gpu --print-to-pdf=j:\TEMP\file1.pdf http://www.example.com/

I've got a half baked example of using Chrome, if it helps I could tidy it up a little and post it to GitHub.
 

Sean Connelly · May 4, 2018

Hi Jaoa,

You need to make sure you check the "Create Business Operation" in the wizard and type in a valid package name for the operation, request and response classes (the text inputs below the tick box).

You then add the generated operation to your production and set the credential settings.

Next, look through the generated request classes and find the two classes that have the name of the methods that you want to call on Exchange.

If you send an instance of these request classes to the operation, then the operation will implement the method call with the given request details, it will then return a response object that will be derived from one of the generated response classes.

Sean Connelly · Apr 7, 2018

FOO>d ##class(FOO.Boxes).main(20)
 
####################
##                ##
# #              # #
#  #            #  #
#   #          #   #
#    #        #    #
#     #      #     #
#      #    #      #
#       #  #       #
#        ##        #
#        ##        #
#       #  #       #
#      #    #      #
#     #      #     #
#    #        #    #
#   #          #   #
#  #            #  #
# #              # #
##                ##
####################


FOO>w ##class(FOO.Boxes).length()
75

Still thinking, seems like it might go a bit smaller...

Sean Connelly · Mar 16, 2018

Hi Keven,

The compilation is failing for the unit tests which I have included in the src folder, but thought I had removed from the main build file. I was planning on a new push so I will tidy that up. For now you can ignore those particular errors, the main Cogs.JsonClass will not be effected.

If you are on a new version of Caché then you should check to see if $ZCVT supports JSON, if you are currently rolling your own JSON and just want to escape it then just do...

$zcvt(string,"O","JSON")

Or if you are targeting old and new then you can call the base EscapeJSON method on Cogs which compiles by version to $ZCVT or its own escape method, you can call that with...

##class(Cogs.Lib.Json.Base).EscapeJSON(string)

I would imagin if you are working with QEWD then you might be shuffling lots of data around in globals / arrays.

In this instance you might not want to use the Cogs JSON Class as you will end up having to shim in a whole new set of classes.

The new Cogs push does have more options now, ability to work with globals and arrays and legacy classes (without extending them).

There are also the Zen utilities that might help, again these are only in the more recent versions of Caché.

Sean.

Sean Connelly · Mar 16, 2018

In terms of docs, there is a page here that you can read, its in the git docs folder...

https://github.com/SeanConnelly/Cogs/blob/master/docs/Json/JsonClass.md

If you are planning on using the new REST classes in Caché then you might also be interested in some Cogs additions that are not too far off. These include a REST mapping generator with automatic Swagger / OpenAPI documentation generation.

Sean Connelly · Mar 16, 2018

Hi Keven,

It can depend on what the data is and where (what technology) you are sending it from.

If you are using JavaScript for instance then calling its JSON.stringify() will auto escape the obvious characters for you.

These are mainly the JSON reserved characters and control characters which get prefixed with \u.

If you have binary data then you will want to Base64 encode the data which will make it JSON safe, or send the binary separately with the JSON, for instance in a multi form data request.

I have a backwards compatible (and battle tested library) for handling JSON inside Caché that demonstrates encoding and decoding these characters, you can find it here...

https://github.com/SeanConnelly/Cogs

I also have some updates to push to it which will auto convert binary data in and out of JSON without the 3.6GB large string limitation.

Sean.

Sean Connelly · Mar 13, 2018

Looks like a couple of issues, I can't see where you are saving the document.

Its a 5 minutes hack, but here is an example of what I might do...

Class Foo.QueryDataToCSV Extends Ens.BusinessOperation
{
Parameter ADAPTER = "EnsLib.SQL.OutboundAdapter";
Property Adapter As EnsLib.SQL.OutboundAdapter;
Parameter INVOCATION = "Queue";

Method OnMessage(pRequest As Ens.StringContainer, Output pResponse As %Library.Persistent) As %Status
{
    set sql="select * from Foo.Person"
    
    #dim rs as EnsLib.SQL.GatewayResultSet
    set sc=..Adapter.ExecuteQuery(.rs,sql)
    if $$$ISERR(sc) quit sc

    set file = ##class(%Stream.FileCharacter).%New()
    set file.Filename="C:\Temp\123.csv"
    
    while rs.Next() {
        set (comma,rec)=""
        for i=1:1:rs.GetColumnCount() {
            set rec=rec_comma_""""_rs.GetData(i)_""""
            set comma=","    
        }
        set sc=file.WriteLine(rec)    
    }
    
    quit file.%Save()
}

}
Sean Connelly · Mar 13, 2018

I was only half right, %DisplayFormatted does not return a value so it would cause a COMMAND error, but Robert correctly pointed out its happening on a different line, I must have green colour blindness today.

On second glance zMessageHandler is a generated method, so as Robert mentioned, you will need to return a status code from your CheckResults method to stop the COMMAND error in the first instance.

It's important to bubble up any errors or a final status OK value from your operation methods so that they are caught and reported correctly in the Ensemble logs, otherwise you will have silent errors and not know where things failed, as in this instance.

For every step in your code where a status is returned you should check the status and immediately quit if its an error, e.g.

if $$$ISERR(sc) quit sc

Where sc is the status code, or you can use the $$$QuitOnError macro.

Can you paste all of your code...