%FromJSON() failing with content larger than 32kb

Question

Question

Jack Smith · May 19, 2019

#Business Service #JSON #ObjectScript #Ensemble

We have a business service that extends Ens.BusinessService and uses EnsLib.File.InboundAdapter.

This service polls for json files in a folder.

Method signature is like this:

Method OnProcessInput(pInput As %FileCharacterStream, pOutput As %RegisteredObject) As %Status

And inside that method it tries to read the content like this:

set tData = pInput.ReadLine(,.tStatus, .tEOL)
if ($$$ISERR(tStatus)) quit
set tJSON = {}.%FromJSON(tData)

I added some trace calls before and after lines and noticed it's always that last line that fails if content is larger than 32kb.

Any workaround for this? Or is this wrong way to do it to begin with and something else should be there instead?

Our Ensemble version is 2016.2.1

Discussion (7)1

Log in or sign up to continue

Jeffrey Drumm · May 19, 2019

The ReadLine() method of %FileCharacterStream defaults to a length of 32K.

Have you tried simply passing the stream object, pInput, as the argument to %FromJSON()?

3 0

score 0 · Answer 1 · 2019-05-19T13:11:10-04:00

Wow, that actually works! Thanks so much, Jeffrey!

This isn't originally my code so I don't know why that

set tData = pInput.ReadLine(,.tStatus, .tEOL)

part is there and tdata is used as argument instead of pinput but probably for some reason? So even thou I'm happy your suggestion actually works, I'm also worried if that change breaks something else. But on the other hand, it did not on my initial tests anyway. I get exactly the same output with those json files that worked earlier but now also larger files work fine. Hmmm...

score 2 · Answer 2 · 2019-05-20T10:53:37-04:00

Exception still can contain some useful info, so:

    #dim tSC As %Status = $$$OK
    ...
    try {
        set tJSON = {}.%FromJSON(pInput)
    } Catch ex {
        do ex.Log()
        set tempSC = $$$ERROR($$$GeneralError,"Badly formed JSON")
        set tSC = $$$ADDSC(tempSC, ex.AsStatus())
    }

    quit tSC

score 0 · Answer 3 · 2019-05-20T11:13:59-04:00

Jeffrey Drumm May 20, 2019 to Eduard Lebedyuk

Thank you @Eduard Lebedyuk! Good to know

0 0

score 0 · Answer 4 · 2019-05-19T14:37:13-04:00

The only reason I can think of doing it the way it was done is that there was no expectation by the original author of the code that the JSON object would be larger than 32KB. It's also possible that he/she was unfamiliar with the types of objects that %FromJSON() works with and always assumed it was a simple string ... which is what the ReadLine() method returns.

Here's the documentation for for the %FromJSON() method.

score 1 · Answer 5 · 2019-05-19T14:42:13-04:00

I'd also recommend to remove or change this line:

if ($$$ISERR(tStatus)) quit

because your method signature has a promise to return a %Status:

Method OnProcessInput(pInput As %FileCharacterStream, pOutput As %RegisteredObject) As %Statu

however above mentioned line quits nothing.

You can rewrite it like this (to return status):

if $$$ISERR(tStatus) quit tStatus

or my personal preference like this using postconditionals:

quit:$$$ISERR(tStatus) tStatus

Well in this exact case you need to remove the line altogether because you no longer get status for ReadLine, but it's just some food for thought.

score 0 · Answer 6 · 2019-05-20T09:24:46-04:00

As @Eduard Lebedyuk pointed out, the method should return a status as indicated by its signature, so at the very minimum you should have a "Return $$$OK" at the end of the method.

That said, %FromJSON() throws an exception when it fails, and the generic error you'll get in the Event Log won't tell you (or your support team) much. You may want to wrap a try/catch around the call to %FromJSON() so that you can return a status that provides a little more insight:

    ...

    Try {
        set tJSON = {}.%FromJSON(pInput)
    } Catch {
        Return $$$ERROR($$$GeneralError,"Badly formed JSON")
    }

...

    Return $$$OK
 }