%FromJSON doesn't support %Stream on a property basis

Hello.

I just noticed that whenever I parse a JSON using %FromJSON, it always return the property values as primary data types, including long strings.

I did a quick stress test to measure it's capabilities and found out that characteristic,  this is something I needed to figure out beforehand  to implement my REST lib.

What I concluded is that %Dynamic depends solely on volatile storage? What if the user doesn't have long string support enabled or if the server has not enough memory to handle long strings?

Is there a fallback for handling it?

Please note that is an extreme and unusual use case, specially the last one, but I wanted to figure how to handle it accordingly.

  • 0
  • 2
  • 454
  • 7
  • 2

Answers

Use old json provider:

set st = ##class(%ZEN.Auxiliary.jsonProvider).%ConvertJSONToObject(json, class,.obj, $$$YES)

Instead of class argument, the json can contain _class property.

Thank you for your suggestion, however the error still happens.

I also didn't made it clear, but the payload doesn't have a class to be reflected from, it can be abstract, requiring it to be parsed with %DynamicObject or %DynamicArray.

Since there's no class with %Stream.* property, %ConvertJSONToObject cannot decide what to do with long strings.
The ideal would be to detect the $$$MaxStringSize size and fallback to an internal %Stream instance before assigning the value to the proxy.

Also, I'm trying to avoid using %ZEN.proxyObject due to issues with stacking many objects for a single process. So the class parameter should be something inheriting from %DynamicAbstractObject.

 

Here's is the step-by-step:

DEV>set f = ##class(%Stream.FileCharacter).%New()
 
DEV>do f.LinkToFile("/temp/lorem.json")
 
DEV>set sc = ##class(%ZEN.Auxiliary.jsonProvider).%ConvertJSONToObject(f,, .o)
 
DEV>if $System.Status.IsError(sc) do $System.OBJ.DisplayError(sc)
 
ERRO #5002: Erro Caché: <MAXSTRING>%ParseJSON+290^%ZEN.Auxiliary.jsonProvider.1

This won't happen if I have long strings enabled.

For the most part, dynamic objects use local memory that is accounted for by $zstorage and $storage. However, some of it comes from other heap-allocated memory, similar to long strings. You can round trip a large JSON object through a dynamic object with %FromJSON()/%ToJSON(), even if an individual field exceeds the string limit. However, it is not currently possible to get the value of such a field within Caché.

Note that you should call %ToJSON() such that it outputs to a device or writes to a stream:

USER>d o.%ToJSON(stream)

USER>d o.%ToJSON()

Otherwise, you may get a STRINGSTACK error:

USER>w o.%ToJSON()

W o.%ToJSON()
^
<STRINGSTACK>

Hello Jon.

 

Please note that the large JSON is not problem itself, but a specific field instead.

I'm already avoiding a round trip by outputting to the device instead of using streams. 
As a side note, I noticed that parsing/serializing using %Dynamic API is much faster and more efficient than %ZEN.proxyObject.

 

So I'd prefer keep using it if possible. But since you said that Caché cannot get such values I suppose I can advice the user to enable long strings. I don't think any property content could even get close to 3GB.

Unfortunately, long strings can't be 3GB, but rather almost 3.5MB. For any sane usage of JSON this should be enough, but I've seen REST APIs where e.g. PDF streams are returned in base64-encoded JSON properties, that come uncomfortably close to that size.

It would be really nice if the %DynamicObject class added support for streams to deal with these large payloads!

You're right, it's not 3GB. Yes, objects containing base64 binary can be troublesome.
For now a workaround I would suggest is to split the string into chunks and put them inside an array, that way you can use .join('').

Comments

Pls. don't forget to mark your question as "answered" on Developer Community,
please click the checkmark alongside the answer you (as author of the question) accept

It´s beem 18 months and Itersystems did not come with any solution such as Rubens´ suggestion to this common need yet?