New post
Question
· Aug 9, 2017

%FromJSON doesn't support %Stream on a property basis

#JSON #Object Data Model #Caché

Hello.

I just noticed that whenever I parse a JSON using %FromJSON, it always return the property values as primary data types, including long strings.

I did a quick stress test to measure it's capabilities and found out that characteristic,  this is something I needed to figure out beforehand  to implement my REST lib.

What I concluded is that %Dynamic depends solely on volatile storage? What if the user doesn't have long string support enabled or if the server has not enough memory to handle long strings?

Is there a fallback for handling it?

Please note that is an extreme and unusual use case, specially the last one, but I wanted to figure how to handle it accordingly.

Discussion (8)3
Log in or sign up to continue
Rubens Silva · Aug 9, 2017

Thank you for your suggestion, however the error still happens.

I also didn't made it clear, but the payload doesn't have a class to be reflected from, it can be abstract, requiring it to be parsed with %DynamicObject or %DynamicArray.

Since there's no class with %Stream.* property, %ConvertJSONToObject cannot decide what to do with long strings.
The ideal would be to detect the $$$MaxStringSize size and fallback to an internal %Stream instance before assigning the value to the proxy.

Also, I'm trying to avoid using %ZEN.proxyObject due to issues with stacking many objects for a single process. So the class parameter should be something inheriting from %DynamicAbstractObject.

 

Here's is the step-by-step:

DEV>set f = ##class(%Stream.FileCharacter).%New()
 
DEV>do f.LinkToFile("/temp/lorem.json")
 
DEV>set sc = ##class(%ZEN.Auxiliary.jsonProvider).%ConvertJSONToObject(f,, .o)
 
DEV>if $System.Status.IsError(sc) do $System.OBJ.DisplayError(sc)
 
ERRO #5002: Erro Caché: <MAXSTRING>%ParseJSON+290^%ZEN.Auxiliary.jsonProvider.1

This won't happen if I have long strings enabled.

0 0
Rubens Silva · Aug 10, 2017

Hello Jon.

 

Please note that the large JSON is not problem itself, but a specific field instead.

I'm already avoiding a round trip by outputting to the device instead of using streams. 
As a side note, I noticed that parsing/serializing using %Dynamic API is much faster and more efficient than %ZEN.proxyObject.

 

So I'd prefer keep using it if possible. But since you said that Caché cannot get such values I suppose I can advice the user to enable long strings. I don't think any property content could even get close to 3GB.

0 0
Jon Willeke · Aug 9, 2017

For the most part, dynamic objects use local memory that is accounted for by $zstorage and $storage. However, some of it comes from other heap-allocated memory, similar to long strings. You can round trip a large JSON object through a dynamic object with %FromJSON()/%ToJSON(), even if an individual field exceeds the string limit. However, it is not currently possible to get the value of such a field within Caché.

Note that you should call %ToJSON() such that it outputs to a device or writes to a stream:

USER>d o.%ToJSON(stream)

USER>d o.%ToJSON()

Otherwise, you may get a STRINGSTACK error:

USER>w o.%ToJSON()

W o.%ToJSON()
^
<STRINGSTACK>
0 0
Gertjan Klein · Aug 10, 2017

Unfortunately, long strings can't be 3GB, but rather almost 3.5MB. For any sane usage of JSON this should be enough, but I've seen REST APIs where e.g. PDF streams are returned in base64-encoded JSON properties, that come uncomfortably close to that size.

It would be really nice if the %DynamicObject class added support for streams to deal with these large payloads!

1 0