Question
Eduard Lebedyuk · Jan 27, 2022

Storing dynamic object properties larger than 3641144 symbols

I have defined a class with a dynamic object property:

 
My class

The issue I encounter is that if a length of a serialized json property is more than 3641144 symbols, the object fails to save with the following error (either MAXSTRING or STRINGSTACK):

Length: 100 Result: OK
Length: 3641143 Result: OK
Length: 3641144 Result: OK
Length: 3641145 Result: ERROR #5002: ObjectScript error: <MAXSTRING>%GetSerial+1^%Library.DynamicAbstractObject.1
Length: 7282288 Result: ERROR #5002: ObjectScript error: <STRINGSTACK>%GetSerial+1^%Library.DynamicAbstractObject.1

Is there a workaround? I need to store moderately large jsons (10-15 Mb) and dynamic object properties allow for a very convenient access.

$ZSTORAGE is set to -1.

Product version: IRIS 2021.2
$ZV: IRIS for Windows (x86-64) 2021.2 (Build 617U) Thu Dec 9 2021 09:54:09 EST
2
0 382
Discussion (6)3
Log in or sign up to continue

A serious problem I also struggled with.
I think inside IRIS there is not much chance to get across the MAXSTRING limit.
 

A possible work-around could be the class below. In short, you work with your json property as intended, merely before saving the object, you save the json-property into a stream and after opening an instance, you restore the json-property from the the stream - that's all. The drawback, no SQL over the json property...

Class DC.Dyn Extends %Persistent
{
Property json As %DynamicObject [ Transient ];
Property jstr As %GlobalCharacterStream [ Internal, Private ];

ClassMethod MyTest(kill = 0)
{
   if kill do ..%KillExtent(1,1)

   set obj=..%New()
   set obj.json.short="A short test text"
   set obj.json.maxstr=$tr($j("",$$$MaxStringLength)," ","X")
   do obj.json.%Set("hugedata",..stream(obj),"stream")

   write "Status : ",obj.%Save(),!
   set id=obj.%Id()
   write "ID : ",id,!
   kill (id)

   set obj=..%OpenId(id)
   write "short : ",obj.json.short,!
   write "maxstr : ",$e(obj.json.maxstr,1,20),"... Size: ",$length(obj.json.maxstr),!
   set stream=obj.json.%Get("hugedata",,"stream")
   write "hugedata: ",stream.Read(20),"... Size: ",stream.Size,!
}

ClassMethod stream(obj)
{
   set stream=##class(%Stream.TmpCharacter).%New()
   do stream.Write(obj.json.short)
   do stream.Write(obj.json.maxstr)
   do stream.Write(obj.json.maxstr)
   quit stream
}

Method %OnOpen() As %Status [ Private, ServerOnly = 1 ]
{
   if ..jstr {
      do ..jstr.Rewind()
      set ..json=##class(%DynamicAbstractObject).%FromJSON(..jstr)
   }
   Quit $$$OK
}

Method %OnAddToSaveSet(depth As %Integer = 3, insert As %Integer = 0, callcount As %Integer = 0) As %Status [ Private, ServerOnly = 1 ]
{
   do ..jstr.Clear(), ..json.%ToJSON(..jstr)
   Quit $$$OK
}
}

Some testing...

IDEV:USER>d ##class(DC.Dyn).MyTest(1)
Status  : 1
ID      : 1
short   : A short test text
maxstr  : XXXXXXXXXXXXXXXXXXXX... Size: 3641144
hugedata: A short test textXXX... Size: 7282305

If your code uses obj.%Reload() then %OnReload() and %OnOpen() should contain the same code.

Great idea!

I'd only add check to prevent rewrite of a stream on every save unless the dynamic object was modified:

Method %OnAddToSaveSet(depth As %Integer = 3, insert As %Integer = 0, callcount As %Integer = 0) As %Status [ Private, ServerOnly = 1 ]
{
   do:m%json ..jstr.Clear(), ..json.%ToJSON(..jstr)
   Quit $$$OK
}

Improvements are always welcome... ;-)

The %GlobalCharacterStream class is deprecated (although I believe it will continue working.)  You should consider replacing it with the %Stream.GlobalCharacter class.
 

Yeah, old habits never die...