Storing dynamic object properties larger than 3641144 symbols

Question

Question

Eduard Lebedyuk · Jan 27, 2022

I have defined a class with a dynamic object property:

My class

Class test.Dyn Extends %Persistent
{

Property json As %DynamicObject;

/// do ##class(test.Dyn).test()
ClassMethod test()
{
  do ..%KillExtent()
  for len = 100, $$$MaxStringLength - 1, $$$MaxStringLength, $$$MaxStringLength+1, $$$MaxStringLength *2 {
    set sc = ..save(len)
    write "Len: ",len, " Result: ",$case($$$ISERR(sc), $$$YES: $System.Status.GetErrorText(sc), : "OK"),!
  }
}

ClassMethod save(len)
{
  set json = {}
  set stream = ..getStream(len-8) // -8 because {"a":""}
  do json.%Set("a", stream, "stream")
 
  set obj = ..%New()
  set obj.json = json
  quit obj.%Save()
}

ClassMethod getStream(len)
{
  set stream = ##class(%Stream.TmpCharacter).%New()
 
  if len<$$$MaxStringLength {
    do stream.Write($tr($j("",len)," ","A"))
  } else {
    for i=1:$$$MaxStringLength:len {
      do stream.Write($tr($j("",$$$MaxStringLength)," ","A"))
    }
    do stream.Write($tr($j("",len-i)," ","A"))
  }
  quit stream
}
}

The issue I encounter is that if a length of a serialized json property is more than 3641144 symbols, the object fails to save with the following error (either MAXSTRING or STRINGSTACK):

Length: 100 Result: OK
Length: 3641143 Result: OK
Length: 3641144 Result: OK
Length: 3641145 Result: ERROR #5002: ObjectScript error: <MAXSTRING>%GetSerial+1^%Library.DynamicAbstractObject.1
Length: 7282288 Result: ERROR #5002: ObjectScript error: <STRINGSTACK>%GetSerial+1^%Library.DynamicAbstractObject.1

Is there a workaround? I need to store moderately large jsons (10-15 Mb) and dynamic object properties allow for a very convenient access.

$ZSTORAGE is set to -1.

Product version: IRIS 2021.2

$ZV: IRIS for Windows (x86-64) 2021.2 (Build 617U) Thu Dec 9 2021 09:54:09 EST

Discussion (6)3

Log in or sign up to continue

Julius Kavay · Jan 28, 2022

A possible work-around could be the class below. In short, you work with your json property as intended, merely before saving the object, you save the json-property into a stream and after opening an instance, you restore the json-property from the the stream - that's all. The drawback, no SQL over the json property...

Class DC.Dyn Extends %Persistent
{
Property json As %DynamicObject [ Transient ];
Property jstr As %GlobalCharacterStream [ Internal, Private ];

ClassMethod MyTest(kill = 0)
{
   if kill do ..%KillExtent(1,1)

   set obj=..%New()
   set obj.json.short="A short test text"
   set obj.json.maxstr=$tr($j("",$$$MaxStringLength)," ","X")
   do obj.json.%Set("hugedata",..stream(obj),"stream")

   write "Status : ",obj.%Save(),!
   set id=obj.%Id()
   write "ID : ",id,!
   kill (id)

   set obj=..%OpenId(id)
   write "short : ",obj.json.short,!
   write "maxstr : ",$e(obj.json.maxstr,1,20),"... Size: ",$length(obj.json.maxstr),!
   set stream=obj.json.%Get("hugedata",,"stream")
   write "hugedata: ",stream.Read(20),"... Size: ",stream.Size,!
}

ClassMethod stream(obj)
{
   set stream=##class(%Stream.TmpCharacter).%New()
   do stream.Write(obj.json.short)
   do stream.Write(obj.json.maxstr)
   do stream.Write(obj.json.maxstr)
   quit stream
}

Method %OnOpen() As %Status [ Private, ServerOnly = 1 ]
{
   if ..jstr {
      do ..jstr.Rewind()
      set ..json=##class(%DynamicAbstractObject).%FromJSON(..jstr)
   }
   Quit $$$OK
}

Method %OnAddToSaveSet(depth As %Integer = 3, insert As %Integer = 0, callcount As %Integer = 0) As %Status [ Private, ServerOnly = 1 ]
{
   do ..jstr.Clear(), ..json.%ToJSON(..jstr)
   Quit $$$OK
}
}

Some testing...

IDEV:USER>d ##class(DC.Dyn).MyTest(1)
Status  : 1
ID      : 1
short   : A short test text
maxstr  : XXXXXXXXXXXXXXXXXXXX... Size: 3641144
hugedata: A short test textXXX... Size: 7282305

If your code uses obj.%Reload() then %OnReload() and %OnOpen() should contain the same code.

5 1

score 0 · Answer 1 · 2022-01-27T17:37:55-05:00

A serious problem I also struggled with.
I think inside IRIS there is not much chance to get across the MAXSTRING limit.

score 1 · Answer 2 · 2022-01-28T09:28:00-05:00

Great idea!

I'd only add check to prevent rewrite of a stream on every save unless the dynamic object was modified:

Method %OnAddToSaveSet(depth As %Integer = 3, insert As %Integer = 0, callcount As %Integer = 0) As %Status [ Private, ServerOnly = 1 ]
{
   do:m%json ..jstr.Clear(), ..json.%ToJSON(..jstr)
   Quit $$$OK
}

score 0 · Answer 3 · 2022-01-28T09:36:29-05:00

Julius Kavay · Jan 28, 2022

Improvements are always welcome... ;-)

0 0

score 0 · Answer 4 · 2022-01-28T13:10:42-05:00

The %GlobalCharacterStream class is deprecated (although I believe it will continue working.) You should consider replacing it with the %Stream.GlobalCharacter class.

score 0 · Answer 5 · 2022-01-28T14:29:57-05:00

Julius Kavay · Jan 28, 2022

Yeah, old habits never die...

0 0