Storing dynamic object properties larger than 3641144 symbols

Question

Question

Eduard Lebedyuk · Jan 27, 2022

I have defined a class with a dynamic object property:

My class

Class test.Dyn Extends %Persistent
{

Property json As %DynamicObject;

/// do ##class(test.Dyn).test()
ClassMethod test()
{
  do ..%KillExtent()
  for len = 100, $$$MaxStringLength - 1, $$$MaxStringLength, $$$MaxStringLength+1, $$$MaxStringLength *2 {
    set sc = ..save(len)
    write "Len: ",len, " Result: ",$case($$$ISERR(sc), $$$YES: $System.Status.GetErrorText(sc), : "OK"),!
  }
}

ClassMethod save(len)
{
  set json = {}
  set stream = ..getStream(len-8) // -8 because {"a":""}
  do json.%Set("a", stream, "stream")
 
  set obj = ..%New()
  set obj.json = json
  quit obj.%Save()
}

ClassMethod getStream(len)
{
  set stream = ##class(%Stream.TmpCharacter).%New()
 
  if len<$$$MaxStringLength {
    do stream.Write($tr($j("",len)," ","A"))
  } else {
    for i=1:$$$MaxStringLength:len {
      do stream.Write($tr($j("",$$$MaxStringLength)," ","A"))
    }
    do stream.Write($tr($j("",len-i)," ","A"))
  }
  quit stream
}
}

The issue I encounter is that if a length of a serialized json property is more than 3641144 symbols, the object fails to save with the following error (either MAXSTRING or STRINGSTACK):

Length: 100 Result: OK
Length: 3641143 Result: OK
Length: 3641144 Result: OK
Length: 3641145 Result: ERROR #5002: ObjectScript error: <MAXSTRING>%GetSerial+1^%Library.DynamicAbstractObject.1
Length: 7282288 Result: ERROR #5002: ObjectScript error: <STRINGSTACK>%GetSerial+1^%Library.DynamicAbstractObject.1

Is there a workaround? I need to store moderately large jsons (10-15 Mb) and dynamic object properties allow for a very convenient access.

$ZSTORAGE is set to -1.

Product version: IRIS 2021.2

$ZV: IRIS for Windows (x86-64) 2021.2 (Build 617U) Thu Dec 9 2021 09:54:09 EST

Discussion (8)3

Log in or sign up to continue

Julius Kavay · Jan 28, 2022

A possible work-around could be the class below. In short, you work with your json property as intended, merely before saving the object, you save the json-property into a stream and after opening an instance, you restore the json-property from the the stream - that's all. The drawback, no SQL over the json property...

Class DC.Dyn Extends %Persistent
{
Property json As %DynamicObject [ Transient ];
Property jstr As %GlobalCharacterStream [ Internal, Private ];

ClassMethod MyTest(kill = 0)
{
   if kill do ..%KillExtent(1,1)

   set obj=..%New()
   set obj.json.short="A short test text"
   set obj.json.maxstr=$tr($j("",$$$MaxStringLength)," ","X")
   do obj.json.%Set("hugedata",..stream(obj),"stream")

   write "Status : ",obj.%Save(),!
   set id=obj.%Id()
   write "ID : ",id,!
   kill (id)

   set obj=..%OpenId(id)
   write "short : ",obj.json.short,!
   write "maxstr : ",$e(obj.json.maxstr,1,20),"... Size: ",$length(obj.json.maxstr),!
   set stream=obj.json.%Get("hugedata",,"stream")
   write "hugedata: ",stream.Read(20),"... Size: ",stream.Size,!
}

ClassMethod stream(obj)
{
   set stream=##class(%Stream.TmpCharacter).%New()
   do stream.Write(obj.json.short)
   do stream.Write(obj.json.maxstr)
   do stream.Write(obj.json.maxstr)
   quit stream
}

Method %OnOpen() As %Status [ Private, ServerOnly = 1 ]
{
   if ..jstr {
      do ..jstr.Rewind()
      set ..json=##class(%DynamicAbstractObject).%FromJSON(..jstr)
   }
   Quit $$$OK
}

Method %OnAddToSaveSet(depth As %Integer = 3, insert As %Integer = 0, callcount As %Integer = 0) As %Status [ Private, ServerOnly = 1 ]
{
   do ..jstr.Clear(), ..json.%ToJSON(..jstr)
   Quit $$$OK
}
}

Some testing...

IDEV:USER>d ##class(DC.Dyn).MyTest(1)
Status  : 1
ID      : 1
short   : A short test text
maxstr  : XXXXXXXXXXXXXXXXXXXX... Size: 3641144
hugedata: A short test textXXX... Size: 7282305

If your code uses obj.%Reload() then %OnReload() and %OnOpen() should contain the same code.

5 1

score 0 · Answer 1 · 2022-01-27T17:37:55-05:00

A serious problem I also struggled with.
I think inside IRIS there is not much chance to get across the MAXSTRING limit.

score 1 · Answer 2 · 2022-01-28T09:28:00-05:00

Great idea!

I'd only add check to prevent rewrite of a stream on every save unless the dynamic object was modified:

Method %OnAddToSaveSet(depth As %Integer = 3, insert As %Integer = 0, callcount As %Integer = 0) As %Status [ Private, ServerOnly = 1 ]
{
   do:m%json ..jstr.Clear(), ..json.%ToJSON(..jstr)
   Quit $$$OK
}

score 0 · Answer 3 · 2022-01-28T09:36:29-05:00

Julius Kavay · Jan 28, 2022

Improvements are always welcome... ;-)

0 0

score 0 · Answer 4 · 2022-01-28T13:10:42-05:00

The %GlobalCharacterStream class is deprecated (although I believe it will continue working.) You should consider replacing it with the %Stream.GlobalCharacter class.

score 0 · Answer 5 · 2022-01-28T14:29:57-05:00

Julius Kavay · Jan 28, 2022

Yeah, old habits never die...

0 0

score 0 · Answer 6 · 2025-08-29T11:17:53-04:00

I have a similar problem when reading CCDA with narrative text from one of the section has text size larger than 3.4mb. I have FHIR composition resource build and the FHIR Section narrative text div has a maxlength = 1000000. Even if i increase, i can hold max of 3.4mb.. But i do get text size larger than this.

score 0 · Answer 7 · 2025-08-31T19:52:49-04:00

Just to mention there is another option to storing large JSON objects. You could use DocDB for unstructured JSON objects i.e. if the JSON structure is unknown. Or, my preference, if the structure is well known, you could use %JSONAdaptor to map a Storage class to the same values in your json string and then just import the json, ending up with an IRIS Persistent object.

JSON Adaptor