Storing dynamic object properties larger than 3641144 symbols
I have defined a class with a dynamic object property:
My class
Class test.Dyn Extends %Persistent
{
Property json As %DynamicObject;
/// do ##class(test.Dyn).test()
ClassMethod test()
{
do ..%KillExtent()
for len = 100, $$$MaxStringLength - 1, $$$MaxStringLength, $$$MaxStringLength+1, $$$MaxStringLength *2 {
set sc = ..save(len)
write "Len: ",len, " Result: ",$case($$$ISERR(sc), $$$YES: $System.Status.GetErrorText(sc), : "OK"),!
}
}
ClassMethod save(len)
{
set json = {}
set stream = ..getStream(len-8) // -8 because {"a":""}
do json.%Set("a", stream, "stream")
set obj = ..%New()
set obj.json = json
quit obj.%Save()
}
ClassMethod getStream(len)
{
set stream = ##class(%Stream.TmpCharacter).%New()
if len<$$$MaxStringLength {
do stream.Write($tr($j("",len)," ","A"))
} else {
for i=1:$$$MaxStringLength:len {
do stream.Write($tr($j("",$$$MaxStringLength)," ","A"))
}
do stream.Write($tr($j("",len-i)," ","A"))
}
quit stream
}
}The issue I encounter is that if a length of a serialized json property is more than 3641144 symbols, the object fails to save with the following error (either MAXSTRING or STRINGSTACK):
Length: 100 Result: OK
Length: 3641143 Result: OK
Length: 3641144 Result: OK
Length: 3641145 Result: ERROR #5002: ObjectScript error: <MAXSTRING>%GetSerial+1^%Library.DynamicAbstractObject.1
Length: 7282288 Result: ERROR #5002: ObjectScript error: <STRINGSTACK>%GetSerial+1^%Library.DynamicAbstractObject.1Is there a workaround? I need to store moderately large jsons (10-15 Mb) and dynamic object properties allow for a very convenient access.
$ZSTORAGE is set to -1.
Comments
A serious problem I also struggled with.
I think inside IRIS there is not much chance to get across the MAXSTRING limit.
A possible work-around could be the class below. In short, you work with your json property as intended, merely before saving the object, you save the json-property into a stream and after opening an instance, you restore the json-property from the the stream - that's all. The drawback, no SQL over the json property...
Class DC.Dyn Extends %Persistent
{
Property json As %DynamicObject [ Transient ];
Property jstr As %GlobalCharacterStream [ Internal, Private ];
ClassMethod MyTest(kill = 0)
{
if kill do ..%KillExtent(1,1)
set obj=..%New()
set obj.json.short="A short test text"
set obj.json.maxstr=$tr($j("",$$$MaxStringLength)," ","X")
do obj.json.%Set("hugedata",..stream(obj),"stream")
write "Status : ",obj.%Save(),!
set id=obj.%Id()
write "ID : ",id,!
kill (id)
set obj=..%OpenId(id)
write "short : ",obj.json.short,!
write "maxstr : ",$e(obj.json.maxstr,1,20),"... Size: ",$length(obj.json.maxstr),!
set stream=obj.json.%Get("hugedata",,"stream")
write "hugedata: ",stream.Read(20),"... Size: ",stream.Size,!
}
ClassMethod stream(obj)
{
set stream=##class(%Stream.TmpCharacter).%New()
do stream.Write(obj.json.short)
do stream.Write(obj.json.maxstr)
do stream.Write(obj.json.maxstr)
quit stream
}
Method %OnOpen() As %Status [ Private, ServerOnly = 1 ]
{
if ..jstr {
do ..jstr.Rewind()
set ..json=##class(%DynamicAbstractObject).%FromJSON(..jstr)
}
Quit $$$OK
}
Method %OnAddToSaveSet(depth As %Integer = 3, insert As %Integer = 0, callcount As %Integer = 0) As %Status [ Private, ServerOnly = 1 ]
{
do ..jstr.Clear(), ..json.%ToJSON(..jstr)
Quit $$$OK
}
}Some testing...
IDEV:USER>d ##class(DC.Dyn).MyTest(1)
Status : 1
ID : 1
short : A short test text
maxstr : XXXXXXXXXXXXXXXXXXXX... Size: 3641144
hugedata: A short test textXXX... Size: 7282305If your code uses obj.%Reload() then %OnReload() and %OnOpen() should contain the same code.
Great idea!
I'd only add check to prevent rewrite of a stream on every save unless the dynamic object was modified:
Method %OnAddToSaveSet(depth As %Integer = 3, insert As %Integer = 0, callcount As %Integer = 0) As %Status [ Private, ServerOnly = 1 ]
{
do:m%json ..jstr.Clear(), ..json.%ToJSON(..jstr)
Quit $$$OK
}Improvements are always welcome... ;-)
The %GlobalCharacterStream class is deprecated (although I believe it will continue working.) You should consider replacing it with the %Stream.GlobalCharacter class.
Yeah, old habits never die...
I have a similar problem when reading CCDA with narrative text from one of the section has text size larger than 3.4mb. I have FHIR composition resource build and the FHIR Section narrative text div has a maxlength = 1000000. Even if i increase, i can hold max of 3.4mb.. But i do get text size larger than this.
Just to mention there is another option to storing large JSON objects. You could use DocDB for unstructured JSON objects i.e. if the JSON structure is unknown. Or, my preference, if the structure is well known, you could use %JSONAdaptor to map a Storage class to the same values in your json string and then just import the json, ending up with an IRIS Persistent object.