MAXSTRING limit issue
MAXSTRING (longstring) is enabled.
We have a Class containing a property definition
Property SettingsJSON As %Text(MAXLEN = 3600000)
The property is used for storing a string of JSON data however in some cases we get a Cache error: %SaveData error when trying to save a string of JSON a lot less than maximum "allowed" length, any ideas anyone?
Comments
Why not store it as %Stream.GlobalCharacter
Unfortunately not possible to make such changes currently.
But thanks anyway.
It's the only reasonable way to go. It's a bad idea to store such data as %String
I would suggest wrapping this field with Get and Set to make it compatible with the previous storage
For %Stream.GlobalCharacter cannot use an index, but it can be used for %Text.
Not sure that indexing whole JSON is a case for them
I'm not sure if it's %Text at all.
Made a small example:
Class dc.test Extends %Persistent
{
Index mySimilarityIndex On SettingsJSON(KEYS) [ Data = SettingsJSON(ELEMENTS) ];
Property SettingsJSON As %Text(LANGUAGECLASS = "%Text.English", MAXLEN = 3600000, SIMILARITYINDEX = "mySimilarityIndex");
ClassMethod Test()
{
d ..%KillExtent()
s json=$tr($j("",3600000)," ","0")
&sql(insert into del.t(SettingsJSON) values(:json))
w $l(json),":",SQLCODE
}
}
Output:
3600000:0
Please see my addon...
Please see my addon...
Maybe you will help the trick taken from Storing dynamic object properties larger than 3641144 symbols
If your long strings are coming from JSON representation then
Set DynObj=##class(%DynamicObject).%FromJSON(...)
will create a %Library.DynamicObject or %Library.DynamicArray object in memory containing the JSON array/object elements where the sizes are limited only by the amount of virtual memory the platform will allocate to your process. A string element of an object/array can have many gigabytes of characters (virtual memory permitting) and you can get the value of such a huge string element in the form of an in-memory, read-only %Stream doing:
Set StreamVal=DynObj.%Get(key,,"stream")
in cases where DynObj.%Get(key) would get a <MAXSTRING>.
The StreamVal (class %Stream.DynamicBinary or %Stream.DynamicCharacter) is a read-only, random-access %Stream and it shares the same buffer space as the 'key' element of the DynObj (class %Library.DynamicObject) so the in-memory %Stream does not need additional virtual memory.
You can then create a persistent object from a class in the %Steam package (%Stream.GlobalBinary, %Stream.GlobalCharacter, %Stream.FileBinary, %Stream.FileCharacter, or some other appropriate class.) You can then use the CopyFrom method to populate the persistent %Stream from the read-only, in-memory %Stream.DynamicBinary/Character.
Please see my addon...
Thanks for the overall input but the problem is that in some cases it seems that a string of json with the length of just a few characters beyond the half of the MAXSTRING limit is unable to saved.
In the class in question there actually is a second property (Property SettingsJSONbackup As %Text(MAXLEN = 3600000)) and it all most seems as if the content of thoose two are calculated (summed) together, but only in same cases?
By default, data in the global is stored as
glbD(ID)=$LB(%%CLASSNAME,prop1,prop2,..,propN)
The total size of the string cannot exceed 3641144. Therefore, if you have a field length >3.6E6, and there are several such fields, the limit is exceeded. To work around this, you need to change storage for your class.
For example so:
glbD(ID)=$LB(%%CLASSNAME) glbD(ID,"prop1")=prop1 glbD(ID,"prop2")=prop2 ... glbD(ID,"propN")=propN
Simple example
Class dc.test Extends %Persistent
{
Index mySimilarityIndex On SettingsJSON(KEYS) [ Data = SettingsJSON(ELEMENTS) ];
Index mySimilaritybackupIndex On SettingsJSONbackup(KEYS) [ Data = SettingsJSONbackup(ELEMENTS) ];
Property SettingsJSON As %Text(LANGUAGECLASS = "%Text.English", MAXLEN = 3600000, SIMILARITYINDEX = "mySimilarityIndex");
Property SettingsJSONbackup As %Text(LANGUAGECLASS = "%Text.English", MAXLEN = 3600000, SIMILARITYINDEX = "mySimilaritybackupIndex");
ClassMethod Test()
{
d ..%KillExtent()
s json=$tr($j("",3600000)," ","0")
&sql(insert into dc.test(SettingsJSON,SettingsJSONbackup) values(:json,:json))
w $l(json),":",SQLCODE
}
Storage Default
{
<Data name="backup">
<Attribute>SettingsJSONbackup</Attribute>
<Structure>node</Structure>
<Subscript>"SettingsJSONbackup"</Subscript>
</Data>
<Data name="json">
<Attribute>SettingsJSON</Attribute>
<Structure>node</Structure>
<Subscript>"SettingsJSON"</Subscript>
</Data>
<Data name="testDefaultData">
<Value name="1">
<Value>%%CLASSNAME</Value>
</Value>
</Data>
<DataLocation>^dc.testD</DataLocation>
<DefaultData>testDefaultData</DefaultData>
<IdLocation>^dc.testD</IdLocation>
<IndexLocation>^dc.testI</IndexLocation>
<StreamLocation>^dc.testS</StreamLocation>
<Type>%Library.CacheStorage</Type>
}
}Thank you. I think you just solved my problem.