New post
Question
· Apr 11, 2022

MAXSTRING limit issue

#JSON #Caché

MAXSTRING (longstring) is enabled.

We have a Class containing a property definition

Property SettingsJSON As %Text(MAXLEN = 3600000)

The property is used for storing a string of JSON data however in some cases we get a Cache error: %SaveData error when trying to save a string of JSON a lot less than maximum "allowed" length, any ideas anyone? 

Product version: Caché 2017.1
Discussion (14)1
Log in or sign up to continue
Vitaliy Serdtsev Apr 11, 2022 to Dmitry Maslennikov

I'm not sure if it's %Text at all.

Made a small example:

Class dc.test Extends %Persistent
{

Index mySimilarityIndex On SettingsJSON(KEYS) [ Data = SettingsJSON(ELEMENTS) ];

Property SettingsJSON As %Text(LANGUAGECLASS = "%Text.English", MAXLEN = 3600000, SIMILARITYINDEX = "mySimilarityIndex");

ClassMethod Test()
{
  d ..%KillExtent()
  
  s json=$tr($j("",3600000)," ","0")
  &sql(insert into del.t(SettingsJSON) values(:json))
  
  w $l(json),":",SQLCODE
}

}

Output:
3600000:0

0 0
Steven Hobbs · Apr 11, 2022

If your long strings are coming from JSON representation then

   Set DynObj=##class(%DynamicObject).%FromJSON(...)

will create a  %Library.DynamicObject or %Library.DynamicArray object in memory containing the JSON array/object elements where the sizes are limited only by the amount of virtual memory the platform will allocate to your process.  A string element of an object/array can have many gigabytes of characters (virtual memory permitting) and you can get the value of such a huge string element in the form of an in-memory, read-only %Stream doing:

   Set StreamVal=DynObj.%Get(key,,"stream")

in cases where DynObj.%Get(key) would get a <MAXSTRING>.

The StreamVal (class %Stream.DynamicBinary or %Stream.DynamicCharacter) is a read-only, random-access %Stream and it shares the same buffer space as the 'key' element of the DynObj (class %Library.DynamicObject) so the in-memory %Stream does not need additional virtual memory.

You can then create a persistent object from a class in the %Steam package (%Stream.GlobalBinary, %Stream.GlobalCharacter, %Stream.FileBinary, %Stream.FileCharacter, or some other appropriate class.)  You can then use the CopyFrom method to populate the persistent %Stream from the  read-only, in-memory %Stream.DynamicBinary/Character.

1 0
Claus Odgaard · Apr 12, 2022

Thanks for the overall input but the problem is that in some cases it seems that a string of json with the length of just a few characters beyond the half of the MAXSTRING limit is unable to saved.

In the class in question there actually is a second property (Property SettingsJSONbackup As %Text(MAXLEN = 3600000)) and it all most seems as if the content of thoose two are calculated (summed) together, but only in same cases?

0 0
Vitaliy Serdtsev Apr 12, 2022 to Claus Odgaard

By default, data in the global is stored as

glbD(ID)=$LB(%%CLASSNAME,prop1,prop2,..,propN)

The total size of the string cannot exceed 3641144. Therefore, if you have a field length >3.6E6, and there are several such fields, the limit is exceeded. To work around this, you need to change storage for your class.

For example so:

glbD(ID)=$LB(%%CLASSNAME)
glbD(ID,"prop1")=prop1
glbD(ID,"prop2")=prop2
...
glbD(ID,"propN")=propN
 
Simple example
1 0