Question
Claus Odgaard · Apr 11

MAXSTRING limit issue

MAXSTRING (longstring) is enabled.

We have a Class containing a property definition

Property SettingsJSON As %Text(MAXLEN = 3600000)

The property is used for storing a string of JSON data however in some cases we get a Cache error: %SaveData error when trying to save a string of JSON a lot less than maximum "allowed" length, any ideas anyone? 

Product version: Caché 2017.1
0
0 423
Discussion (14)2
Log in or sign up to continue

Unfortunately not possible to make such changes currently.

But thanks anyway.

It's the only reasonable way to go. It's a bad idea to store such data as %String

I would suggest wrapping this field with Get and Set to make it compatible with the previous storage

I'm not sure if it's %Text at all.

Made a small example:

Class dc.test Extends %Persistent
{

Index mySimilarityIndex On SettingsJSON(KEYS) [ Data = SettingsJSON(ELEMENTS) ];

Property SettingsJSON As %Text(LANGUAGECLASS "%Text.English"MAXLEN 3600000SIMILARITYINDEX "mySimilarityIndex");

ClassMethod Test()
{
  ..%KillExtent()
  
  json=$tr($j("",3600000)," ","0")
  &sql(insert into del.t(SettingsJSONvalues(:json))
  
  w $l(json),":",SQLCODE
}

}

Output:
3600000:0

If your long strings are coming from JSON representation then

   Set DynObj=##class(%DynamicObject).%FromJSON(...)

will create a  %Library.DynamicObject or %Library.DynamicArray object in memory containing the JSON array/object elements where the sizes are limited only by the amount of virtual memory the platform will allocate to your process.  A string element of an object/array can have many gigabytes of characters (virtual memory permitting) and you can get the value of such a huge string element in the form of an in-memory, read-only %Stream doing:

   Set StreamVal=DynObj.%Get(key,,"stream")

in cases where DynObj.%Get(key) would get a <MAXSTRING>.

The StreamVal (class %Stream.DynamicBinary or %Stream.DynamicCharacter) is a read-only, random-access %Stream and it shares the same buffer space as the 'key' element of the DynObj (class %Library.DynamicObject) so the in-memory %Stream does not need additional virtual memory.

You can then create a persistent object from a class in the %Steam package (%Stream.GlobalBinary, %Stream.GlobalCharacter, %Stream.FileBinary, %Stream.FileCharacter, or some other appropriate class.)  You can then use the CopyFrom method to populate the persistent %Stream from the  read-only, in-memory %Stream.DynamicBinary/Character.

Thanks for the overall input but the problem is that in some cases it seems that a string of json with the length of just a few characters beyond the half of the MAXSTRING limit is unable to saved.

In the class in question there actually is a second property (Property SettingsJSONbackup As %Text(MAXLEN = 3600000)) and it all most seems as if the content of thoose two are calculated (summed) together, but only in same cases?

By default, data in the global is stored as

glbD(ID)=$LB(%%CLASSNAME,prop1,prop2,..,propN)

The total size of the string cannot exceed 3641144. Therefore, if you have a field length >3.6E6, and there are several such fields, the limit is exceeded. To work around this, you need to change storage for your class.

For example so:

glbD(ID)=$LB(%%CLASSNAME)
glbD(ID,"prop1")=prop1
glbD(ID,"prop2")=prop2
...
glbD(ID,"propN")=propN
 
Simple example

Thank you. I think you just solved my problem.