Question
John Flippance · Sep 27

Strings Longer than "Long Strings"...

Hello!

Am after some advice on how to approach an issue I am currently having with one of our interfaces within HealthShare HealthConnect.

We send documents to a third party system via their API. 

The API contains a string property, which you use to pass in the document metadata and the Base64 encoded document itself.

Property rsXML As%String;

However, a few times a day we are getting errors as some documents exceed the ‘long string’ length within HealthShare for Strings.

Is anyone able to please advise how I should be passing this through?  The supplier has confirmed they can take larger documents, so am trying to work out how I pass these through?  Am playing around with changing the Request object property to a Stream but not having much joy?

Product version: HealthShare 2018.1
0
0 320
Discussion (18)2
Log in or sign up to continue

Just a dumb question, what is the very difference between %VarString and %String regarding the storable string length? As far as I know, both have a length limit of 3641144 characters.

%String as defined by OP (without MAXLEN) only holds 50 characters, %VarString holds 3641144.

So one has a DEFAULT length of 50 and the other 3641144. But I thought more of the difference HOW MANY characters can they hold.

So VarString does not help me in this situation... as it holds the same amount as a 'Long String' (which isnt enough for some of my documents),

I have nothing in common with HealtShare... but a property (Property rsXML As %String), like you mentioned, can hold up to 3.6E6 chars. If your data are longer then 3641144 characters then you definitely need to use streams (and if you intend to Base64 encode the data, then your data will be 33% longer)! You need to change the above property to something like

Property rsXML As %GlobalBinaryStream;

BUT this means, you have to adapt all your programs, methods, etc. where this property is in use!

That won't be a joy rather a pain

Thanks Julius,

I have a bad feeling this may be the case, will contact the supplier and see if their API can take a stream property instead...

The only important thing is,  do they accept data with the arbitrary length. It's not of importance for you, how their API the  incomming data handles: as string, as longstring, as stream, as array of bytes, etc.

Hello,

Have solved this, the third party API accepted the stream as a string, therefore I went through and changed the generated classes accordingly, and then fed in the stream.

Nice and congratulation. One problem less on this world...

The %Library.DynamicArray and %Library.DynamicObject class objects are stored only in process memory and are not affected by the 3641144 character string length of ObjectScript strings.  But they are affected by the amount of virtual memory that the operating system will grant to a process so avoid having many such large things active.  These Dynamic Objects generally hold JSON format data but they can also hold ObjectScript values.  The %ToJSON(output) method can change the contents of Dynamic Object into JSON text and it will send that text to 'output', which can be a file, a device or a %Stream.  If you need to persist a Dynamic Object in a database then you can use %ToJSON(globalstream) to persist the JSON text in a %Stream.GlobalCharacter or %Stream.GlobalBinary class object.  There is a %FromJSON(input) which can build a new Dynamic Object from 'input' which is JSON text from a string, a file, a device or a %Stream.  In recent versions of IRIS, the %Get(key,default,type) and %Set(key,value,type) methods can let you choose 'type' as some form of %Stream so you can examine and modify elements of a Dynamic Object when that element needs to be larger than 3614411 characters.  The default %Stream-s generated by the %Get method use the %Stream.DynamicBinary and %String.DynamicCharacter classes, which also store data in process memory so you should avoid having many such large streams active at the same time.  They can be copied out to other streams which can be stored a database or into a file or a device.

If I define %DynamicObject property and its serialization is longer than 3641144 characters, would that work?

Yes, it would

kill
set $p(data,"x",2E6+1)=""
set dynArr=[].%Push(data).%Push(data).%Push(data)
set stream=##class(%Stream.GlobalCharacter).%New()
do dynArr.%ToJSON(stream)
try { set x=dynArr.%ToJSON() } catch e { write e.Name,! }
write stream.Size,!
write $length(data),!

The output is

<MAXSTRING>
6000010
2000000

As Steven Hobbs  wrote, the limit is only set by the size of the (virtual) memory.

Does not work with dynamic object class properties unfortunately:

Class User.DO Extends %Persistent
{

Property MyProp As %DynamicObject;

/// do ##class(User.DO).Test()
ClassMethod Test()
{
    do ..%KillExtent()
    // 11 = $length({"prop":""}.%ToJSON())
    for len = 100, $$$MaxStringLength - 11, $$$MaxStringLength - 11 + 1 {
        set sc = ..Create(len)
        write "len: ", len, " result: ", $case($$$ISOK(sc), $$$YES: "OK", : "ERROR: " _ $system.Status.GetErrorText(sc)), !
        quit:$$$ISERR(sc)
    }
}

ClassMethod Create(len) As %Status
{
    set obj = ..%New()
    do obj.MyProp.%Set("prop", ..GetStream(len), "stream")
    set sc = obj.%Save()
    quit sc
}

ClassMethod GetStream(len) As %Stream.TmpCharacter
{
    set chunk = 1000000
    set stream = ##class(%Stream.TmpCharacter).%New()
    for i=1:chunk:len-chunk {
        do stream.Write($tr($j("", chunk)," ", 0))		
    }
    do stream.Write($tr($j("", len#chunk)," ", 0))
    quit stream
}

}

Results in:

len: 100 result: OK
len: 3641133 result: OK
len: 3641134 result: ERROR: ERROR #5002: ObjectScript error: <MAXSTRING>%GetSerial+1^%Library.DynamicAbstractObject.1 [%GetSerial+1^%Library.DynamicAbstractObject.1:XF]

Same issue if there are several short properties in dynamic object with total length > 3641144 characters.

We need something like:

Property MyProp As %DynamicObject(STORAGE="stream");

You try to store a whole stream in one property of an persistent class, that won't work! But the solution is already there,  see this article

Riiight, I thought we had this discussion.

Thanks for finding it.