Question
Dmitriy Maslennikov · Apr 30, 2020

%JSON.Adaptor for streams

Okay, we've got a quite useful way to very easily Import and export our objects as JSON, similar to what we already had before for XML.

So, It's a %JSON.Adaptor. But the issue here I faced with, working with Stream properties. 

I have an example, when I generate an object, with stream binary stream properties. Export and Import the same, but getting the different resulting objects, depends on the original size of streams.

Class User.Test Extends (%JSON.Adaptor, %RegisteredObject)
{ 

Property Name As %String(%JSONFIELDNAME = "name"); 

Property HexStream As %Stream.GlobalBinary(%JSONFIELDNAME = "hex_stream", ENCODING = "hex"); 

Property Base64Stream As %Stream.GlobalBinary(%JSONFIELDNAME = "base64_stream", ENCODING = "base64"); 

ClassMethod Test(streamSize = {$system.SYS.MaxLocalLength()})
{
  Set object = ..%New()
  Set object.Name = "testing"
  Set maxLength = $system.SYS.MaxLocalLength()
  If streamSize<0 {
    Set streamSize = maxLength + streamSize
  }
  
  Set parts = (streamSize \ maxLength) + (streamSize # maxLength > 0)
  For p=1:1:parts {
    Set data = ""
    Set length = $Select(streamSize = maxLength: streamSize, p=parts: streamSize # maxLength, 1: maxLength)    
    For i=1:1:length {
      Set data = data _ $Char($Random(256))
    }
    Do object.HexStream.Write(data)
    Do object.Base64Stream.Write(data)
  }  
  Write !,"original HexStream size = ", $Fnumber(object.HexStream.Size, ",")
  Write !,"original Base64Stream size = ", $Fnumber(object.Base64Stream.Size, ",")
 
  Do object.%JSONExportToStream(.jsonStream)
  Kill object   
  
  Write !,"Exported JSON size = ", $Fnumber(jsonStream.Size, ",")
  
  Set newObject = ..%New()
  Set tSC = newObject.%JSONImport(.jsonStream)
  If $$$ISERR(tSC) {
    Do $System.OBJ.DisplayError(tSC)
    Quit 
  }
  
  Write !
  Write !,"imported object HexStream size = ", $Fnumber(newObject.HexStream.Size, ",")
  Write !,"imported object Base64Stream size = ", $Fnumber(newObject.Base64Stream.Size, ",")
} 

}

So, let's to some tests.

Quite small size of the stream. All good.

USER>do ##class(User.Test).Test(10000)    
original HexStream size = 10,000
original Base64Stream size = 10,000
Exported JSON size = 33,389 

imported object HexStream size = 10,000
imported object Base64Stream size = 10,000

Let's make it bigger. One of the resulting streams reached the limit. But the import did not fail, it returns that it was OK. Even when we got broken stream at the end.

USER>do ##class(User.Test).Test(40000) 
original HexStream size = 40,000
original Base64Stream size = 40,000
Exported JSON size = 133,389 

imported object HexStream size = 40,000
imported object Base64Stream size = 32,656

Let's find when it fails for HexStream as well.

USER>do ##class(User.Test).Test($system.SYS.MaxLocalLength()/2) 
original HexStream size = 1,820,572
original Base64Stream size = 1,820,572
Exported JSON size = 6,068,773 

imported object HexStream size = 1,820,572
imported object Base64Stream size = 32,656

USER>do ##class(User.Test).Test($system.SYS.MaxLocalLength()/2+1) 
original HexStream size = 1,820,573
original Base64Stream size = 1,820,573
Exported JSON size = 6,068,779
ERROR #5002: ObjectScript error: <MAXSTRING>%JSONImportInternal+24^User.Test.1

So, it looks like, it does not work at all for streams bigger than 1,820,572, half size of MaxLocalLength (hex format just converts binary to a hex string, so, 1 byte to 2 bytes). And if used base64 for JSON, stream can't be bigger even than just 32656. 

 

Did I miss anything here, and I can make it work? Is there any way, I can override this behavior somehow, rewrite the way, that will eliminate these errors?

I would split such streams in chunks, as an array of smaller string values during Export. And import would just easily convert it to Stream back.

But the issue also looks like in native JSON, which can store such a big value, but with no way to access it without getting <MAXSTRING> error, or I did not find that way.

00
1 5 175 1

Replies

There's a workaround for import:

  Set dynObj = {}.%FromJSON(jsonStream)
  Set hexStream = dynObj.%Get("hex_stream",,"stream")
  Set base64Stream = dynObj.%Get("base64_stream",,"stream")
 
  Set dynObj."hex_stream" = ""
  Set dynObj."base64_stream" = ""
  Do dynObj.%Remove("hex_stream")
  Do dynObj.%Remove("base64_stream")
 
  Set newObject = ..%New()
  Set newObject.HexStream = hexStream
  Set newObject.Base64Stream = base64Stream
 
  Set tSC = newObject.%JSONImport(dynObj)

Thanks, a lot, it may help.

But unfortunately, Export in case of Base64 does not work correctly, and anyway have to be fixed. Due to the specialty of Base64, a string which has to be encoded in a chunked manner, have to be devisable by 3. At the moment it reads original stream with default size of chunk, 32656 (instead of 32656\3*3=32655), so, base64 becomes incorrect, and decoded back is different from the original.

Unfortunately, there are errors in the methods GenExportBinaryStream/GenImportBinaryStream in the class %JSON.Generator.

Yeah, already reported on WRC, with workaround