%JSON.Adaptor for streams

Okay, we've got a quite useful way to very easily Import and export our objects as JSON, similar to what we already had before for XML.

So, It's a %JSON.Adaptor. But the issue here I faced with, working with Stream properties. 

I have an example, when I generate an object, with stream binary stream properties. Export and Import the same, but getting the different resulting objects, depends on the original size of streams.

Class User.Test Extends (%JSON.Adaptor, %RegisteredObject)
{ 

Property Name As %String(%JSONFIELDNAME = "name"); 

Property HexStream As %Stream.GlobalBinary(%JSONFIELDNAME = "hex_stream", ENCODING = "hex"); 

Property Base64Stream As %Stream.GlobalBinary(%JSONFIELDNAME = "base64_stream", ENCODING = "base64"); 

ClassMethod Test(streamSize = {$system.SYS.MaxLocalLength()})
{
  Set object = ..%New()
  Set object.Name = "testing"
  Set maxLength = $system.SYS.MaxLocalLength()
  If streamSize<0 {
    Set streamSize = maxLength + streamSize
  }
  
  Set parts = (streamSize \ maxLength) + (streamSize # maxLength > 0)
  For p=1:1:parts {
    Set data = ""
    Set length = $Select(streamSize = maxLength: streamSize, p=parts: streamSize # maxLength, 1: maxLength)    
    For i=1:1:length {
      Set data = data _ $Char($Random(256))
    }
    Do object.HexStream.Write(data)
    Do object.Base64Stream.Write(data)
  }  
  Write !,"original HexStream size = ", $Fnumber(object.HexStream.Size, ",")
  Write !,"original Base64Stream size = ", $Fnumber(object.Base64Stream.Size, ",")
 
  Do object.%JSONExportToStream(.jsonStream)
  Kill object   
  
  Write !,"Exported JSON size = ", $Fnumber(jsonStream.Size, ",")
  
  Set newObject = ..%New()
  Set tSC = newObject.%JSONImport(.jsonStream)
  If $$$ISERR(tSC) {
    Do $System.OBJ.DisplayError(tSC)
    Quit 
  }
  
  Write !
  Write !,"imported object HexStream size = ", $Fnumber(newObject.HexStream.Size, ",")
  Write !,"imported object Base64Stream size = ", $Fnumber(newObject.Base64Stream.Size, ",")
} 

}

So, let's to some tests.

Quite small size of the stream. All good.

USER>do ##class(User.Test).Test(10000)    
original HexStream size = 10,000
original Base64Stream size = 10,000
Exported JSON size = 33,389 

imported object HexStream size = 10,000
imported object Base64Stream size = 10,000

Let's make it bigger. One of the resulting streams reached the limit. But the import did not fail, it returns that it was OK. Even when we got broken stream at the end.

USER>do ##class(User.Test).Test(40000) 
original HexStream size = 40,000
original Base64Stream size = 40,000
Exported JSON size = 133,389 

imported object HexStream size = 40,000
imported object Base64Stream size = 32,656

Let's find when it fails for HexStream as well.

USER>do ##class(User.Test).Test($system.SYS.MaxLocalLength()/2) 
original HexStream size = 1,820,572
original Base64Stream size = 1,820,572
Exported JSON size = 6,068,773 

imported object HexStream size = 1,820,572
imported object Base64Stream size = 32,656

USER>do ##class(User.Test).Test($system.SYS.MaxLocalLength()/2+1) 
original HexStream size = 1,820,573
original Base64Stream size = 1,820,573
Exported JSON size = 6,068,779
ERROR #5002: ObjectScript error: <MAXSTRING>%JSONImportInternal+24^User.Test.1

So, it looks like, it does not work at all for streams bigger than 1,820,572, half size of MaxLocalLength (hex format just converts binary to a hex string, so, 1 byte to 2 bytes). And if used base64 for JSON, stream can't be bigger even than just 32656. 

Did I miss anything here, and I can make it work? Is there any way, I can override this behavior somehow, rewrite the way, that will eliminate these errors?

I would split such streams in chunks, as an array of smaller string values during Export. And import would just easily convert it to Stream back.

But the issue also looks like in native JSON, which can store such a big value, but with no way to access it without getting <MAXSTRING> error, or I did not find that way.