Size Limitation for %ToJSON() with large stream

Question

Question

Mathieu Van Sevenant · Feb 6, 2020

Hello everyone

Here is a try of sending a stream object with %ToJSON() by HTTP POST request on a remote server.

First the JSON structure:

{
    "document": {
        "id": "{43495441-4445-4C4C-4500-0500129C762E}",
        "patient_id": "5003171TC01",
        "language": "fr",
        "doc_type": "CONS",
        "doc_time": "2019-06-12 11:02:00",
       (...)
        "Sections": [
            {
                "sectionId": "main",
               "Content": “eyJkb2N1bWVudCI6eyJpZCI6Ins0MzQ5NTQ0MS00NDQ1LTRDNEMtNDUwMC0wNTAwMTI5Qzc2MkV9\r\nIiwicGF0aWVudF9pZCI6IjUwMDMxNzFUQzAxIiwibGFuZ3VhZ2UiOiJmciIsImRvY190eXBlIjoi\r\nQ09OU(...)"
           }
        ]
    }
}

Actually the Content property must contain what is inside a base64 stream (e.g. from a PDF file, see code below). No problem for regular small files. But for large ones (up to 60 MB) this property is truncated to a maximum size of 32 kB.

What could we do for removing this size limitation for large streams ?

Here are some lines from the used code:

set httpRequest = ##class(%Net.HttpRequest).%New()
set httpRequest.Server = …..
set httpRequest.Port = ….
set httpRequest.Https = 0
set objDoc = {}
set objInObjDoc = {}
set arr = []
set objInArrMain = {}
set objInArrMain."sectionId" = "main"
do fileStreamBase64.Rewind()
set objInArrMain."Content" = fileStreamBase64.Read()
do arr.%Push(objInArrMain)

set objInObjDoc."id" = request.id
set objInObjDoc."patient_id" = request.patNdos
set objInObjDoc."language" = "fr"
(...)

set objInObjDoc."Sections" = arr

set objDoc."document" = objInObjDoc

set httpRequest.ContentType="application/json"
do httpRequest.EntityBody.Write(objDoc.%ToJSON())
set sc=httpRequest.Post("/index/push", 0)

Building fileStreamBase64:

set fileStream = ##class(%Stream.FileCharacter).%New()
              set fileStreamBase64 = ##class(%Stream.FileCharacter).%New()
              set file=##class(%File).%New(completedFilename)
              Do file.Open("RU")
              Set sc=fileStream.CopyFrom(file)
              Do file.Close()
              Do ##class(HS.Util.StreamUtils).Base64Encode(fileStream,.fileStreamBase64)

Best regards,
Mathieu

Discussion (7)2

Log in or sign up to continue

Mathieu Van Sevenant · Feb 17, 2020

Thanks again to you all,

We have actually built the stream ourselves, helped with some tricks got from the internet, and it seems to work within our test environment. Here is the pseudo-code for memo:

Method OnRequest(request As %Library.Persistent, Output response As %Library.Persistent) As %Status
{
              set httpRequest = ##class(%Net.HttpRequest).%New()
          set httpRequest.Server = "(...)"
          set httpRequest.Port = (...)
              Set filename="(...)"
              Set file=##class(%File).%New(filename)
              Do file.Open("RU")

Set sc=fileStream.CopyFrom(file)
Do file.Close()

do fileStream.Rewind()

              set myStream = ##class(%Stream.TmpCharacter).%New()
              set startJsString = "{""document"":{""id"":""@GUID@"",""language"":""fr"",""patient_id"":""@NDOS@"",""doc_type"":""@DOCUMENTTYPE@"","    //...and so on
              set startJsStringData = $REPLACE(startJsString, "@GUID@", request.id,    1, -1, 1)
              set startJsStringData = $REPLACE(startJsStringData, "@NDOS@", request.patNdos ,    1, -1, 1)
              //...and so on

do myStream.Write(startJsStringData)

s startJsContentString = """sections"":[{""section_id"":""main"",""content"":"""
s endJsString = """}]}}"

            do myStream.Write(startJsContentString)
            set sc = ..Base64Encode(fileStream, myStream,,1)
            do myStream.Write(endJsString)
            do myStream.Rewind()

            set len = 32000
            While (len > -1) {
                set sRead = myStream.Read(.len)
                do httpRequest.EntityBody.Write(sRead)
            }

set sc=httpRequest.Post("/index/push", 0)

}

ClassMethod Base64Encode(tIn As %Stream.FileCharacter, tOut As %Stream.TmpCharacter, chunk As %Integer = 32000, Flags As %Integer = 1) As %Status
{
   set sc = $$$OK
   if $g(tIn)="" quit $$$ERROR(5001, "Input stream required")
   if '$IsObject(tIn) quit $$$ERROR(5001,"Input is not a stream object")
   if 'tIn.%IsA("%Stream.Object") quit $$$ERROR(5001,"Input object is not a stream")
   if 'tOut.%IsA("%Stream.Object") quit $$$ERROR(5001,"Output object is not a stream")
   set chunk=chunk-(chunk#3)
   do tIn.Rewind()
   While 'tIn.AtEnd {
    set sc= tOut.Write($SYSTEM.Encryption.Base64Encode(tIn.Read(chunk),Flags))
    if 'sc Quit
   }
   Quit sc
}

Best regards

0 0

score 1 · Answer 1 · 2020-02-06T06:01:10-05:00

Read method does read 32000 characters by default, you can specify an amount of characters up to $$$MaxStringLength (3 641 144 characters).

If you want to set property to a stream you can do it like this:

do objInArrMain.%Set("Content", fileStreamBase64, "stream")

I'm not sure when it became available, here's a simple check code:

set stream = ##class(%Stream.TmpCharacter).%New()
do stream.Write(123)
d obj.%Set("prop", stream, "stream")

It should output: {"prop":"123"}

One other note: you're using file streams for temporary outputs (when building fileStreamBase64), replace file stream with temp stream for better performance.

score 0 · Answer 2 · 2020-02-06T08:09:48-05:00

Thank you for your quick answer, Eduard.

Unfortunately, it seems that our version (2017.2.2) does not support the code you were suggesting to me, as we got an "Illegal value" error when testing it. So I think we will have to look for another way to transfer the data.

Best regards anyway !

score 1 · Answer 3 · 2020-02-07T06:45:49-05:00

Eduard Lebedyuk · Feb 7, 2020

You can try %ZEN.Auxiliary.altJSONProvider or %ZEN.Auxiliary.JSONProvider.

1 0

score 1 · Answer 4 · 2020-02-10T18:49:29-05:00

Extension of the %Set and %Get methods of the %DynamicAbstractObject classes to support type parameters (such as using "stream" as a type) is an IRIS feature and not a Caché feature. In Caché the %ToJSON and %FromJSON methods can accept %File-s and %Stream-s. These methods are generally limited to Dynamic Objects/Arrays involving sizes related to the largest 32-bit signed integer. But in Caché the %ToJSON/%FromJSON methods work only on entire Dynamic Objects/Arrays. Using %Set/%Get in Caché to modify an element of a Dynamic Object/Array is limited by the length of an ObjectScript %String (currently 3,641,144 characters in Caché.)

In recent and future IRIS releases the sizes of a %DynamicAbstractObject will be limited by the amount of virtual memory available to the process. (Please consider avoiding the activation of many multi-gigabyte Dynamic objects/arrays at the same time.) The future IRIS versions of the %Get/%Set methods will support type keywords involving streams which will allow a %Get/%Set to access Dynamic Object/Array elements involving any size that can be fetched from (or stored into) a %Stream. Future versions of these methods will also encode and decode Base64 representation while transferring byte data between a %Stream and an element of a Dynamic Object/Array.

score 1 · Answer 5 · 2020-02-11T15:57:30-05:00

Rubens Silva · Feb 11, 2020

For your version, you'll need to create your own serializer.

Here's an idea on how to do it.

1 0

score 1 · Answer 6 · 2020-02-17T10:13:46-05:00

You can replace

do myStream.Rewind()
set len = 32000
While (len > -1) {
    set sRead = myStream.Read(.len)
    do httpRequest.EntityBody.Write(sRead)
}

With

set sc = httpRequest.EntityBody.CopyFrom(myStream)