New post
Question
· Jul 16, 2021

JSON serialization of stream

#InterSystems IRIS

Hi.

I'm trying to serialize/deserialize an .PNG picture. After serialization in C# (using System.Text.Json.JsonSerializer) and sending it to IRIS it is deserialized with %JSONImport method and saved to %FileBinaryStream. File is created on server and can be normaly opened with Paint. But when I want to recall it in C# (serialization with %JSONExportToString and deserialization with System.Text.Json.JsonSerializer), I get an error "FormatException: Cannot decode JSON text that is not encoded as valid Base64 to bytes". Why %JSONExportToString does not encode stream as Base64 string and how to solve/bypass it?

 

Regards,

Matjaž

Product version: IRIS 2020.1
$ZV: IRIS for Windows (x86-64) 2020.1 (Build 215U) Mon Mar 30 2020 20:14:33 EDT
Discussion (12)4
Log in or sign up to continue
Julius Kavay · Jul 16, 2021 

Class Test.JD Extends (%Persistent, %JSON.Adaptor) [ Language = objectscript ]
{
Property Name As %String;
Property Type As %String;
Property Image As %Stream.GlobalBinary;
}

set obj=##class(Test.JD).%New()
set obj.Name="Joe"
do obj.Image.Write($system.Encryption.Base64Decode("iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAIAAACQkWg2AAAABnRSTlMAAAAAAABupgeRAAAACXBIWXMA"))
do obj.Image.Write($system.Encryption.Base64Decode("AA7EAAAOxAGVKw4bAAAAGElEQVQokWNk+M9AEmAiTfmohlENQ0kDAD8vAR+xLJsiAAAAAElFTkSuQmCC"))

if obj.%JSONExportToString(.string) write string --> 

{"Name":"Joe","Image":"iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAIAAACQkWg2AAAABnRSTlMAA
AAAAABupgeRAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAGElEQVQokWNk+M9AEmAiTfmohlENQ0kDAD8vA
R+xLJsiAAAAAElFTkSuQmCC"}

As you see in the above example, the export works and does Base64Encoding for the appropriate property (in my example this is 16x16 pixel green rectangle). Please show the relevant part (property) of your class (definition).

0 0
Matjaz Murko · Jul 16, 2021

Hi.

Tnx for reply. 

Classes:

Class User.API [ Abstract ]
{

ClassMethod Load(Ident As %String) As %String [ Language = objectscript ]
{
Set data=##class(User.Data).%OpenId(Ident)
Quit:data="" "null"

Set model=##class(User.Model).%New()
Set model.Ident=data.Ident
Do model.PNG.CopyFrom(data.PNG)

Do model.%JSONExportToString(.json)

Quit json
}

ClassMethod Save(Ident As %String, Data As %String) As %Status [ Language = objectscript ]
{
Set data=##class(User.Data).%OpenId(Ident)
If (data="")
{
Set data=##class(User.Data).%New()
Set data.Ident=Ident
}

Set model=##class(User.Model).%New()
Do model.%JSONImport(Data)

Do data.PNG.CopyFrom(model.PNG)

Quit data.%Save()
}

Class User.Model Extends (%RegisteredObject, %JSON.Adaptor)
{

Property Ident As %String;

Property PNG As %Stream.TmpBinary;

}

Class User.Data Extends %Persistent
{

Property Ident As %String;

Property PNG As %Stream.FileBinary;

Index Ident On Ident [ IdKey, Unique ];

Storage Default
{
<Data name="DataDefaultData">
<Value name="1">
<Value>%%CLASSNAME</Value>
</Value>
<Value name="2">
<Value>Ident</Value>
</Value>
<Value name="3">
<Value>PNG</Value>
</Value>
</Data>
<DataLocation>^User.DataD</DataLocation>
<DefaultData>DataDefaultData</DefaultData>
<IdLocation>^User.DataD</IdLocation>
<IndexLocation>^User.DataI</IndexLocation>
<StreamLocation>^User.DataS</StreamLocation>
<Type>%Storage.Persistent</Type>
}

}

0 0
Julius Kavay · Jul 17, 2021

Zdravo Matjaž,

for the first glance, the above methods should work. Although I don't understand why you need the User.Model class? You can achieve the same thing by defining

Class User.Data Extends (%Persistend, %JSONAdapter) { }

and then 

ClassMethod Load(Ident As %String) As %String [ Language = objectscript ]
{
  Set data=##class(User.Data).%OpenId(Ident)
  Quit:data="" "null"

  Do data.%JSONExportToString(.json)
  Quit json
}

By the way, in the User.Data class, you can shorten the index definition to

Index Ident On Ident [ IdKey ];

because an IdKey is ALWAYS unique.

If you say, your C# gets a garbage PNG, then I would check two things:

1) does the PNG property (in User.Data) contain a valid PNG? Issue in a terminal session following commands: 

set data=##class(User.Data).%OpenId("...")
do data.PNG.Rewind() // not necessary straight after an open
zzdump data.PNG.Read(16)

the output should be:
0000: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52         .PNG........IHDR

2) are you sure, C# gets the JSON-string returned by Load() method? Are you sure, no intermediate process (a middleware) changes this string by applying extra encoding or decoding?

 

Pozdrav z Beča (Dunaja)

0 0
Matjaz Murko · Jul 17, 2021

Hi.

I use the model class when I want to export some extra data (e.g properties from referenced objects) - model class attached is just a demo class to reproduce behaviour. I've also tried to export on origin class as you've suggested but the result is the same. If I open saved stream on server it looks as a valid PNG file. Maybe you should test with larger PNG file to reproduce the error, because it works also fine for me with smaller PNGs.

About IdKey index it is my habit to add the Unique parameter, although it is unique by it self :)...

I use a workaround in Load method which creates a valid Base64 string on output:

Set data=##class(User.Data).%OpenId(Ident)
Quit:data="" "null"

Set model=##class(User.Model).%New()
Set model.Ident=data.Ident

Do model.%JSONExportToString(.json)

Set jsonBase64={}.%FromJSON(json)
Do jsonBase64.%Set("PNG",data.PNG,"stream>base64")

Quit jsonBase64.%ToJSON()

I did also some logging on strings created with original Load method and workaround and it shows many differences... And I did it on IRIS server to ensure no extra encoding/decoding is done.

Viele Grüsse nach Wien.

Matjaž

0 0
Julius Kavay · Jul 17, 2021

Hello Matjaž,

I have a suspicion...

I tested your case with a 16201607 bytes large PNG file (and it worked).

First, to be able to do this, I had to made a change in the User.API class:

//Do model.%JSONExportToString(.json)
//Quit json

Do model.%JSONExportToStream(.str)
Quit str

so you get back a STREAM instead of a string.

As for IRIS and Cache, a string can't have more then 3641144 chars!

And take into account, a base64 encoded string is 33% longer then the orginal (exact: newSize = oldSize  + 2 \ 3 * 4), so you can use stringvariables up to an original picture size of (roughly, not counting the padding(s)): 

3641144 - 19 - $l(identname)  \ 4 * 3  //  19 bytes for {Ident:"","PNG":""}

By the way, can you output the encoded size (i.e. the length) of the JSONString you send and then the size of the same string in C#? Are they the same?

0 0
Matjaz Murko · Jul 17, 2021

Hi.

I've tried already with method %JSONExportToStream but C# InterSystems.Data.IRISClient.ADO.IRIS native provider does not have method to return stream. And I checked the string length on server side and client (C#) side - they are the same. As I've already mentioned the workaround in method Load works fine, so the string length is not the problem.

Pls check your e-mail for attached PNG.

Regards, Matjaž.

0 0
Matjaz Murko · Jul 17, 2021

Bellow is modified Load method:

ClassMethod Load(Ident As %String) As %String [ Language = objectscript ]
{
Set data=##class(User.Data).%OpenId(Ident)
Quit:data="" "null" // Original
Set model=##class(User.Model).%New()
Set model.Ident=data.Ident
Do model.PNG.CopyFrom(data.PNG)
Do model.%JSONExportToString(.json) Set file=##class(%File).%New("C:\Tmp\1.txt") //Log the json string
Do file.Open("WSN")
Do file.Write(json)
Do file.Close() // Workaround
Set model=##class(User.Model).%New()
Set model.Ident=data.Ident
Do model.%JSONExportToString(.json) // CHANGE in workaround
Set a={}.%FromJSON(json)
Do a.%Set("PNG",data.PNG,"stream>base64")
// Set file=##class(%File).%New("C:\Tmp\2.txt") //Log the json string
Do file.Open("WSN")
Do file.Write(a.%ToJSON())
Do file.Close() Quit json
}

Compare files 1.txt and 2.txt, they are different, but they should be equal. With string recorded in 2.txt c# app doesn't throw the exception.

0 0