Written by

Ikara Ikliki

Question Ikara Ikliki · Jan 22, 2020

%GlobalBinaryStream to %Binary

#Caché #InterSystems IRIS for Health

When reading %GlobalBinaryStream into a %String I am aware therte is a size limit of 3,6MB.

My question is since %Binary is underlying a %String is there a limit on this as well?

Can I safely read any size %GlobalBinaryStream into a %Binary?

Discussion (3)0

Add reply

Comments

Vitaliy Serdtsev · Jan 22, 2020

Can I safely read any size %GlobalBinaryStream into a %Binary?

No. Any string (%Binary, %String, %VarString, etc.) has size limit of 3,6MB

PS: if it's not a secret, why do you need to read from a stream to a string?

0 0

Nigel Salm · Jan 22, 2020

The short answer is yes, you are constrained by the 3.6Mb string size limit and bear in mind that if you are exposing that property to SQL and ODBC you are further constrained by the limitations within ODBC itself.

I have been giving this whole concept a lot of thought recently. My context is slightly different as I am working in the context of FHIR and JSON and for example you can send in a patient's photo which in JSON has to be represented as a string of Base64Encoded Binary . In principal that photo could be of any size but when the JSON string is converted into a FHIR Patient Object the underlying datatype of the photo.data.value is ultimately derived from a class called %xsd.Base64Binary which is esentially a string and therefore has the size limitation. It doesn't really matter what my underlying persisted datatype is i.e. %GlobalBinaryStream because I am working with the IRIS FHIR Classes as intemediary object classes between my JSON and my underlying persisted classes I have to work with the constraint that the JSON Base64Binary content will fail if it's length exceeds the 3.6Mb (something I wish to address with ISC/WRC at some point).

The alternative of course, to persisting larger data content of something link an image, is to have a url that points to the image location and can be retrieved from the image server for presentation purposes.

So where I know that my images are not going to be that large I read in or write out the base64binary from/to my JSON message but where I know the image is larger such as an x-ray or histo image then that content lives on the appropriate image server as is retirieved when required for presentation purposes.

Nigel Salm

0 0

Vitaliy Serdtsev Jan 23, 2020 to Nigel Salm

Hi Nigel.

I hasten to inform you that new classes have been added to IRIS for writing/reading streams in JSON format: %Stream.DynamicBinary, %Stream.DynamicCharacter.

Here is a sample code:

Class dc.test Extends (%RegisteredObject, %JSON.Adaptor)
{
</FONT><FONT COLOR="#000080">Property </FONT><FONT COLOR="#000000">string </FONT><FONT COLOR="#000080">As %VarString</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#000080">%JSONFIELDNAME </FONT><FONT COLOR="#000000">= </FONT><FONT COLOR="#800080">"longstring"</FONT><FONT COLOR="#000000">);
</FONT><FONT COLOR="#000080">Property </FONT><FONT COLOR="#000000">cs </FONT><FONT COLOR="#000080">As %Stream.TmpCharacter</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#000080">%JSONFIELDNAME </FONT><FONT COLOR="#000000">= </FONT><FONT COLOR="#800080">"cstream"</FONT><FONT COLOR="#000000">);
</FONT><FONT COLOR="#000080">/// d ##class(dc.test).test()
ClassMethod </FONT><FONT COLOR="#000000">test()
{
  </FONT><FONT COLOR="#0000ff">#define </FONT><FONT COLOR="#000000">fill(%len,%val) </FONT><FONT COLOR="#0000ff">$tr($j(</FONT><FONT COLOR="#008000">""</FONT><FONT COLOR="#0000ff">,%len),</FONT><FONT COLOR="#008000">" "</FONT><FONT COLOR="#0000ff">,%val)
  try</FONT><FONT COLOR="#800080">{
    
    </FONT><FONT COLOR="#0000ff">s </FONT><FONT COLOR="#800000">tmp</FONT><FONT COLOR="#000000">=..</FONT><FONT COLOR="#0000ff">%New</FONT><FONT COLOR="#000000">()
    
    </FONT><FONT COLOR="#008000">; the maximum string length when exporting to JSON depends on the length of the property name
    </FONT><FONT COLOR="#0000ff">s </FONT><FONT COLOR="#800000">tmp</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">string</FONT><FONT COLOR="#000000">=</FONT><FONT COLOR="#0000ff">$$$fill</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#0000ff">$$$MaxLocalLength</FONT><FONT COLOR="#000000">-(</FONT><FONT COLOR="#0000ff">$l</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#008000">"longstring"</FONT><FONT COLOR="#000000">)+5),</FONT><FONT COLOR="#0000ff">$c</FONT><FONT COLOR="#000000">(351))
    
    </FONT><FONT COLOR="#0000ff">$$$ThrowOnError</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">tmp</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">cs</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">Write</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#0000ff">$$$fill</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#0000ff">$$$MaxLocalLength</FONT><FONT COLOR="#000000">,</FONT><FONT COLOR="#0000ff">$c</FONT><FONT COLOR="#000000">(355))))
    </FONT><FONT COLOR="#0000ff">$$$ThrowOnError</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">tmp</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">cs</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">Write</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#0000ff">$$$fill</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#0000ff">$$$MaxLocalLength</FONT><FONT COLOR="#000000">,</FONT><FONT COLOR="#0000ff">$c</FONT><FONT COLOR="#000000">(354))))
    </FONT><FONT COLOR="#0000ff">w </FONT><FONT COLOR="#008000">"[Export] len(string): "</FONT><FONT COLOR="#000000">,</FONT><FONT COLOR="#0000ff">$l</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">tmp</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">string</FONT><FONT COLOR="#000000">),</FONT><FONT COLOR="#008000">", len(cs): "</FONT><FONT COLOR="#000000">,</FONT><FONT COLOR="#800000">tmp</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">cs</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">Size</FONT><FONT COLOR="#000000">,!!
    </FONT><FONT COLOR="#0000ff">s </FONT><FONT COLOR="#800000">file</FONT><FONT COLOR="#000000">=</FONT><FONT COLOR="#000080">##class</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#008080">%Stream.FileCharacter</FONT><FONT COLOR="#000000">).</FONT><FONT COLOR="#0000ff">%New</FONT><FONT COLOR="#000000">()
    </FONT><FONT COLOR="#0000ff">s </FONT><FONT COLOR="#800000">file</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">Filename</FONT><FONT COLOR="#000000">=</FONT><FONT COLOR="#008000">"VeryBigObject.json"
    
    </FONT><FONT COLOR="#0000ff">$$$ThrowOnError</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">tmp</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">%JSONExportToStream</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">file</FONT><FONT COLOR="#000000">))
    </FONT><FONT COLOR="#0000ff">$$$ThrowOnError</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">file</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">%Save</FONT><FONT COLOR="#000000">())
    
    </FONT><FONT COLOR="#008000">;------------------
    
    </FONT><FONT COLOR="#0000ff">s </FONT><FONT COLOR="#800000">dobj</FONT><FONT COLOR="#000000">=</FONT><FONT COLOR="#ff00ff">{}</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">%FromJSON</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">file</FONT><FONT COLOR="#000000">)
    </FONT><FONT COLOR="#0000ff">w </FONT><FONT COLOR="#008000">"1. [Import    %DynamicObject] len(longstring): "</FONT><FONT COLOR="#000000">,</FONT><FONT COLOR="#0000ff">$l</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">dobj</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">longstring</FONT><FONT COLOR="#000000">),</FONT><FONT COLOR="#008000">", len(cstream): "</FONT><FONT COLOR="#000000">,</FONT><FONT COLOR="#800000">dobj</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">%Get</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#008000">"cstream"</FONT><FONT COLOR="#000000">,,</FONT><FONT COLOR="#008000">"stream"</FONT><FONT COLOR="#000000">).</FONT><FONT COLOR="#0000ff">Size</FONT><FONT COLOR="#000000">,!
    </FONT><FONT COLOR="#0000ff">s </FONT><FONT COLOR="#800000">obj1</FONT><FONT COLOR="#000000">=..</FONT><FONT COLOR="#0000ff">%JSONNew</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">dobj</FONT><FONT COLOR="#000000">)
    </FONT><FONT COLOR="#0000ff">w </FONT><FONT COLOR="#008000">"2. [Import %RegisteredObject] len(string): "</FONT><FONT COLOR="#000000">,</FONT><FONT COLOR="#0000ff">$l</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">obj1</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">string</FONT><FONT COLOR="#000000">),</FONT><FONT COLOR="#008000">", len(cs): "</FONT><FONT COLOR="#000000">,</FONT><FONT COLOR="#800000">obj1</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">cs</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">Size</FONT><FONT COLOR="#000000">,!
    </FONT><FONT COLOR="#008000">/*
    
    The following command will generate an error (tested on IRIS 2019.1.1 CE, Unicode):
    
    ERROR: <MAXSTRING>  zWrite^%Stream.TmpCharacter.1
    
    To avoid errors, you need to fix the code in the GenImportCharacterStream method (see the patch below)
    
    */
    
    </FONT><FONT COLOR="#0000ff">s </FONT><FONT COLOR="#800000">obj2</FONT><FONT COLOR="#000000">=..</FONT><FONT COLOR="#0000ff">%New</FONT><FONT COLOR="#000000">()
    </FONT><FONT COLOR="#0000ff">$$$ThrowOnError</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">obj2</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">%JSONImport</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">file</FONT><FONT COLOR="#000000">))
    </FONT><FONT COLOR="#0000ff">w </FONT><FONT COLOR="#008000">"3. [Import %RegisteredObject] len(string): "</FONT><FONT COLOR="#000000">,</FONT><FONT COLOR="#0000ff">$l</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">obj2</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">string</FONT><FONT COLOR="#000000">),</FONT><FONT COLOR="#008000">", len(cs): "</FONT><FONT COLOR="#000000">,</FONT><FONT COLOR="#800000">obj2</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">cs</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">Size</FONT><FONT COLOR="#000000">,!
  </FONT><FONT COLOR="#800080">}</FONT><FONT COLOR="#0000ff">catch</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">ex</FONT><FONT COLOR="#000000">) </FONT><FONT COLOR="#800080">{
    </FONT><FONT COLOR="#0000ff">w </FONT><FONT COLOR="#008000">"ERROR: "</FONT><FONT COLOR="#000000">,</FONT><FONT COLOR="#800000">ex</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">DisplayString</FONT><FONT COLOR="#000000">(),!
  </FONT><FONT COLOR="#800080">}
</FONT><FONT COLOR="#000000">}
</FONT><FONT COLOR="#000080">/// Get an instance of an JSON enabled class.<br><br>
/// 
/// You may override this method to do custom processing (such as initializing
/// the object instance) before returning an instance of this class.
/// However, this method should not be called directly from user code.<br>
/// Arguments:<br>
///     dynamicObject is the dynamic object with thee values to be assigned to the new object.<br>
///     containerOref is the containing object instance when called from JSONImport.
ClassMethod </FONT><FONT COLOR="#000000">%JSONNew(
  </FONT><FONT COLOR="#ff00ff">dynamicObject </FONT><FONT COLOR="#000080">As %DynamicObject</FONT><FONT COLOR="#000000">,
  </FONT><FONT COLOR="#ff00ff">containerOref </FONT><FONT COLOR="#000080">As %RegisteredObject </FONT><FONT COLOR="#000000">= </FONT><FONT COLOR="#800080">""</FONT><FONT COLOR="#000000">) </FONT><FONT COLOR="#000080">As %RegisteredObject </FONT><FONT COLOR="#000000">[ </FONT><FONT COLOR="#000080">GenerateAfter </FONT><FONT COLOR="#000000">= %JSONGenerate, </FONT><FONT COLOR="#000080">ServerOnly </FONT><FONT COLOR="#000000">= 1 ]
{
  </FONT><FONT COLOR="#0000ff">#dim </FONT><FONT COLOR="#800000">r </FONT><FONT COLOR="#0000ff">As </FONT><FONT COLOR="#008080">dc.test</FONT><FONT COLOR="#000000">=</FONT><FONT COLOR="#0000ff">$s</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#0000ff">$IsObject</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">containerOref</FONT><FONT COLOR="#000000">):</FONT><FONT COLOR="#800000">containerOref</FONT><FONT COLOR="#000000">,1:..</FONT><FONT COLOR="#0000ff">%New</FONT><FONT COLOR="#000000">())
  </FONT><FONT COLOR="#0000ff">s </FONT><FONT COLOR="#800000">r</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">string</FONT><FONT COLOR="#000000">=</FONT><FONT COLOR="#800000">dynamicObject</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">longstring
  d </FONT><FONT COLOR="#800000">r</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">cs</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">CopyFrom</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">dynamicObject</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">%Get</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#008000">"cstream"</FONT><FONT COLOR="#000000">,,</FONT><FONT COLOR="#008000">"stream"</FONT><FONT COLOR="#000000">))
  </FONT><FONT COLOR="#0000ff">q </FONT><FONT COLOR="#800000">r
</FONT><FONT COLOR="#000000">}
}</FONT>

Result:

USER><FONT COLOR="#0000ff">d </FONT><FONT COLOR="#000080">##class</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#008080">dc.test</FONT><FONT COLOR="#000000">).</FONT><FONT COLOR="#0000ff">test</FONT><FONT COLOR="#000000">()</FONT>
[Export] len(string): 3641129, len(cs): 7282288


[Import    %DynamicObject] len(longstring): 3641129, len(cstream): 7282288


[Import %RegisteredObject] len(string): 3641129, len(cs): 7282288


[Import %RegisteredObject] len(string): 3641129, len(cs): 7282288

Patch for 2019.1.1CE:

Instead of
 <s><FONT COLOR="#0000ff">$$$GENERATE</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">indent</FONT><FONT COLOR="#000000">_</FONT><FONT COLOR="#008000">&quot;&nbsp;&nbsp;Set&nbsp;sc=stream.Write(%JSONObject.&quot;</FONT><FONT COLOR="#000000">_</FONT><FONT COLOR="#0000ff">$$$QN</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#0000ff">$$$jsonfieldname</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">propertyMap</FONT><FONT COLOR="#000000">))_</FONT><FONT COLOR="#008000">&quot;)&nbsp;If&nbsp;$$$ISERR(sc)&nbsp;Goto&nbsp;%JSONImportExit&quot;</FONT><FONT COLOR="#000000">)</FONT></s>



need
	<b><FONT COLOR="#0000ff">$$$GENERATE</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">indent</FONT><FONT COLOR="#000000">_</FONT><FONT COLOR="#008000">&quot;&nbsp;&nbsp;Set&nbsp;testInvalidField=0,&nbsp;sc=stream.CopyFrom(%JSONObject.%Get(field,,&quot;&quot;stream&quot;&quot;))&nbsp;If&nbsp;$$$ISERR(sc)&nbsp;Goto&nbsp;%JSONImportExit&quot;</FONT><FONT COLOR="#000000">)</FONT></b></pre>

It is possible that in version 2019.4.x has already fixed everything.

0 0