Large Stream to base64Encoded

Question

Question

I have below method , which receives a PDF ; this method throws error for a PDF file of 3MB size

<MAXSTRING>zBase64Encode+9 -- logged as '-' number - @' set encodedData = $system.Encryption.Base64Encode(content)'

How do I fix this.

----------------

ClassMethod Base64Encode(pStream As %Stream) As %Stream
{
set tSC = $$$OK
set tSC = pStream.Rewind()
s pEncoded=##class(%Stream.GlobalCharacter).%New()
while 'pStream.AtEnd {
Set tLen = pStream.Size
set content = pStream.Read(.tLen)
set encodedData = $system.Encryption.Base64Encode(content)
   set encodedData=$translate(encodedData, $c(13,10))
s tSC=pEncoded.Write(encodedData)
q:$$$ISERR(tSC)
}
//q:$$$ISERR(tSC)
do pEncoded.Rewind()
Quit pEncoded
}

Discussion (2)1

Log in or sign up to continue

Julius Kavay · Sep 23

See the example class below

Class DC.Encoding Extends %RegisteredObject
{

/// Take an raw stream (i.e. unencoded) and
/// output a new, Base64 encoded stream.
/// 
ClassMethod ToBase64(str)
{
	// Base64 encoding means:
	// you take 3*N characters from the source
	// and put  4*N characters into the destination.
	// If the size of the source is not a multiple of 3 then
	// the last one or two bytes will be padded.
	// 
	// If you take an N such that 4*N less or equal 32767
	// (the max size of a short string) then Cache or IRIS
	// can work with short strings, which perform (usually)
	// better than long strings
	// 
	// N is integer.
	// 
	// A good value for N is 8190,
	// so you read 24570 bytes from the source and write 32760 to the destination
	// 
	// Of course, you can take whatever number up to  910286
	// (3 * 910286 = 2730858,  4 * 910286 = 3641144)
	// 
	set len=8190*3
	set flg=1 // this flag instructs $system.Encryption.Base64Encode
			// not to insert linebreaks at every 76 characters
	set new=##class(%Stream.GlobalCharacter).%New()
	do str.Rewind()
	while 'str.AtEnd {
		do new.Write($system.Encryption.Base64Encode(str.Read(len),flg))
	}
	quit new
}

/// Take a Base64 encoded stream
/// and decode it to a new stream
/// 
/// The method itself has no information about the decoded data
/// hence it assumens binary data, but you, the caller (hopefully)
/// knows more about your data and can provide the correct stream
/// type for the decoder.
/// For exaple a character stream instead of binary.
ClassMethod FromBase64(str, new = 0)
{
	// Base64 decoding means:
	// you take 4*N characters from the source
	// and put  3*N characters into the destination
	// 
	// If you take an N such that 4*N less or equal 32767
	// (the max size of a short string) then Cache or IRIS
	// can work with short strings, which perform (usually)
	// better than long strings
	// 
	// N is integer.
	// 
	// A good value for N is 8190,
	// so you read 24570 bytes from the source and write 32760 to the destination
	// 
	// Of course, you can take whatever number up to  910286
	// (3 * 910286 = 2730858,  4 * 910286 = 3641144)
	// 
	
	set len=8190*4
	set:'new new=##class(%Stream.GlobalBinary).%New()
	do str.Rewind()
	while 'str.AtEnd {
		do new.Write($system.Encryption.Base64Decode(str.Read(len)))
	}
	quit new
}

ClassMethod Test(file)
{
	set str=##class(%Stream.FileBinary).%New()
	do str.LinkToFile(file)
	write str.Size,!
	
	set enc=..ToBase64(str)
	write enc.Size,!
	
	set dec=..FromBase64(enc)
	write dec.Size,!
}

}

1 0

score 0 · Answer 1 · 2024-09-23T14:08:47-04:00

There is a limit to string length, as explained in the documentation.

To convert a binary stream to a base64 encoded character stream, you have to call the function in a loop using a buffer. Due the way the function works, the buffer length must be a multiple of 57.

Here a sample implementation, from ks.lib.stream.Utils class in the ks-iris-lib package available on Open Exchange.

/// encode stream data into encoded using bufferSize (due the implementation of $system.Encryption.Base64Encode, this must be a multiple of 57)
ClassMethod Base64Encode(stream As %Stream.Object, Output encoded As %Stream.Object, bufferSize As %Integer = 5700) As %Status
{
  #Dim sc as %Status
  #Dim ex as %Exception.AbstractException

  #Dim len As %Integer
  
  s sc = $$$OK
  try {
    throw:((bufferSize#57)'=0) ##class(%Exception.General).%New("buffer size must be a multiple of 57")
    $$$TOE(sc,stream.Rewind())
    s:'$d(encoded) encoded = ##class(%Stream.TmpCharacter).%New()
    s len=bufferSize
    while 'stream.AtEnd {
      $$$TOE(sc,encoded.Write($system.Encryption.Base64Encode(stream.Read(.len),1)))
      s len = bufferSize
    }
  }
  catch (ex) {
    s sc = ex.AsStatus()
  }
  return sc
}