Roger Merchberger · Jan 10, 2020

How to force binary hash algorithm in Ensemble production?

TL;DR: If I set an Ensemble  Production Service based on EnsLib.File.PassthroughService to a Binary charset encoding, it breaks the file handling. Any ideas?

Full long post:


I set up an Ensemble production to transfer files via SFTP, which works fine sending the files to my Linux server. Then I was informed that we needed to save a hash of the file, for comparison with the destination to verify the file arrived unmodified. I looked at the base64 encoded hash on my linux server:


and I created a class to extend the StreamContainer to add a hash value:

Class SFTP.HashFile Extends Ens.StreamContainer [ Inheritance = right, ProcedureBlock ]
{ Property Hash As %String(MAXLEN = 256); }

which seems to work fine. But when the production runs, the hash is listed as:


I narrowed that down to Ensemble automatically choosing the stream type - in this case, it sees it as a FC type stream - File Character - but Linux sees it as a Binary encoding. I verified the issue with this bit of code:

 New bstream,cstream,sc,bhash,chash,zfile
 Set zfile="d:\Patch\Test_File_1.CSV"
 Set bstream=##class(%Stream.FileBinary).%New()
 Set sc=bstream.LinkToFile(zfile)
 Do bstream.Rewind()
 Set bhash=$SYSTEM.Encryption.SHAHashStream(256,bstream)
 "Binary: "_$SYSTEM.Encryption.Base64Encode(bhash),!
 Do bstream.%Close()
 Set cstream=##class(%Stream.FileCharacter).%New()
 Set sc=cstream.LinkToFile(zfile)
 Do cstream.Rewind()
 Set chash=$SYSTEM.Encryption.SHAHashStream(256,cstream)
 "Char: "_$SYSTEM.Encryption.Base64Encode(chash),!!
 Do cstream.%Close()

And when run, here's the output:

Binary: G7QWAP6FcLInFWP8ECRL/EI2NfKdaf6TtrpwYuvSOEc=
Char:   Vn0ygZZN8DewD8KfKKYF8BTV9K1LsuFBEXnAkqXBEFs=

I found a setting in the EnsLib.File.PassthroughService  to manually set the charset to Binary, and when I set that, the production will give me the correct hash and sends the file - but then recreates the file with a randomized OriginalFilename, waits 5 seconds (or whatever's set in the Retry Interval parameter), and sends it again... and again... and again.

When I look at the message viewer, the <Stream></Stream>  is empty:

<?xml version="1.0" ?>
<!-- type: SFTP.HashFile  id: 281 -->
<AttributesItem AttributesKey="1" xsi:nil="true"></AttributesItem>

Even though file is sent intact successfully.  (You can see an example of the 'random' filename above.)

Any ideas that would be easier than rewriting  a custom EnsLib.File.PassthroughService and/or Ens.StreamContainer to only handle binary encoded streams?

Thanks for any and all input!

0 278
Discussion (8)3
Log in or sign up to continue

Try change TranslateTable:

Class dc.test Abstract ]

ClassMethod runtests()
  ;d ##class(dc.test).runtests()

  "data: ",?15,data,!



  cstream.TranslateTable="" ;or RAW, SAME


data:          тест
Binary:        409t7BLE9FmeugePMa6BOUINIbG9LXztfSKwnCB0+0g=
Char:          2eEII+27ZRfvbZvK4XNsx7WPDb+82DymPPOAdJ0p1SQ=
Char->Binary:  409t7BLE9FmeugePMa6BOUINIbG9LXztfSKwnCB0+0g=


Thanks for the code, but how would I integrate changing the the TranslateTable in the DTL of my Production Process? That's where the hash is getting created; I tried several permutations of ' set source.Stream.TranslateTable="" '  but all gave me a <PROPERTY DOES NOT EXIST> error. The CharEncodingTable is a Readonly Internal value, and trying to changing that gives me a <CANNOT SET THIS PROPERTY> error.

Although, looking at the source code of the Ens.StreamContainer class, that might give me an idea how to rewrite the class _easily_ to accomplish what I need...

Thanks for the input!

Yes, recreates in the same folder, as the production then re-sends the file (with the randomized filename) via SFTP to the remote server.

If it makes a difference, the production is on a Windows 10 system, although I have access to a Linux server for testing as well.


File Path: C:\Export\

Archive Path: C:\Export\CSV_Complete

[[ I have two different business services looking for two different extensions)

Work Path: [[ Null ]]

I don't see a 'DeleteFromServer' setting... will continue digging.

Also, Conform Complete: Readable (default)

File Access Timeout: 2

Hope this helps, and thanks!

You can solve the issue "head-on", namely, to re-convert character stream using %IO.StringStream:

Class dc.test Abstract ]

ClassMethod test()

  "dump char:" zzdump char
  !!,"dump binary:" zzdump bin

  stream.CharEncoding="UTF8" ; here should be the encoding of your CSV file
  stream.CopyFrom(cs; or d stream.CopyFrom(source.Stream)


  Attention! The following code is needed to work around an error in method the SHAHashStream,
  since it expects "Rewind() As %Status", but class the %IO.StringStream uses "Rewind(Output pSC As %Status)"


  "Char (stream):",?21,$system.Encryption.Base64Encode($system.Encryption.SHAHashStream(256,cs)),!,
  "Binary (stream):",?21,$system.Encryption.Base64Encode($system.Encryption.SHAHashStream(256,bs)),!!



dump char:
0000: 0442 0435 0441 0442                                     тест
dump binary:
0000: D1 82 D0 B5 D1 81 D1 82                                 Ñ.еÑ.Ñ.
Char:                2eEII+27ZRfvbZvK4XNsx7WPDb+82DymPPOAdJ0p1SQ=
Binary:              409t7BLE9FmeugePMa6BOUINIbG9LXztfSKwnCB0+0g=
Char (stream):       2eEII+27ZRfvbZvK4XNsx7WPDb+82DymPPOAdJ0p1SQ=
Binary (stream):     409t7BLE9FmeugePMa6BOUINIbG9LXztfSKwnCB0+0g=

SubdirectoryLevels: 0

I honestly didn't think that it would make a difference, as when Charset = Native, it works fine. Only when Charset = Binary is when it acts odd.

That said, I changed ArchivePath to c:\Export2\CSV_Complete\ -- restarted the production and tested -- no change in behaviour.


I'm out of ideas. There should not be any changes between file and character streams besides encoding.

I think you need to share a minimal sample that reproduces this error or contact the WRC.

What's your SubdirectoryLevels setting value?

Try to move Archive Path outside of File Path.

recreates the file with a randomized OriginalFilename

Recreates in the same folder?

What is the value of FilePath, WorkPath, ArchivePath, DeleteFromServer settings?

You need to solve the problem with file recreation, as specifying "binary" as a Charset setting gives you the correct hash.