· Jan 10, 2020

How to force binary hash algorithm in Ensemble production?

TL;DR: If I set an Ensemble  Production Service based on EnsLib.File.PassthroughService to a Binary charset encoding, it breaks the file handling. Any ideas?

Full long post:


I set up an Ensemble production to transfer files via SFTP, which works fine sending the files to my Linux server. Then I was informed that we needed to save a hash of the file, for comparison with the destination to verify the file arrived unmodified. I looked at the base64 encoded hash on my linux server:


and I created a class to extend the StreamContainer to add a hash value:

Class SFTP.HashFile Extends Ens.StreamContainer [ Inheritance = right, ProcedureBlock ]
{ Property Hash As %String(MAXLEN = 256); }

which seems to work fine. But when the production runs, the hash is listed as:


I narrowed that down to Ensemble automatically choosing the stream type - in this case, it sees it as a FC type stream - File Character - but Linux sees it as a Binary encoding. I verified the issue with this bit of code:

 New bstream,cstream,sc,bhash,chash,zfile
 Set zfile="d:\Patch\Test_File_1.CSV"
 Set bstream=##class(%Stream.FileBinary).%New()
 Set sc=bstream.LinkToFile(zfile)
 Do bstream.Rewind()
 Set bhash=$SYSTEM.Encryption.SHAHashStream(256,bstream)
 "Binary: "_$SYSTEM.Encryption.Base64Encode(bhash),!
 Do bstream.%Close()
 Set cstream=##class(%Stream.FileCharacter).%New()
 Set sc=cstream.LinkToFile(zfile)
 Do cstream.Rewind()
 Set chash=$SYSTEM.Encryption.SHAHashStream(256,cstream)
 "Char: "_$SYSTEM.Encryption.Base64Encode(chash),!!
 Do cstream.%Close()

And when run, here's the output:

Binary: G7QWAP6FcLInFWP8ECRL/EI2NfKdaf6TtrpwYuvSOEc=
Char:   Vn0ygZZN8DewD8KfKKYF8BTV9K1LsuFBEXnAkqXBEFs=

I found a setting in the EnsLib.File.PassthroughService  to manually set the charset to Binary, and when I set that, the production will give me the correct hash and sends the file - but then recreates the file with a randomized OriginalFilename, waits 5 seconds (or whatever's set in the Retry Interval parameter), and sends it again... and again... and again.

When I look at the message viewer, the <Stream></Stream>  is empty:

 <?xml version="1.0" ?>
<!-- type: SFTP.HashFile  id: 281 -->
<AttributesItem AttributesKey="1" xsi:nil="true"></AttributesItem>

Even though file is sent intact successfully.  (You can see an example of the 'random' filename above.)

Any ideas that would be easier than rewriting  a custom EnsLib.File.PassthroughService and/or Ens.StreamContainer to only handle binary encoded streams?

Thanks for any and all input!

Discussion (8)2
Log in or sign up to continue

Try change TranslateTable:

Class dc.test Abstract ]

ClassMethod runtests()
  ;d ##class(dc.test).runtests()

  "data: ",?15,data,!



  cstream.TranslateTable="" ;or RAW, SAME


data:          тест
Binary:        409t7BLE9FmeugePMa6BOUINIbG9LXztfSKwnCB0+0g=
Char:          2eEII+27ZRfvbZvK4XNsx7WPDb+82DymPPOAdJ0p1SQ=
Char->Binary:  409t7BLE9FmeugePMa6BOUINIbG9LXztfSKwnCB0+0g=


Thanks for the code, but how would I integrate changing the the TranslateTable in the DTL of my Production Process? That's where the hash is getting created; I tried several permutations of ' set source.Stream.TranslateTable="" '  but all gave me a <PROPERTY DOES NOT EXIST> error. The CharEncodingTable is a Readonly Internal value, and trying to changing that gives me a <CANNOT SET THIS PROPERTY> error.

Although, looking at the source code of the  Ens.StreamContainer class, that might give me an idea how to rewrite the class _easily_ to accomplish what I need...

Thanks for the input!

Yes, recreates in the same folder, as the production then re-sends the file (with the randomized filename) via SFTP to the remote server.

If it makes a difference, the production is on a Windows 10 system, although I have access to a Linux server for testing as well.


File Path: C:\Export\

Archive Path: C:\Export\CSV_Complete

[[ I have two different business services looking for two different extensions)

Work Path: [[ Null ]]

I don't see a 'DeleteFromServer' setting... will continue digging.

Also, Conform Complete: Readable (default)

File Access Timeout: 2

Hope this helps, and thanks!

You can solve the issue "head-on", namely, to re-convert character stream using %IO.StringStream:

Class dc.test Abstract ]

ClassMethod test()

  "dump char:" zzdump char
  !!,"dump binary:" zzdump bin

  stream.CharEncoding="UTF8" ; here should be the encoding of your CSV file
  stream.CopyFrom(cs; or d stream.CopyFrom(source.Stream)


  Attention! The following code is needed to work around an error in method the SHAHashStream,
  since it expects "Rewind() As %Status", but class the %IO.StringStream uses "Rewind(Output pSC As %Status)"


  "Char (stream):",?21,$system.Encryption.Base64Encode($system.Encryption.SHAHashStream(256,cs)),!,
  "Binary (stream):",?21,$system.Encryption.Base64Encode($system.Encryption.SHAHashStream(256,bs)),!!



dump char:
0000: 0442 0435 0441 0442                                     тест
dump binary:
0000: D1 82 D0 B5 D1 81 D1 82                                 Ñ.еÑ.Ñ.
Char:                2eEII+27ZRfvbZvK4XNsx7WPDb+82DymPPOAdJ0p1SQ=
Binary:              409t7BLE9FmeugePMa6BOUINIbG9LXztfSKwnCB0+0g=
Char (stream):       2eEII+27ZRfvbZvK4XNsx7WPDb+82DymPPOAdJ0p1SQ=
Binary (stream):     409t7BLE9FmeugePMa6BOUINIbG9LXztfSKwnCB0+0g=