Written by

Independent Consultant

Question Stella Ticker · Mar 13, 2017

Fastest way to strip invalid characters from stream

A method to convert certain non-readable ASCII characters in a %Stream.FileCharacter object first copies it into a %Stream.FileBinary object and then loops through each character one at a time to find and convert these offensive characters to our interpretation of their readable ASCII equivalents . The loop is sequential (while 'bStream.AtEnd) and is taking too long for large files. Any suggestions for an existing Cache method to do this or is there another way, besides a sequential loop, to examine every character of a binary Stream?

Discussion (2)1

Add reply

Comments

Mark Hanson · Mar 13, 2017

Can you provide the code you are using currently so we have something definitive to base comments off, but have you tried using $translate and reading the data in big chunks e.g. 16k at a time?

While 'binarystream.AtEnd {

  Set sc=outputstream.Write($translate(binarystream.Read(16000),badchars,goodchars)

}

Where binarystream is your binary input stream, outputstream is your output stream with the converted characters and badchars is a list of the bad characters you need to convert and goodchars is the list of the values you want the badchars converted into.

0 0

Martin Fukátko Mar 14, 2017 to Mark Hanson

Or you can use $zconvert instead of $translate. It is more useable in some cases.

0 0