New post
Question
· Mar 30

Deflate compression

#ObjectScript #InterSystems IRIS

During the realization of some required functionality, I discovered that I need to use Deflate compression, and found, that IRIS does not offer any options, $system.Util.Compress, works one way open (GZIP/COMPRESS) totally different, even for the same algorithms, I see that what I would expect as a result for Deflate somewhere there, but Compress returns something more. Is there any way to get a correct result using just ObjectScript without a need to use external tools? Was it really so difficult to have Deflate implemented in the first place?

  set original = "my very long string which needs to be deflated"
  zwrite original

  write !,"using $system.Util.Compress",!
  set compressed = $system.Util.Compress(original, "zlib")
  zwrite compressed
  zzdump compressed write !
  set deflated = $system.Encryption.Base64Encode(compressed, 1)
  zwrite deflated
  
  set tmpFile = ##class(%File).TempFilename("bin")
  
  set io = $io
  open tmpFile:("WNS":::/COMPRESS="zlib":/GZIP=0):0
  use tmpFile
  write original
  use io
  close tmpFile

  set stream = ##class(%Stream.FileBinary).%New()
  set stream.Filename = tmpFile
  set stream.RemoveOnClose = 1
  set compressed = stream.Read(stream.Size)  
  
  write !,"using open file",!
  zwrite compressed
  zzdump compressed write !
  set deflated = $system.Encryption.Base64Encode(compressed, 1)
  zwrite deflated

  write !,"Expected",!
  set base64 = "y61UKEstqlTIyc9LVyguKcoEUuUZmckZCnmpqSnFCiX5CkmpCimpaTmJJakpAA=="
  set compressed = $system.Encryption.Base64Decode(base64)
  zwrite compressed
  zzdump compressed write !

Why the results are different, and how can I make it understandable by not only IRIS?

What is the point of having a compress function, which says that it uses some standards but only IRIS can decompress?

But results I got from IRIS

ZLIB using OPEN command recognized as GZIP, not a Deflate anyway

Product version: IRIS 2023.3
$ZV: IRIS for UNIX (Ubuntu Server LTS for ARM64 Containers) 2023.3 (Build 254U) Wed Nov 8 2023 13:04:07 EST
Discussion (7)2
Log in or sign up to continue
Enrico Parisi · Mar 31

I've been there before on IRIS zlib compression library....

The problem is explained in Compress(), from Class Reference %SYSTEM.Util, Compress() method (emphasis mine):

Returns the compressed form of a given string.
When Type is specified, it is the compress algorithm used to compress the data. Its value can be "zlib", "zstd" or "lz4". If Type is not specified then the "zlib" compression algorithm is used. A compressed string is a compressed data block with a customized wrapper. Use Decompress() to decompress it.

So, after Compress() you need to remove the "customized wrapper":

set compressed=$e(compressed,4,*-5)

Unfortunately Decompress() needs the custom wrapper, and I don't know how to make it up, however, as in your sample, I/O can be used, using file in your sample.

Regarding the file I/O, note that form Open command documentation:

"zlib" — Use the zlib compression library. /COMPRESS="zlib" is equivalent to /GZIP=1

So, if you change:

open tmpFile:("WNS":::/COMPRESS="zlib":/GZIP=0):0

With:

open tmpFile:("WNS":::/GZIP=0):0

It will then work just fine.

I fully agree that it's a pity to have zilib compress/uncompress utility "custimized" with proprietary wrapper that makes it unusable for any "standard use". However in that case you can use I/O, file or other devices. Personally I use XDEV for compress/decompress and leave the silly %SYSTEM.Util.Compress()/Decompress() alone.

0 0
Enrico Parisi · Apr 1

For DEFLATE use:

open tmpFile:("WNS":::/COMPRESS="DEFLATE"):0

	set original = "my very long string which needs to be deflated"
	zwrite original

	write !,"using $system.Util.Compress",!
	set compressed = $system.Util.Compress(original, "zlib")
	set compressed=$e(compressed,4,*-5)
	zwrite compressed
	zzdump compressed write !
	set deflated = $system.Encryption.Base64Encode(compressed, 1)
	zwrite deflated

	set tmpFile = ##class(%File).TempFilename("bin")

	set io = $io
	open tmpFile:("WNS":::/COMPRESS="DEFLATE"):0
	use tmpFile
	write original
	use io
	close tmpFile

	set stream = ##class(%Stream.FileBinary).%New()
	set stream.Filename = tmpFile
	set stream.RemoveOnClose = 1
	set compressed = stream.Read(stream.Size)  

	write !,"using open file",!
	zwrite compressed
	zzdump compressed write !
	set deflated = $system.Encryption.Base64Encode(compressed, 1)
	zwrite deflated

	write !,"Expected",!
	set base64 = "y61UKEstqlTIyc9LVyguKcoEUuUZmckZCnmpqSnFCiX5CkmpCimpaTmJJakpAA=="
	set compressed = $system.Encryption.Base64Decode(base64)
	zwrite compressed
	zzdump compressed write !

Result:

original="my very long string which needs to be deflated"
 
using $system.Util.Compress
compressed="Ë­T(K-ªTÈÉÏKW(.)Ê"_$c(4)_"Rå"_$c(25,153)_"É"_$c(25,10)_"y©©)Å"_$c(10)_"%ù"_$c(10)_"I©"_$c(10)_")©i9"_$c(137)_"%©)"_$c(0)
 
0000: CB AD 54 28 4B 2D AA 54 C8 C9 CF 4B 57 28 2E 29         Ë­T(K-ªTÈÉÏKW(.)
0010: CA 04 52 E5 19 99 C9 19 0A 79 A9 A9 29 C5 0A 25         Ê.Rå..É..y©©)Å.%
0020: F9 0A 49 A9 0A 29 A9 69 39 89 25 A9 29 00               ù.I©.)©i9.%©).
deflated="y61UKEstqlTIyc9LVyguKcoEUuUZmckZCnmpqSnFCiX5CkmpCimpaTmJJakpAA=="
 
using open file
compressed="Ë­T(K-ªTÈÉÏKW(.)Ê"_$c(4)_"Rå"_$c(25,153)_"É"_$c(25,10)_"y©©)Å"_$c(10)_"%ù"_$c(10)_"I©"_$c(10)_")©i9"_$c(137)_"%©)"_$c(0)
 
0000: CB AD 54 28 4B 2D AA 54 C8 C9 CF 4B 57 28 2E 29         Ë­T(K-ªTÈÉÏKW(.)
0010: CA 04 52 E5 19 99 C9 19 0A 79 A9 A9 29 C5 0A 25         Ê.Rå..É..y©©)Å.%
0020: F9 0A 49 A9 0A 29 A9 69 39 89 25 A9 29 00               ù.I©.)©i9.%©).
deflated="y61UKEstqlTIyc9LVyguKcoEUuUZmckZCnmpqSnFCiX5CkmpCimpaTmJJakpAA=="
 
Expected
compressed="Ë­T(K-ªTÈÉÏKW(.)Ê"_$c(4)_"Rå"_$c(25,153)_"É"_$c(25,10)_"y©©)Å"_$c(10)_"%ù"_$c(10)_"I©"_$c(10)_")©i9"_$c(137)_"%©)"_$c(0)
 
0000: CB AD 54 28 4B 2D AA 54 C8 C9 CF 4B 57 28 2E 29         Ë­T(K-ªTÈÉÏKW(.)
0010: CA 04 52 E5 19 99 C9 19 0A 79 A9 A9 29 C5 0A 25         Ê.Rå..É..y©©)Å.%
0020: F9 0A 49 A9 0A 29 A9 69 39 89 25 A9 29 00               ù.I©.)©i9.%©).
2 0