Stream Compression Not Applied to Some Text

Question

Question

Ashok Kumar · Nov 6, 2024

#InterSystems IRIS #InterSystems IRIS for Health

Hello Community,

I have a stream property which uses %Stream.GlobalCharacter and I set some dummy texts like "a1a2a3a4a5a6a7...."(more than 1024 characters) in to that stream and the stream values are not compressed. However, The streams are compressed If I use the Lorem ipsum is a dummy or placeholder text or other text "This is part four of a long memo." which length is greater than 1024

I opened the stream object and confirmed the compress property is 0 for first entry and 1 for 2nd object

Class Samples.Strms Extends %Persistent
{
Property MyId As %String;
Property MyBioStrm As %Stream.GlobalCharacter;
}

why the automatic stream Compression not applied for the first text

Thanks!

Product version: IRIS 2024.1

$ZV: IRIS for Windows (x86-64) 2024.1.1 (Build 347U) Thu Jul 18 2024 17:40:10 EDT

Discussion (3)2

Log in or sign up to continue

Enrico Parisi · Nov 6, 2024

Evidently "The first 4KB of the stream cannot be compressed by at least 20 percent."

Demonstration:

USER>set data="" for i=1:1:400 set data=data_"a"_i
 
USER>set compressed=##class(%SYSTEM.Util).Compress(data,"lz4")
 
USER>write $length(data)," - ",$length(compressed),!
1492 - 1481

P.S.: IRIS use lz4 compression for streams

1 0

score 0 · Answer 1 · 2024-11-06T09:43:52-05:00

From Stream Compression documentation:

.....new stream data is automatically compressed, except for the following cases, where the data is not suitable for compression:

The stream can be stored in a single chunk less than 1024 characters long.

The stream is already compressed, that is, the first chunk of data matches a typical compressed file format, such as JPEG, MP3, or ZIP.

The first 4KB of the stream cannot be compressed by at least 20 percent.

I guess your test runs into these cases.

score 0 · Answer 2 · 2024-11-06T12:48:52-05:00

Hello @Enrico Parisi

I just a generated text by executing this set data="" for i=1:1:400 set data=data_"a"_i and it's 1492 characters long and it's not other JPEG or other file formats. But this text is not compressed. However, If I use random text/place holder text more than 1500 characters and that text was compressed automatically.