For your response I assume you are searching for a class used by a Business Host component within a production.

Using IRIS you can search for that, and much more, using "Interface Reference" within the Management portal:

Unfortunately you are using an old version where this feature is not available.

Add this to the (possibly long) list of good reasons to move to IRIS.

For DEFLATE use:

open tmpFile:("WNS":::/COMPRESS="DEFLATE"):0

	set original = "my very long string which needs to be deflated"
	zwrite original

	write !,"using $system.Util.Compress",!
	set compressed = $system.Util.Compress(original, "zlib")
	set compressed=$e(compressed,4,*-5)
	zwrite compressed
	zzdump compressed write !
	set deflated = $system.Encryption.Base64Encode(compressed, 1)
	zwrite deflated

	set tmpFile = ##class(%File).TempFilename("bin")

	set io = $io
	open tmpFile:("WNS":::/COMPRESS="DEFLATE"):0
	use tmpFile
	write original
	use io
	close tmpFile

	set stream = ##class(%Stream.FileBinary).%New()
	set stream.Filename = tmpFile
	set stream.RemoveOnClose = 1
	set compressed = stream.Read(stream.Size)  

	write !,"using open file",!
	zwrite compressed
	zzdump compressed write !
	set deflated = $system.Encryption.Base64Encode(compressed, 1)
	zwrite deflated

	write !,"Expected",!
	set base64 = "y61UKEstqlTIyc9LVyguKcoEUuUZmckZCnmpqSnFCiX5CkmpCimpaTmJJakpAA=="
	set compressed = $system.Encryption.Base64Decode(base64)
	zwrite compressed
	zzdump compressed write !

Result:

original="my very long string which needs to be deflated"
 
using $system.Util.Compress
compressed="Ë­T(K-ªTÈÉÏKW(.)Ê"_$c(4)_"Rå"_$c(25,153)_"É"_$c(25,10)_"y©©)Å"_$c(10)_"%ù"_$c(10)_"I©"_$c(10)_")©i9"_$c(137)_"%©)"_$c(0)
 
0000: CB AD 54 28 4B 2D AA 54 C8 C9 CF 4B 57 28 2E 29         Ë­T(K-ªTÈÉÏKW(.)
0010: CA 04 52 E5 19 99 C9 19 0A 79 A9 A9 29 C5 0A 25         Ê.Rå..É..y©©)Å.%
0020: F9 0A 49 A9 0A 29 A9 69 39 89 25 A9 29 00               ù.I©.)©i9.%©).
deflated="y61UKEstqlTIyc9LVyguKcoEUuUZmckZCnmpqSnFCiX5CkmpCimpaTmJJakpAA=="
 
using open file
compressed="Ë­T(K-ªTÈÉÏKW(.)Ê"_$c(4)_"Rå"_$c(25,153)_"É"_$c(25,10)_"y©©)Å"_$c(10)_"%ù"_$c(10)_"I©"_$c(10)_")©i9"_$c(137)_"%©)"_$c(0)
 
0000: CB AD 54 28 4B 2D AA 54 C8 C9 CF 4B 57 28 2E 29         Ë­T(K-ªTÈÉÏKW(.)
0010: CA 04 52 E5 19 99 C9 19 0A 79 A9 A9 29 C5 0A 25         Ê.Rå..É..y©©)Å.%
0020: F9 0A 49 A9 0A 29 A9 69 39 89 25 A9 29 00               ù.I©.)©i9.%©).
deflated="y61UKEstqlTIyc9LVyguKcoEUuUZmckZCnmpqSnFCiX5CkmpCimpaTmJJakpAA=="
 
Expected
compressed="Ë­T(K-ªTÈÉÏKW(.)Ê"_$c(4)_"Rå"_$c(25,153)_"É"_$c(25,10)_"y©©)Å"_$c(10)_"%ù"_$c(10)_"I©"_$c(10)_")©i9"_$c(137)_"%©)"_$c(0)
 
0000: CB AD 54 28 4B 2D AA 54 C8 C9 CF 4B 57 28 2E 29         Ë­T(K-ªTÈÉÏKW(.)
0010: CA 04 52 E5 19 99 C9 19 0A 79 A9 A9 29 C5 0A 25         Ê.Rå..É..y©©)Å.%
0020: F9 0A 49 A9 0A 29 A9 69 39 89 25 A9 29 00               ù.I©.)©i9.%©).
 

I've been there before on IRIS zlib compression library....

The problem is explained in Compress(), from Class Reference %SYSTEM.Util, Compress() method (emphasis mine):

Returns the compressed form of a given string.
When Type is specified, it is the compress algorithm used to compress the data. Its value can be "zlib", "zstd" or "lz4". If Type is not specified then the "zlib" compression algorithm is used. A compressed string is a compressed data block with a customized wrapper. Use Decompress() to decompress it.

So, after Compress() you need to remove the "customized wrapper":

set compressed=$e(compressed,4,*-5)

Unfortunately Decompress() needs the custom wrapper, and I don't know how to make it up, however, as in your sample, I/O can be used, using file in your sample.

Regarding the file I/O, note that form Open command documentation:

"zlib" — Use the zlib compression library. /COMPRESS="zlib" is equivalent to /GZIP=1

So, if you change:

open tmpFile:("WNS":::/COMPRESS="zlib":/GZIP=0):0

With:

open tmpFile:("WNS":::/GZIP=0):0

It will then work just fine.

I fully agree that it's a pity to have zilib compress/uncompress utility "custimized" with proprietary wrapper that makes it unusable for any "standard use". However in that case you can use I/O, file or other devices. Personally I use XDEV for compress/decompress and leave the silly %SYSTEM.Util.Compress()/Decompress() alone.

I'm no expert in vectors so I might be wrong, however it seems to me that the problem is not in the VECTOR_COSINE() function, instead it seems to me the problem is in the TO_VECTOR() function inside VECTOR_COSINE().

The documentation for the TO_VECTOR() function for the first argument (data) says:

If you are using TO_VECTOR in Dynamic SQL, other data types are accepted. The allowed data types are determined by the SelectMode. If the SelectMode is ODBC or Display, the data argument may be passed in as a DynamicArray, as well as a string. If the SelectMode is Logical, the data argument must be entered as a $vector.

In this case the SelectMode is Logical and it does not seems to me that the TO_VECTOR() first argument is passed as $vector.