%ToJSON not detecting open websocket as current device

Question

Question

Kevin B Lavallee · Sep 30, 2022

Hello,

I am trying to use the %ToJSON method on my dynamic object, calling it with a "DO" and not passing in outstrm parameter.

We are trying to avoid MAXSTRING errors that we get with some of our abnormally large messages. In order to do this, I am trying to update my code to not call the socket's "Write" method after converting the dynamic object to a JSON string using the %ToString method's output. Per the documentation on the %ToJSON method:
If outstrm is not specified and the method is called via DO, the JSON string is written to the current output device

The previous code (extending from %CSP.WebSocket of course) was doing this:

Method Send(message As %Library.DynamicObject) As %Integer
{
do ..Write(message.Msg.%ToJSON())
quit 1
}

The updated method is doing this:

Method Send(message As %Library.DynamicObject) As %Integer
{
do message.Msg.%ToJSON()
quit 1
}

When I execute the new code it does not write the JSON object directly to the websocket. It is as if the function does not see the websocket as being the current device.

Any suggestions would be greatly appreciated.

Product version: IRIS 2022.1

$ZV: IRIS for UNIX (Red Hat Enterprise Linux 8 for x86-64) 2022.1 (Build 209_0_21727U)

Discussion (10)2

Log in or sign up to continue

Julius Kavay · Sep 30, 2022

The documentation of the %ToJSON() method is correct and yes, you can do

 do obj.%ToJSON()

merely, this works only "on devices without protocol" like terminal, (sequential) file, etc. Everywhere, wehere the data bytes goes direct to the target. WebSocket isn't such a device. There is a "header part", with information fields like the command, masking, the length of the data, etc.

You have two possibilities, a) you ask WRC for a "WriteStream()" method or b) you handle the whole WebSocket by hand (is not impossible) or c) you change your application logic and send the those messages in chunks.

1 0

score 0 · Answer 1 · 2022-10-11T07:53:10-04:00

Thank you Julius for explaining why this does not work on websockets and for the suggestion about the WriteStream method. I did use Nicholai's sample method to implement the changes to work around this issue. Much appreciated.

score 0 · Answer 2 · 2022-09-30T11:23:01-04:00

Why do not pass a stream to %ToJSON, and then Write the stream content?

Set stream = ##class(%Stream.TmpCharacter).%New()
Do message.Msg.%ToJSON(stream)

But, I'm sure that you will not be able to send the whole big stream as a whole message. You still have to split it and receive it knowing that it's not complete.

score 0 · Answer 3 · 2022-09-30T11:42:41-04:00

Julius Kavay · Sep 30, 2022

According to WebSocket protocol, the maximum payload size is (2**(8*8))-1 octets, if I recall it right.

0 0

score 0 · Answer 4 · 2022-09-30T11:46:24-04:00

Protocol and its implementation

I've looked at the realization in IRIS, and it does not support any streams, So, any call of Write, is like a complete message and it accepts only string.

score 0 · Answer 5 · 2022-09-30T11:55:19-04:00

Julius Kavay · Sep 30, 2022

Hence I wrote to OP, quote from my answer, "you ask WRC for a 'WriteStream()' method"

0 0

score 0 · Answer 6 · 2022-09-30T16:19:43-04:00

The %ToJSON method sends JSON text to a device or %Stream one element at a time. Starting in IRIS 2019.1.0 it broke up long JSON strings into blocks of 5460 characters so that strings exceeding the maximum length could be sent to the device. Make sure you are using an IRIS 2019.1.0 or later if you are getting a <MAXSTRING> signal.

In a future IRIS release (now out in a preview release) a change was made such that sequences of many small items would be blocked together and sent to the device in a larger buffer. This is being done to improve the performance of %ToJSON method when sending many small elements to a %Stream.

score 1 · Answer 7 · 2022-09-30T16:36:58-04:00

I think what you'll need is a write stream method below (I haven't had a chance to test this so the code comes with no guarantees). This will work for Non-Shared web-socket connections since I don't have the chance to test with shared pools at the moment.

Method Send(message As %Library.DynamicObject) As %Integer
{
    Set stream = ##class(%Stream.TmpCharacter).%New()
    Do message.Msg.%ToJSON(stream)
    Set sc = ..WriteStream(stream)
    If $$$ISERR(sc) $$$SysLog(2,"WebSocket","[Write] Error WRITE Stream on JSON Command",sc)
    quit 1
}

Method WriteStream(data As %Stream.Object) As %Status
{
	Set $ZTrap="WriteError"
	If i%WSClassProtocolVersion > 1 & i%WSDataFraming = 1 {
		Set head=$ZLChar(data.Size)
		If i%BinaryData = 1 {
			Set head=head_"8"
		} Else {
			Set head=head_"7"
		}
	} Else {
		Set head=""
	}
	#; As there is activity on this session reset the session timeout
	Set sc=$$updateTimeout^%SYS.cspServer(i%SessionId) If $$$ISERR(sc) $$$SysLog(2,"WebSocket","[Write] Error updating session timeout",sc)
	#; Only return an error status if there's an issue with the write itself.
	Set sc=$$$OK
	If (i%SharedConnection = 1) {
        // If you want to try and play with the shared pools and large payloads, move the commented code to the loop below

		#; Set sc=$$CSPGWClientRequest^%SYS.cspServer3(i%GWClientAddress,"WSW "_i%WebSocketID_" "_head_data1,-5,.response)
		#; If $$$ISERR(sc) $$$SysLog(2,"WebSocket","[Write] Error sending request",sc)
        $$$SysLog(2,"WebSocket","[Write] Shared Connections Don't Support Stream Sizes over 3.6MB",sc)
        Quit $$$Error($$$GeneralError, "[Write] Shared Connections Don't Support Stream Sizes over 3.6MB")
	} else {
        SET sc = $$$OK
        // Write header
		Write head
        // Rewind Stream
        Do data.Rewind()
        // Loop through stream and write 64kb chunks
        while (data.AtEnd = 0) {
            // Set Buffer to 64KB
            Set BUFSZ = 65536
            // Read Buffer
            SET BUFBLOCK = data.Read(.BUFSZ)
            // Convert UTF8 if we aren't using Binary Web Sockets
            If i%BinaryData '= 1 {
                Try {
                    Set BUFBLOCK=$zconvert(BUFBLOCK,"O","UTF8")
                } Catch exp {
                    $$$SysLog(2, "WebSocket", "[Write] Exception: "_exp.DisplayString(), data)
                    Set BUFBLOCK=BUFBLOCK
                }
            }
            // Use DEVICE WRITE method
            Write BUFBLOCK
            // Quit when done with stream
            BREAK:(BUFSZ < 65536)
        }
        // Send a flush command mnow
        Write *-3
        Quit sc
	}
	Quit sc
WriteError	
	#; No interrupts during cleanup or error processing
	Do event^%SYS.cspServer2("WebSocket Write Error: "_$ZError)
	$$$SetExternalInterrupts(0)
	Set $ZTrap="WriteHalt"
	Hang 5
	Close 0
WriteHalt 
	Halt
    
}

score 0 · Answer 8 · 2022-10-11T07:52:10-04:00

Thank you Nicholai for this method. I made a few modifications to the code but I very much appreciate the great starting point.

score 0 · Answer 9 · 2024-01-19T09:17:34-05:00

Hello. This is a followup after my work with this. We implemented the code but ran into an issue with unprintable characters. We use wingding fonts for some of our data in tables. We could not figure out why messages with unprintable characters got lost in transmission and never got from the server to the client. It turns out that the issue is the fact that the "head" (header) variable was being created based on the size of the stream "before UTF8 conversion". So if the original data size was 500 characters, the header was indicating that the message size was 500. Then, as the data is read in chunks from the stream and the UTF8 conversion was done, there was one unprintable (char 252) that was in the data, causing the converted data size to be 501 characters which cause the message transmission to fail.

To address this, we had to delay the creation of the header until all of the data was read from the stream, UTF8 converted and save into a process private global. Once the entire stream was UTF8 translated into the PPG, we derived the new length of the data and then looped through the PPG and wrote everything to the websocket and that fixed the problem.

I am not thrilled with having to have the data go through two intermediate steps to get it from the dynamic object to the websocket, but, because the UTF8 conversion of unprintable characters, I saw no alternative.

Is there any chance there is another option that anyone is aware of that would allow me to determine the UTF8 character count of the stream without going through a process private global?

Thanks again for the continued support.

Here is a portion of the code from my writeStream method that deals with the new order of processing:

...

do data.Rewind()
// Convert ASCII stream into UTF8 temp global
set streamSeq = 0
set dataLength = 0
while ('data.AtEnd) {
set bufferBlock = data.Read(.BUFSZ)
// Convert UTF8 if we aren't using Binary Web Sockets
if i%BinaryData '= 1 {
try {
set bufferBlock=$zconvert(bufferBlock,"O","UTF8")
} catch exp {
set bufferBlock=bufferBlock
}
}
set ^||streamData($i(streamSeq)) = bufferBlock
set dataLength = dataLength + $length(bufferBlock)
}

// Write header to the socket
if ((i%WSClassProtocolVersion > 1) && (i%WSDataFraming = 1)) {
set head = $ZLChar(dataLength)
if (i%BinaryData = 1) set head = head_"8"
else set head = head_"7"
} else {
set head = ""
}
write head

// Loop through temp global and write to the socket
set seq = 0
for {
set seq = $order(^||streamData(seq))
if (seq = "") { quit }
write ^||streamData(seq)
}

// Send a flush command now
write *-3
...