You should not forget that WebSockets is still used the HTTP protocol, so, you should send some headers first.
Look at this my code, it uses plain OPEN, and I'm not sure if my example 100% correct, but works.
set securityKey = $SYSTEM.Util.CreateGUID()
set securityKey = $SYSTEM.Encryption.MD5Hash(securityKey)
set securityKey = $SYSTEM.Encryption.Base64Encode(securityKey)
set host = "echo.websocket.org"
set url = "/"
set port = 80
set device = "|TCP|"_port
Write "GET ",url," HTTP/1.1",!
Write "Accept: */*",!
Write "Host: ",host,!
Write "Connection: Upgrade",!
Write "Upgrade: websocket",!
Write "Sec-WebSocket-Key: ",securityKey,!
Write "Sec-WebSocket-Version: 13",!
Set fullResponse = ""
Set response = ""
Set fullResponse = fullResponse_response_$char(13)
} While $test
In this case, it only reads the first response, which actually should be with HTTP headers as well. Something like this.
HTTP/1.1 101 Web Socket Protocol Handshake
Date: Tue, 05 Feb 2019 10:51:37 GMT
Server: Kaazing Gateway
It does not contain the first portion of data, which some WebSocket servers may send after initial connect. But if your server sends it, you should see it at the and of response. If you have to send something before, you should do it after the first response, which says that connection established and you can send any data. But not any, it should be in binary format, more details you can find here. Any responses also decoded.