UTF8 has to be set to the socket translation table:

Set sock.TranslationTable="UTF8"

and the solution in objectscript for the Java function:

DataOutputStream out = new DataOutputStream(socket.getOutputStream());

out.writeUTF(str);

is the following:

Do sock.Write($$setShort($Length(str)), 0, .sc)

Do sock.Write(str, 0, .sc)

setShort(num) {
Quit $Reverse($ZWChar(num))
}