Sending HTTP requests to store special characters‽

Question

Question

Yone Moreno · Mar 5, 2020

We would like to be able to store accented characters into an API.

Currently our Rest Operation is the following:

Class Operaciones.REST.NotificacionesPUSH.CrearNotificacion Extends EnsLib.REST.Operation
{ Parameter INVOCATION = "Queue"; Method CrearNotificacion(pRequest As Mensajes.Request.NotificacionesPUSH.CrearNotificacion, pResponse As Mensajes.Response.NotificacionesPUSH.CrearNotificacion) As %Library.Status
{

Set body = {
     "app_id": (pRequest.idApp),
     "headings": {"en":(pRequest.notificacion.titulo),"es":(pRequest.notificacion.titulo)},
     "subtitle": {"en":(pRequest.notificacion.subtitulo),"es":(pRequest.notificacion.subtitulo)},
     "contents": {"en":(pRequest.notificacion.mensaje),"es":(pRequest.notificacion.mensaje)},
     "data":(^data),
     "include_player_ids": (^idsDispositivos)
}

set httpRequest = ##class(%Net.HttpRequest).%New()
set tResponse = ##class(%Net.HttpResponse).%New()
set httpRequest.Server = "onesignal.com"

set httpRequest.ContentType = "application/json"

set httpRequest.Authorization = "Basic "_##class(Util.TablasMaestras).getValorMaestra("NOTIFICACIONESPUSH.PARAMETRIZACIONES","ONESIGNAL_APIKEY_SCS")

set httpRequest.Https = 1 set httpRequest.SSLConfiguration = "Certificado_SCS"

set dataObject = ##class(%DynamicObject).%FromJSON(body.data)

set body.data = dataObject $$$LOGINFO("body.'include_player_ids': "_body."include_player_ids")
set idsDispositivosObject = ##class(%DynamicObject).%FromJSON(body."include_player_ids")

set body."include_player_ids" = idsDispositivosObject //set status =
Do httpRequest.EntityBody.Write(body.%ToJSON()) while(httpRequest.EntityBody.AtEnd=0){
set linea = httpRequest.EntityBody.Read()
}
$$$LOGINFO("linea: "_linea)
set tSC = httpRequest.Post("https://onesignal.com/api/v1/notifications") set tResponse = httpRequest.HttpResponse
if $$$ISERR(tSC){
$$$ThrowOnError(tSC)
}
if (tResponse.Data.AtEnd = 0) {

set linea = tResponse.Data.Read()

} set pResponse = ##class(Mensajes.Response.NotificacionesPUSH.CrearNotificacion).%New()
do pResponse.informacion.Write(linea) Quit pResponse
} XData MessageMap
{
<MapItems>
  <MapItem MessageType="Mensajes.Request.NotificacionesPUSH.CrearNotificacion">
    <Method>CrearNotificacion</Method>
  </MapItem>
</MapItems>
} }

We have read:

https://community.intersystems.com/post/managing-utf-8-characters-databa...

https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls...

We have tried to use:

set httpRequest.ContentType = "application/json;charset=utf-8"

do httpRequest.SetHeader("Content-Type","application/json; charset=utf-8")

set httpRequest.ContentCharset = "utf-8"

And even we tried to convert the headings itself:

//"headings": {"en":($ZCONVERT(pRequest.notificacion.titulo,"O","UTF8")),"es":(pRequest.notificacion.titulo)},

//"headings": {"en":($ZCONVERT(pRequest.notificacion.titulo,"I","UTF8")),"es":(pRequest.notificacion.titulo)},

Currently we see that our operation's LOGINFO let's us see that the JSON contains the accented character, in this case an "'ó"

However when we view the notification stored in the API it shows as a strange character:

Besides we have checked that if we send a create notification request dirctly from SoapUI, it is been stored with strange characters too:

--> However if we set the Encoding to UTF-8 in SoapUI, it is being stored correctly:

So then, we would like to change the charset to utf-8 in Ensemble.

If we try:

do httpRequest.SetHeader("Content-Type","application/json; charset=utf-8")

The API does not recognize the Content-Type: application/json

We have also read:

https://stackoverflow.com/questions/17154967/is-content-encoding-being-s...

How could we send HTTP requests to store special characters‽

Discussion (10)3

Log in or sign up to continue

Tomas Vaverka · Mar 9, 2020

Yone, try to set ContentCharset AFTER setting of ContentType:

set httpRequest.ContentType = "application/json"
set httpRequest.ContentCharset = "UTF-8"

This is from %Net.HttpRequest class documentation:

property ContentCharset as %String [ Calculated ];

This is the charset to encode the contents with. This is actually specified in the HTTP Content-Type header with something like:
Content-Type: text/html; charset=UTF-8

You must set this property after you set the ContentType or it will overwrite this value.

Will it change anything?

0 0

score 0 · Answer 1 · 2020-03-05T12:46:18-05:00

Try defining these settings and see if that solves your issue.

set httpRequest.ContentCharset = "utf-8"
set httpRequest.ContentType = "application/json"
set httpRequest.ContentEncoding = "utf-8"
set httpRequest.NoDefaultContentCharset = 0 // Specifically this one.

score 0 · Answer 2 · 2020-03-06T02:06:34-05:00

Hello, Rubens Silva. We have tried the settings and the message is being sent, however the accented letters still being replaced by strange characters:

We have tried:

set httpRequest.ContentCharset = "utf-8"
set httpRequest.ContentType = "application/json"
set httpRequest.ContentEncoding = "utf-8"
set httpRequest.NoDefaultContentCharset = 0

score 0 · Answer 3 · 2020-03-06T07:02:23-05:00

Is your Caché unicode?

Can you try reading the content of headings.en right before you send it? So that we can see see if that's already corrupted before the HttprRequest takes control or if it's corrupting the input.

score 0 · Answer 4 · 2020-03-06T08:07:20-05:00

We observe:

linea: {"app_id":"5cf57b56-c3b4-4a0d-8938-4ac4466f93af","headings":{"en":"Cita Atención Primaria","es":"Cita Atención Primaria"},"subtitle":{"en":"C.P. ISORA","es":"C.P. ISORA"},"contents":{"en":"Aqui el contenido del mensaje si aplicase","es":"Aqui el contenido del mensaje si aplicase"},"data":{"centro":"C.P. ISORA","fecha":"yyy/mm/dd","hora":"hh:mm","profesional":"nombre del profesional","nomUsuario":"nombre de usuario","codcita":"idCita","sepuedeborrar":"1"},"include_player_ids":["114a63e3-1da9-40fa-a3ec-e6677466e11a"]}

Being the code:

Method CrearNotificacion(pRequest As Mensajes.Request.NotificacionesPUSH.CrearNotificacion, pResponse As Mensajes.Response.NotificacionesPUSH.CrearNotificacion) As %Library.Status
{
$$$LOGINFO("^data: "_^data) /*
Set body = {
     "app_id": (pRequest.idApp),
     "headings": {"en":(pRequest.notificacion.titulo),"es":(pRequest.notificacion.titulo)},
     "subtitle": {"en":(pRequest.notificacion.subtitulo),"es":(pRequest.notificacion.subtitulo)},
     "contents": {"en":(pRequest.notificacion.mensaje),"es":(pRequest.notificacion.mensaje)},
     "data":(^data),
     "include_player_ids": ["c2917a6f-6ecf-4f45-8b31-9b72538580fd"]
}
*/ $$$LOGINFO("^idsDispositivos: "_^idsDispositivos) Set body = {
     "app_id": (pRequest.idApp),
     "headings": {"en":(pRequest.notificacion.titulo),"es":(pRequest.notificacion.titulo)},
     "subtitle": {"en":(pRequest.notificacion.subtitulo),"es":(pRequest.notificacion.subtitulo)},
     "contents": {"en":(pRequest.notificacion.mensaje),"es":(pRequest.notificacion.mensaje)},
     "data":(^data),
     "include_player_ids": (^idsDispositivos)
} //"headings": {"en":($ZCONVERT(pRequest.notificacion.titulo,"O","UTF8")),"es":(pRequest.notificacion.titulo)}, set httpRequest = ##class(%Net.HttpRequest).%New()
set tResponse = ##class(%Net.HttpResponse).%New()
set httpRequest.Server = "onesignal.com" set httpRequest.ContentCharset = "UTF-8"
set httpRequest.ContentType = "application/json"
set httpRequest.ContentEncoding = "UTF-8"
set httpRequest.NoDefaultContentCharset = 0
//set httpRequest.ContentType = "application/json;charset=utf-8"
//do httpRequest.SetHeader("Content-Type","application/json; charset=utf-8")
//set httpRequest.ContentCharset = "utf-8"
//do httpRequest.SetHeader("Content-Encoding","gzip") //set httpRequest.Authorization = "Basic "_##class(Util.TablasMaestras).getValorMaestra("NOTIFICACIONESPUSH.PARAMETRIZACIONES","onesignal_apikey")
set httpRequest.Authorization = "Basic "_##class(Util.TablasMaestras).getValorMaestra("NOTIFICACIONESPUSH.PARAMETRIZACIONES","ONESIGNAL_APIKEY_SCS")
$$$LOGINFO("httpRequest.Authorization: "_httpRequest.Authorization)
set httpRequest.Https = 1 set httpRequest.SSLConfiguration = "Certificado_SCS"
$$$LOGINFO("body.data: "_body.data)
set dataObject = ##class(%DynamicObject).%FromJSON(body.data)
$$$LOGINFO("dataObject: "_dataObject)
set body.data = dataObject $$$LOGINFO("body.'include_player_ids': "_body."include_player_ids")
set idsDispositivosObject = ##class(%DynamicObject).%FromJSON(body."include_player_ids")
$$$LOGINFO("idsDispositivosObject "_idsDispositivosObject)
set body."include_player_ids" = idsDispositivosObject //set status = ##class(%ZEN.Auxiliary.jsonProvider).%WriteJSONStreamFromObject(httpRequest.EntityBody, body,,,,"aeloqtuw")
Do httpRequest.EntityBody.Write(body.%ToJSON())
while(httpRequest.EntityBody.AtEnd=0){
set linea = httpRequest.EntityBody.Read()
}
$$$LOGINFO("linea: "_linea) set tSC = httpRequest.Post("https://onesignal.com/api/v1/notifications") set tResponse = httpRequest.HttpResponse
if $$$ISERR(tSC){
$$$ThrowOnError(tSC)
}
if (tResponse.Data.AtEnd = 0) {
//$$$LOGINFO("En AñadirDispositivo, tamaño de tResponse: "_tResponse.Data.Size)
set linea = tResponse.Data.Read()
//$$$LOGINFO("Linea: "_linea)
} set pResponse = ##class(Mensajes.Response.NotificacionesPUSH.CrearNotificacion).%New()
do pResponse.informacion.Write(linea) Quit pResponse
}

score 0 · Answer 5 · 2020-03-05T13:22:08-05:00

Marc Mundt · Mar 5, 2020

Yone, have you tried setting ContentCharset to "UTF-8" instead of "utf-8"?

0 0

score 0 · Answer 6 · 2020-03-06T02:07:41-05:00

Hello, Marc Mundt. We have tried to use the following:

set httpRequest.ContentCharset = "UTF-8"
set httpRequest.ContentType = "application/json"
set httpRequest.ContentEncoding = "UTF-8"
set httpRequest.NoDefaultContentCharset = 0

And the message is still being stored with strange characters:

score 0 · Answer 7 · 2020-03-06T08:49:27-05:00

I have tried to test your code and found the Content-Length was set to 0 when ContentCharset was set.

I investigated that it was cased by the code reading the data before Posting to server, therefore the EntityBody stream was at end and calculated size after translation to UTF-8 was zero.

To fix it you need to rewind the stream before you send it using Post() method:

set httpRequest.ContentType = "application/json"
set httpRequest.ContentCharset="utf-8"
...

Do httpRequest.EntityBody.Write(body.%ToJSON()) while(httpRequest.EntityBody.AtEnd=0){
set linea = httpRequest.EntityBody.Read()
}

Do httpRequest.EntityBody.Rewind() //REWIND the data to start from the beginning after reading !!!
$$$LOGINFO("linea: "_linea)
set tSC = httpRequest.Post("https://onesignal.com/api/v1/notifications") set tResponse = httpRequest.HttpResponse

Does it help?

score 0 · Answer 8 · 2020-03-09T09:07:19-04:00

Hello, Tomáš Vaverka,

We have tried to rewind httpRequest.EntityBody

And we see it is being stored in the API with the strange character.

The code is:

while(httpRequest.EntityBody.AtEnd=0){
set linea = httpRequest.EntityBody.Read()
}

Do httpRequest.EntityBody.Rewind()

Being the complete code:

Method CrearNotificacion(pRequest As Mensajes.Request.NotificacionesPUSH.CrearNotificacion, pResponse As Mensajes.Response.NotificacionesPUSH.CrearNotificacion) As %Library.Status
{
$$$LOGINFO("^data: "_^data) $$$LOGINFO("^idsDispositivos: "_^idsDispositivos) Set body = {
     "app_id": (pRequest.idApp),
     "headings": {"en":(pRequest.notificacion.titulo),"es":(pRequest.notificacion.titulo)},
     "subtitle": {"en":(pRequest.notificacion.subtitulo),"es":(pRequest.notificacion.subtitulo)},
     "contents": {"en":(pRequest.notificacion.mensaje),"es":(pRequest.notificacion.mensaje)},
     "data":(^data),
     "include_player_ids": (^idsDispositivos)
} //"headings": {"en":($ZCONVERT(pRequest.notificacion.titulo,"O","UTF8")),"es":(pRequest.notificacion.titulo)}, set httpRequest = ##class(%Net.HttpRequest).%New()
set tResponse = ##class(%Net.HttpResponse).%New()
set httpRequest.Server = "onesignal.com" set httpRequest.ContentCharset = "UTF-8"
set httpRequest.ContentType = "application/json"
set httpRequest.ContentEncoding = "UTF-8"
set httpRequest.NoDefaultContentCharset = 0
//set httpRequest.ContentType = "application/json;charset=utf-8"
//do httpRequest.SetHeader("Content-Type","application/json; charset=utf-8")
//set httpRequest.ContentCharset = "utf-8"
//do httpRequest.SetHeader("Content-Encoding","gzip") //set httpRequest.Authorization = "Basic "_##class(Util.TablasMaestras).getValorMaestra("NOTIFICACIONESPUSH.PARAMETRIZACIONES","onesignal_apikey")
set httpRequest.Authorization = "Basic "_##class(Util.TablasMaestras).getValorMaestra("NOTIFICACIONESPUSH.PARAMETRIZACIONES","ONESIGNAL_APIKEY_SCS")
$$$LOGINFO("httpRequest.Authorization: "_httpRequest.Authorization)
set httpRequest.Https = 1 set httpRequest.SSLConfiguration = "Certificado_SCS"
$$$LOGINFO("body.data: "_body.data)
set dataObject = ##class(%DynamicObject).%FromJSON(body.data)
$$$LOGINFO("dataObject: "_dataObject)
set body.data = dataObject $$$LOGINFO("body.'include_player_ids': "_body."include_player_ids")
set idsDispositivosObject = ##class(%DynamicObject).%FromJSON(body."include_player_ids")
$$$LOGINFO("idsDispositivosObject "_idsDispositivosObject)
set body."include_player_ids" = idsDispositivosObject //set status = ##class(%ZEN.Auxiliary.jsonProvider).%WriteJSONStreamFromObject(httpRequest.EntityBody, body,,,,"aeloqtuw")
Do httpRequest.EntityBody.Write(body.%ToJSON())
while(httpRequest.EntityBody.AtEnd=0){
set linea = httpRequest.EntityBody.Read()
} Do httpRequest.EntityBody.Rewind() $$$LOGINFO("linea: "_linea) set tSC = httpRequest.Post("https://onesignal.com/api/v1/notifications") set tResponse = httpRequest.HttpResponse
if $$$ISERR(tSC){
$$$ThrowOnError(tSC)
}
if (tResponse.Data.AtEnd = 0) {
//$$$LOGINFO("En AñadirDispositivo, tamaño de tResponse: "_tResponse.Data.Size)
set linea = tResponse.Data.Read()
//$$$LOGINFO("Linea: "_linea)
} set pResponse = ##class(Mensajes.Response.NotificacionesPUSH.CrearNotificacion).%New()
do pResponse.informacion.Write(linea)
while(pResponse.informacion.AtEnd=0){
set informacion = pResponse.informacion.Read()
}
$$$LOGINFO("informacion: "_informacion)
Do pResponse.informacion.Rewind() Quit pResponse
}

In addition, we observe that in our Operation the notification is:

{"app_id":"5cf57b56-c3b4-4a0d-8938-4ac4466f93af","headings":{"en":"Cita Atención Primaria","es":"Cita Atención Primaria"},"subtitle":{"en":"C.P. ISORA","es":"C.P. ISORA"},"contents":{"en":"Aqui el contenido del mensaje si aplicase","es":"Aqui el contenido del mensaje si aplicase"},"data":{"centro":"C.P. ISORA","fecha":"yyy/mm/dd","hora":"hh:mm","profesional":"nombre del profesional","nomUsuario":"nombre de usuario","codcita":"idCita","sepuedeborrar":"1"},"include_player_ids":["114a63e3-1da9-40fa-a3ec-e6677466e11a"]}

And it is being stored with rare characters:

How could we debug this behaviour?

score 0 · Answer 9 · 2020-03-10T13:22:30-04:00

Hello Tomáš Vaverka,

We have tried the following code:

set httpRequest.ContentType = "application/json"
set httpRequest.ContentCharset = "UTF-8"
set httpRequest.ContentEncoding = "UTF-8"
set httpRequest.NoDefaultContentCharset = 0

And it does store the notification with the accented letters.

We appreciate that you have spent time and effort helping us in this task, Tomáš Vaverka.