WebMethod - Issue with encoding (special character)

Hello guys,

I've got this piece of code which runs the method "WebMethod", that belongs to %SOAP.WebBase.cls.

It grabs the outcome from an internal webservice we have and after that, it writes into a file.

The thing is, when I browse the file contents, I realize that in the place where a special character should be, I see a question mark.

By querying the same webservice from a special soap tool called "SoapSonar" (I've been using this for years), the outcome shows up this special character (shows it properly).

So I wonder... is there kind of encoding parameter I could use within the "WebMethod", so it'd return the correct output?

---

Method RunTree(Tree As %String, Inputs As %String, Debug As %Integer) As RMHCommon.Output [ Final, ProcedureBlock = 1, SoapBindingStyle = document, SoapBodyUse = literal, WebMethod ]
{
   Set p=##class(RMHCommon.Output).%New()
   Set p=..WebMethod("RunTree").Invoke(##this,"http://soap.uk/Interface.Soap.RunTree",.Tree,.Inputs,.Debug)

   set fs=##class(%Stream.FileBinary).%New()
   set fs.Filename="c:\TEMP\SoapTree.xml"
   set tSC=fs.CopyFrom(p.Value)
   set tSC=fs.%Save()
...
}

Many thanks!

  • + 1
  • 0
  • 2404
  • 15
  • 1

Answers

Based on our comment thread I think the most likely cause is that the pInput %Library.GlobalCharacterStream (originating from the Value property of the object that was returned when you Invoke your webmethod) starts with an XML header claiming that its encoding="UTF-8" but I suspect that the characters within that global stream are actually Unicode rather than UTF8-encoded.

To test this theory I suggest you create a new %Library.FileCharacterStream, set its TranslateTable property to "UTF8", then use its CopyFromAndSave method to fill it with the contents of pInput. Now pass your FileCharacterStream to your %XML.Reader's OpenStream method and see if you still get the SAX error.

John,

Just to let you know, I am still a beginner with ObjectScript/Cache/Healthsare, so please excuseany primary mistakes you may encounter :)

Can you kindly help me out to write this piece of code you advised? (P.S: msg.Value is the stream)

set fs=##class(%Library.GlobalCharacterStream).%New()
set fs.TranslateTable = "UTF8"
set tSC=fs.CopyFromAndSave(msg.Value)

  sc= xread.OpenStream(???)

if $$$ISERR(sc)
{
...
}

---

I came up with that a few moments ago: (at this point I have the stream available (msg.Value) and I will handle it in 2 different ways: by writing is a file and trying to open it (OpenFile) and by reading it (OpenStream))

  //
  // 1st approach - it succeeds, so the file errorOpenFile.xml is NOT generated
  //    
  d msg.Value.Rewind()
  set fs=##class(%Stream.FileCharacter).%New()
  set fs.Filename="D:\DATABASES\OVPATH\temp\test.xml"
  set fs.TranslateTable = "UTF8"
  set tSC=fs.CopyFrom(msg.Value)
  set tSC=fs.%Save()
    
  s reader=##class(%XML.Reader).%New()
  Set sc = reader.OpenFile("temp/test.xml")
    
  if $$$ISERR(sc)
  {
    set fs=##class(%Stream.FileCharacter).%New()
    set fs.Filename="D:\DATABASES\OVPATH\temp\errorOpenFile.xml"
    do fs.Write($$$StatusDisplayString(as))
    set tSC=fs.%Save()
  }    

  //
  // 2nd approach - it fails, so I have the file errorOpenStream.xml generated
  //    
  Set xread = ##class(%XML.Reader).%New()
  Set xread.IgnoreNull=1                  
  d msg.Value.Rewind()    
  s sc= xread.OpenStream(msg.Value)
    
  if $$$ISERR(sc)
  {
    set fs=##class(%Stream.FileCharacter).%New()
    set fs.Filename="D:\DATABASES\OVPATH\temp\errorOpenStream.xml"
    do fs.Write($$$StatusDisplayString(sc))
    set tSC=fs.%Save()
  }

I think you could simplify your first approach a little by reverting to calling OpenStream on your reader object rather than using OpenFile:

  // 1st approach - it succeeds, so the file errorOpenFile.xml is NOT generated
  //    
  d msg.Value.Rewind()
  set fs=##class(%Stream.FileCharacter).%New()
  set fs.Filename="D:\DATABASES\OVPATH\temp\test.xml"
  set fs.TranslateTable = "UTF8"
  set tSC=fs.CopyFrom(msg.Value)
  set tSC=fs.%Save()
    
  s reader=##class(%XML.Reader).%New()

  d fs.Rewind() // might not be necessary, but won't hurt
  Set sc = reader.OpenStream(fs)

 

Anyhow, the fact that you don't get an error confirms my hypothesis that the original stream (msg.Value) contains Unicode data but the reader treats it as though it is UTF8-encoded.

In your code above I think you can also omit the line where you set fs.Filename and instead allow the stream to generate its own temporary file. Explicitly naming the file may be handy when debugging, but it will cause problems if more than one process runs this code concurrently.
 

Hello mate,

It generates the same error: ERROR #6301: SAX XML Parser Error: invalid character 0x18 while processing Anonymous Stream at line 1 offset 4183

Here is the code:

msg.Value.Rewind()
set fs=##class(%Stream.FileCharacter).%New()
set fs.Filename="D:\DATABASES\OVPATH\temp\test0.xml"
set fs.TranslateTable = "UTF8"
set tSC=fs.CopyFrom(msg.Value)
set tSC=fs.%Save()

reader=##class(%XML.Reader).%New()
fs.Rewind() // might not be necessary, but won't hurt
Set sc = reader.OpenStream(fs)

if $$$ISERR(sc)
{
set fs=##class(%Stream.FileCharacter).%New()
set fs.Filename="D:\DATABASES\OVPATH\temp\errorOpen0.xml"
do fs.Write($$$StatusDisplayString(sc))
set tSC=fs.%Save()
}

Just in case it's relevant, what is your $ZV string?

w $zv
Cache for Windows (x86-64) 2016.2.1 (Build 803U) Wed Oct 26 2016 12:31:28 EDT

OK John, thanks for your help so far.

I've been in touch with them, but they couldn't so far figure out what is it... we're discussing, but I will post an update here whenever we figure out the resolution (if any, lol).

Cheers

Comments

Please clarify what you mean by "browse the file contents".

Are you opening c:\TEMP\SoapTree.xml in a text editor?

Maybe that editor is assuming that the file is UTF8-encoded.

Can you view it in a tool that shows you the byte values it contains?

Did you have a particular reason for choosing to write the file using an instance of %Stream.FileBinary instead of %Stream.FileCharacter?

Also, be aware that the WebMethod classmethod of %SOAP.WebBase is tagged as "Internal" and commented thus:

/// This method is used internally by Caché. You should not make direct
/// use of it within your applications. There is no guarantee made about either
/// the behavior or future operation of this property.
 

Hello mate,

Thanks for you quick response. Actually there was no specific reason for using the FileBinary, so I managed to change it to FileCharacter and used also the property "TranslateTable = "UTF8", which hence allowed the file to be written with the special character I wanted.

By discovering that, I came back to the 1st place I originally thought the issue would be and now I guess that it's the right place to look at:

Method OnProcessInput(pInput As %Stream, Output pOutput As %RegisteredObject, ByRef pHint As %String) As %Status
{
    // Create an instance of %XML.Reader
    Set xread = ##class(%XML.Reader).%New()
    Set xread.IgnoreNull=1
        
    // Begin processing of the file
    pInput.Rewind()
    
    sc= xread.OpenStream(pInput)
   
     if $$$ISERR(sc)
{
set fs=##class(%Stream.FileCharacter).%New()
set fs.Filename="c:\TEMP\OnProcessInput00.xml"
set fs.TranslateTable = "UTF8"
do fs.Write($$$StatusDisplayString(sc))
set tSC=fs.%Save()
}

---

Here I had the file created and the error message written in it (ERROR #6301: SAX XML Parser Error: invalid character 0x18 while processing Anonymous Stream at line 1 offset 4627).

So I presume that
sc= xread.OpenStream(pInput)

Is erroring, because the stream contains a special character.

What do you think?

Thanks

I'll repeat what I wrote on the previous DC thread you opened about this issue:

I think you need to focus on the input stream. What type of stream is pInput ? You can get its classname using pInput.%ClassName(1)

If it is a file stream, what wrote it? Does its file contain a BOM at the start?

What does the start of the pInput stream contain? One quick but ugly way of checking this would be to write it to a scratch global before the call to OpenStream which errors, e.g.

Set ^tMurillo=pInput.Read(255) Do pInput.Rewind()

Then afterwards run the following in Terminal:

w ^tMurillo

w $a(^tMurillo,1),!,$a(^tMurillo,2),!,$a(^tMurillo,3)

This should show us whether the creator of the stream started it with a BOM character or sequence, and also whether there is an XML header specifying an encoding.

That information is apparently important to the %XML.SAX.StreamAdapter used by %XML.SAX.Parser, which in turn is what the OpenStream method of %XML.Reader uses.

Hello John,

Thanks again for your help!

There you go the outcome:

1st one:

<?xml version="1.0" encoding="UTF-8" ?>< ... (more content)

2nd one:

60
63
120

I'm not sure about the BOM, but the XML returned does state the UTF-8 encoding.

There's no BOM, and the XML header claims that the content is UTF-8 encoded. But on your other post you reported that your content contains a left-single-quote character and that the SAX parser choked on a character, which I suspect was this character in Unicode form rather than UTF-8 encoded.

So, what originally wrote the pInput stream's content? Did it actually UTF-8 encode the data it wrote to the stream?

My 1st message on this post shows the piece of code where the pInput is originally fed.

And as I told there, I write the output content to a file and once I open this file, the contents are shown properly and the character is there, intact.