Thanks John-  This was exactly my problem.  

The only issue I have with this is that in the case of a stream containing a very large amount of data, as I read through the data there is no guarantee that I'm going to get the entire coded entity.  For example: given the following block of text:

{freeText: This is some free text \u001a}

As I read through the stream using .Read() the first read could return "{freeText: this is some free text \u" and then the second call to .Read() could return "001a}".

Can $zcvt work on stream data?

In order to implement this I think I would have to put together some pretty good code to handle these cases.  The documents I am trying to remove characters from are very large and wouldnt be able to be stored in a single string for use with $zcvt, I think.

At the point that the data is accessible as distinct strings the decoding has already been done.  At this point I can just do a $tr to get rid of the non printable characters, which is what I have done.

Problem is this-  At the time that I have access to do this the data is contained in a stream.  Further complicating things is that the $C(26) is actually encoded as \u001a because the contentType of the response from the REST service is Application/JSON which automatically encodes these characters using the JSON notation.

I wanted to implement a more general solution for this, removing all non-printable characters, however, it seems that I need to implement this stripping in the transform where the contents of the data actually contain the non-encoded non-printable characters.

I have a simple method that I have created to remove these characters:

ClassMethod stripNonPrintables(string As %String) As %String

{

f i=0:1:8,11,12,14:1:31 set chars=$g(chars)_$c(i)

quit $tr(string, chars)

}

So, in places where we are seeing these characters, we can simply call this strip method.  Not the solution I wanted to implement, but it will work.

The CSP file should be in the SAMPLES namespace.

Peter-

I can think of two ways, one using the plugin and a second by just adding a new dimension to the cube that compared these two values.  Would there be any benefit to doing it either way?  I would think less coding in the case of adding a dimension to the cube.

Thanks Bachhar, 

Expanding on this a bit, we also have the need to be able to query the message store for messages we have sent/received from a given interface.  These are all HL7 messages, which means they are stored in EnsLib.HL7.Message. 

I'm assuming there is a way to correlate a message in EnsLib.HL7.Message with either Ens.MessageBody or Ens.MessageHeader where regular Ensemble messages are stored.

What would be the proper way to correlate these?  I looked at the fields in EnsLib.HL7.Message but don't see anything that stands out as obvious connection between the two tables.

So if we were to use the APIs to update the dimension table directly for example, this would not be adequate to insure that future queries showed the proper data?

As mentioned already, running ##class(HoleFoods.Cube).%KillCache() would actually go in and purge the internal globals associated with the result cache for that particular cube.  Not really the best thing to do on a production system.

A safer way to invalidate cache for a given cube would be to run %ProcessFact for an individual record within the source table for the cube.

For example

&sql(select top 1 id into :id from <sourcetable>)

do ##class(%DeepSee.Utils).%ProcessFact(<cubename>, id)

replacing <sourcetable> with the source table for your cube and <cubename> with the name of the cube.

This has the effect of causing the result cache for the cube to be invalidated without completely purging the internal globals that store the cache.  This is much safer to run during production times than the %KillCache() method.

One concern with using this method is that it actually goes in and kills all of the cache for the cube by killing the globals containing the cache.  What affect will this have if it is done while users are using the Analyzer for example?

And yes, there are many ways to accomplish this task from the Cache server side. 

The document data model doesn't really apply here as this is not a new application.

Once 2016.2 is released, we could just map the globals as objects using CacheSQLStorage and then use the .$toJSON() method to export to JSON at that point.

That's one of the great things about the Cache technology, there are so many different options and choices for doing the same thing.

This post is really just about a discussion on what the best way to represent a global structure in JSON, not the specifics of how to accomplish that in code.

Stefan-

My use case is that I have a Cache application (not object based) and I want to provide a set of REST services for accessing the raw global structures, all methods.  I want this REST interface to be generic in nature such that it can be used to read, set and kill any global node of any global.  In order to do this, I need an encoding method for sending and receiving the global data.  JSON seems the best way to package the data, thus I was looking for a good JSON structure that could represent the global structure.

Continuing my testing with Ens.Utils.HTML.Parser, I am trying to parse information contained within paragraphs.

For example:

<p>This is paragraph one</p>

<p>This is paragraph two with some <i>italics</i>.</p>

<p>This is paragraph three</p>

I want to parse the contents of each of these paragraphs, including the italics content contained within paragraph two.

I setup my template to:

+<p>{paragraph}</p>+

This kind of works, except in the case of the second paragraph, it stops when it hits the <i> tag even though if it were to continue it would eventually hit the </p> tag.  So what ends up being parsed is:

This is paragraph one

This is paragraph two with some

This is paragraph three

What I am expecting is:

This is paragraph one

This is paragraph two with some <i>italics</i>

This is paragraph three

Does this seem to be a limitation of the parser?  Is there any way to get what I'm trying to get from this document using the parser?

I was able to use this method (Ens.Utils.HTML.Parser) to successfully parse disease information from the CDCs website.  Basically I created a persistent class to store disease names along with the source URLs from the CDC's a-z web pages.

The template looks like this:

<div,class=span16><ul>+<li><a,class=noLinking,href={pageurl}>{pagetitle}</a></li>+</ul>

and the Class method that actually does the parsing of all of the pages:

ClassMethod getDiseasesOrCondition(Output tCount As %Integer) As %Status
{
set tCount=0
set template="<div,class=span16><ul>+<li><a,class=noLinking,href={pageurl}>{pagetitle}</a></li>+</ul>"
for alpha=1:1:26 {
kill tOut
set url="http://www.cdc.gov/DiseasesConditions/az/"_$c(96+alpha)_".html"
do ##class(Ens.Util.HTML.Parser).test(url, template, .tOut)
for i=1:1 {
quit:'$d(tOut("pageurl",i))
if tOut("pageurl",i)?1"http://www.cdc.gov/".e {
set iCDC=##class(iCDC.DiseaseOrCondition).%New()
set iCDC.title=tOut("pagetitle",i)
set iCDC.sourceUrl=tOut("pageurl",i)
set tSC=iCDC.%Save()
set tCount=$i(tCount)
}
}
}
quit $$$OK
}

There was a little checking that was needed to verify that the url that was returned for the source url's were actually pointing to the CDCs website, but other than that, the Ens.Utils.HTML.Parser worked exactly as I had hoped it would.

Very clean and straight forward for my needs.