All
Accepted answers
Eduard Lebedyuk · Nov 16, 2018 go to post

It only matters that the async node is caught up.

Also you need to use SSL.

It all depends on connectivity (bandwidth and lag).

Eduard Lebedyuk · Nov 16, 2018 go to post

ZSTART and ZSTOP are kind of vague, how does that fit into ccontrol start and ccontrol stop. Is there any setting where this can be set to automatic?

ZSTART and ZSTOP are executed on startup/shut down automatically.

Eduard Lebedyuk · Nov 16, 2018 go to post

For a case with one index there are only two variables:

  • Number of distinct values
  • Number of total records.

Or is there anything else?

And on a Z axis the timing.

Sure in a real environment we see interference of a several indices, but here we're talking about one index and how it affects query timings.

Eduard Lebedyuk · Nov 16, 2018 go to post

Ok, then, why it is so important, why I it is not recommend to use bitmap in case of so many unique values?

Bitmap indices lose efficiency with a big number of distinct values.

And what should be used instead?

Normal indices.

Actually the question gathered a fair amount of interest. I think I would run tests and publish the results.

My plan is:

1. Create a class with two properties:

Property Number As %Integer(MINVAL=1, MAXVAL=<DISTINCT VALUES COUNT>);

Property Data As %String;

2. Increase MAXVAL up by one from 2 to 20000;

3. Repopulate the class with 10 000 000 values.

4. Switch between Normal and Bitmap indices

5. Rebuild indices.

6. Purge queries.

7. Tune table.

8. Remount the database to purge cache.

9. Run two queries 10 times:

  • select id, by random condition on Number
  • select data by random condition on number

10. Write results into new table { distinct values, index type, cold run, avg run, max run}

11. Go to 2.

Does that pest plan makes sense? Any ideas? Should probably test on cases where index fits into globuf and where it does not.

Eduard Lebedyuk · Nov 16, 2018 go to post

2% or more means less than distinct 50 values.

Thank you, Robert! New value for my list. I think I got 64 once but that's a new one.

UPD: Nevermind this comment.

Eduard Lebedyuk · Nov 16, 2018 go to post

That's the number I hear most often (5-10k rang) when I ask about maximum number of unique values for bitmap indices.

I heard from a few hundred to 20k as a max value.

Eduard Lebedyuk · Nov 10, 2018 go to post

Thanks!

I see it still uses $zf(-1) in latest instead of $zf(-100). I thought $zf(-1) is unavailable now.

Eduard Lebedyuk · Nov 10, 2018 go to post

You need so specify your base URL as URL property and pass the parameters in a method:

 set sc = ..Adapter.Get(.httpResponse, "key,key2", "value", "value2")

It would send the following request:

GET /baseURL?key=value&key2=value2 HTTP/1.1
Eduard Lebedyuk · Nov 9, 2018 go to post

If it's a REST/CSP so you can just write the response to the current device add at the beginning of your method:

set %response.ContentType = "html"
do ##class(%CSP.Utils).DisplayAllObjects()
return $$$OK

It would output all headers as Cache sees them.

Have you tried:

write %request.GetCgiEnv($zcvt("HTTP_MyCustomHeader", "U"))
Eduard Lebedyuk · Nov 9, 2018 go to post

Note that you should build bitmap indices only for properties that have less than ~6400 distinct values.
Also building indices may take time.
Don't forget to recompile embedded SQL and purge dynamic SQL queries.

Eduard Lebedyuk · Nov 1, 2018 go to post

Use ValidateStream method of Ens.Util.XML.Validator class. It does exactly what you need.

Eduard Lebedyuk · Oct 25, 2018 go to post

As I'm the author of RESTForms, I'd recommend that. The advantage is that it offers not only JSON<->object transformation, but a complete CRUD REST API for your data.

Calling @Stefan Cronje

Eduard Lebedyuk · Oct 25, 2018 go to post

All approaches you are considering can work. if you're storing everything in one database, I'd recommend RLS.

Much more important question is do you need storing everything separately or in the same place.

Advantages of separated databases/servers:

  • Easy to scale
  • Would work faster on a lot of cases
  • Easy to delete/rebuild a chunk
  • Easier security

Advantages of unified database/server:

  • Cross-chunk queries are easier (For example: This venue is sold out on the dates you need. We recommend the following nearby venues)
  • Simple Backup/HA strategies
Eduard Lebedyuk · Oct 22, 2018 go to post

You can just pass the stream directly, no need to read from it:

set parObj = {}.%FromJSON(%request.Content)
Eduard Lebedyuk · Oct 18, 2018 go to post

Achieving  FIFO with Ensemble is an interesting task.

Do you control queue population?

If so  you can send the message only if the queue is empty and process is idle.

For example create a proxy process that sends only one message at a time.

Eduard Lebedyuk · Oct 16, 2018 go to post

There are far more efficient ways to do that.

  1. Move tables you want shared into a separate Namespace with separate Code/Data databases.
  2. Map your data and code into original namespace.
  3. Verify that it all works as expected. So far it should work as before.
  4. Create a mirroring configuration.
  5. Add 2 created databases to the mirror configuration.
  6. Add second server as a DR mirror.
  7. Move databases to a DR mirror and mount them there.
  8. (Optional) Create a namespace with 2 mirrored databases.
  9. Add desired mappings on a second system.

Docs.