All
Accepted answers
Alexey Maslov · Jan 25, 2017 go to post

Always call "do a.AmethodName()"

I don't think it's a good idea as each extra nested call inevitably introduces extra performance drawback.

Anyway, imagine an existing large app with 100x classes and 1000x of methods calls, some of which can be done indirectly. In most common case an ability to do such transformation (all a.m() to a.Am()) sounds unrealistic, while your approach may fit small projects with simple calling conventions, especially when they are being started from scratch.

Alexey Maslov · Jan 25, 2017 go to post

1) The following:

USER>do a.Awtf()
 
    do $method(, "OnBefore" _ Method, Args...)
    ^
<METHOD DOES NOT EXIST>%DispatchMethod+2^Utils.Annotations1.1 *OnBeforenBeforenBeforenBeforenBeforenBeforenBeforenBeforenBeforenBeforenBeforenBefo,Utils.Annotations1
USER 58d1>

is not a great problem as can be easily corrected by adding a check whether a method named $e(method,2,*) really exists in the class.

2) The whole approach seems to be too complicated as you omitted an important step:
all calls of all app's methods should be switchable from "do a.methodName" to "do a.AmethodName" back and  forth. How to implement this?

Alexey Maslov · Jan 23, 2017 go to post

My answer would be simple: monitor your database growth and do the same on global level. It can help to estimate the future growth and with some luck even to point out inefficient solutions that may waste the space. 

E.g. we have a task that calculates all globals sizes (usually once a night) and store this data in a global that can be used as a source for visualization and allerting by means of some external toolset. At the moment integration with Zabbix is implemented.

Alexey Maslov · Jan 23, 2017 go to post 
 set mbSize=$$AllocatedSize^%GSIZE($Name(@Gref))/1048576 // Global @Gref size in MegaBytes

Pros:
- uses proven calculation algorithm, the same as interactive %GSIZE utility uses (we met errors in %GlobalEdit::GetGlobalSizeBySubscript() calculation, while it was in "rather old" v.2015.1.2, not sure about 2017.1);
- supports the whole global (Gref="^A") as well as its parts (Gref=$name(^A("subs1") or Gref=$name(^A("subs1","subs2")) or ...) as arguments.
Con(s):
- allows to perform only quick global size estimation, while in most cases it's quite enough;
- presize (slow) version can also be derived from %GSIZE source, but it's more tricky exercise...

Alexey Maslov · Jan 20, 2017 go to post

It seems to be pretty peculiar case, at least for me. Sounds like Cache is supplied with English Unicode locale which does not support all unicode characters. What for?

Alexey Maslov · Jan 19, 2017 go to post

Murray, did you mean that LVM snapshot volume is a sequential file in contrast to VMWare snapshot delta disk which is random access one? So, with the similar DB write load the snapshot size would be greater in case of LVM, won't it?

I tried LVM snapshot and backup with Caché freezing once upon a time. AFAIR, the space should be preliminary reserved inside LV for snapshot volume, and if it was too low the snapshot could fail. 

Alexey Maslov · Jan 17, 2017 go to post

Hi Timur, thank you for the thorough writing! 

BTW, our company uses home-brewed package manager (while we don't call it that way) for our app updates distribution. Typical package for Caché comprises classes and routines in XML format, globals in FF (our proprietary) format, and a manifest file which contains the update description as well as its before- and after installation actions.

Alexey Maslov · Jan 17, 2017 go to post

You may also have a look at The Universal MUMPS(M) commander (Alt-NC) (http://minimdb.com/tools/altnc412.html).
It is similar to other commanders well known outside of Caché/M communities, such as Midnight Commander (mc, Linux), File and Archive Manager (far.exe, MS Windows), and of course their predecessor famous Norton Commander (nc, MS DOS) in many ways. Full screen editor which is included into commander is very simple and easy to learn. It is not so sophisticated as great programmers' tools of the past (such as DEC's EDT), but a bit smarter than notepad (MS Windows) as it is full functional without a mouse.

Pros: it runs from a command line, it does't force any extra tcp port to be opened, it supports as much terminal types as OS does, it has a non-% version so you don't need CACHESYS:RW privilege to load the code.

Cons: the screen design is pretty ancient, but should we expect too much from CHUI application which roots come from early 1990x?

Alexey Maslov · Jan 13, 2017 go to post

Hi Murray,
thank you for continuing your series.

Don't you think that VM image backup (despite of its importance) has a drawback as it may contain pretty huge amount of data that is unnecessary for simple database restoration? E.g., VM image may contain hundreds of gigabytes of journals useless for the database state in backup file. IMHO, in this case a kind of selective backup can be attractive. Not aware of Veeam, but sure that Acronis can do it on file system level. I wonder if selective external backup (e.g., in the case of Veeam) can be integrated with Cache DB freeze/thaw features with the same ease as a full one?

Alexey Maslov · Jan 3, 2017 go to post

As we say in this country, "all psychics are on vacations".
Did you notice any messages that may appear in console log or in backup log?
 

Alexey Maslov · Dec 29, 2016 go to post

c. Map the relevant globals to a non-journaled database

Sometimes it's being done just to conceal some typical app level drawbacks (such as missing the cases when  temporary globals can be used, (excessive) looped rewrites of persistent data, etc) although it may lead to more serious administrative level drawbacks.

Normally all app's globals (whether they are journaled or not) should by coordinated with each other; if a global should not (or may not), it is a good candidate to be mapped to CACHETEMP DB. Any fail-over scenario you may imagine includes a step of journal restoration. Believe me, it can be a great problem for admin to decide what to do with each non-journaled global after the server fault: which one can be fully (or partially) KILLed, and which one need to be ^REPAIRed. Mostly the right decision is impossible without the developer's intervention. So, the simple dev level solution can introduce much more serious admin level complications.

IMHO, a global should be considered of one of two types basing on its content type:
1. "normal" persistent data: such global should be placed in journaled DB without any exceptions;
2. temporary data which is need only during process run: map it to CACHETEMP or make it private ^||global.

Sometimes I was told by developers that a third type exists which comprises some pre-generated data that is stored to improve performance of some heavy jobs (e.g. reporting). The pre-generation process can be time (and resource) consuming, so it looks like that the best place for this 3d type globals is a non-journaled DB. After several database reparations I'd answer "No!". Depending on pre-generation process details, each of these globals can (and should) be put in one of two categories taking in account that reboots of modern servers are relatively rare events:
- if it's not too hard to regenerate the data: just automate it in your code and map the global to CACHETEMP;
- if not, consider your global as operational one and place it into journaled DB; to reduce excessive journaling during its generation, just choose between approaches "e" or "d" of Tani's article.

Alexey Maslov · Dec 26, 2016 go to post

Well done, Dima! Just my 2c: any key should be really any. The correction is easy: change exit condition to

r q:0 q:$t

Season greetings to everybody!

Alexey Maslov · Dec 8, 2016 go to post

Hi Murray,
Speaking of ECP, we usually imagine distributed data processing on several app servers. But what about distributed databases? E.g., can the solution to split the databases among several data servers just to distribute heavy journal and/or write daemon load be smart in some cases?
I see some drawbacks of this solution:
#1 AFAIK, there is no distributed transactions support in Caché.
#2 To couple it with mirroring, one should deploy N mirrors, where N is the number of (primary) data servers; having no option of "coherent mirroring", their N backup members can have different latency against their primaries, so (baring in mind #1) switching mirror members can have worse consequences as in traditional case of only one (primary) data server.

Have I missed something? Maybe you've seen some field cases when distributing databases looked smart, haven't you?

Thank you,

Alexey Maslov · Dec 7, 2016 go to post

Thanks, Andreas.

Will Caché_for_Ubuntu Field Test kit be published on your Developer Download portal?

Alexey Maslov · Dec 6, 2016 go to post

Presumably, it's a security issue. Check effective UID and GID of your Caché processes. To do it, you may check parameters.isc file from Caché install directory for lines like these: 

security_settings.cache_user: cacheusr
security_settings.cache_group: cacheusr

Unlikely user cacheusr has access rights to other user's home directory.

csession processes are the exception from others as they inherit calling user's UID.

IMHO, it's better to use some neutral folder for file exchange, e.g. "/tmp/myexchange", as in this case it's much easier to establish appropriate assess rights for each side involved in exchange.

P.S. UNIX® Users, Groups and Permissions stuff is well-documented, see:  http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY…

Alexey Maslov · Nov 27, 2016 go to post

Eugene mentions OMI and ECP as traditional interconnection kinds of transport. None of them is quite secure, one may check docs for proof (as to ECP: http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY…;).

ECP is defined as a basic service in contrast to resource based ones. Basic services can't efficiently use resource/roles/users security model by design. It's understood as their layer is usually too low to apply it.

Eugene's services can be dealt with the same precautions as other basic services, i.e. inside secured perimeter only. Port 500x can (and should) be closed on external firewall.

Alexey Maslov · Nov 22, 2016 go to post

Did it work better than what?

Both classes are really the same one, as %Library is the default package name for %-classes.

Alexey Maslov · Nov 18, 2016 go to post

Hi Wolf,

Thank you for the info. As to big string data caching enhancement, I guess that it was targeted mostly to bitmap indexes performance improvement; maybe somebody from InterSystems would like to comment it.

Alexey Maslov · Nov 17, 2016 go to post

Thank you, John. I've corrected this terminological mix which occurred due to my efforts of making introduction part as short as possible.
P.S. Of course, I remember DCP and DDP (not mentioning RVG). Cheers!

Alexey Maslov · Nov 4, 2016 go to post

Hi all,
Setting a startup routine like this can't lock non-terminal based Cache functionality (CSP, etc) as the routine is started on terminal kind of logins only, while I agree that LOGIN^%ZSTART should be sufficient in most cases. IMHO, this per user settings of startup routine in SMP comes from "the good old days" of terminal oriented apps which were often tied to terminals to prevent end users from entering MUMPS commands.
=
Cheers,
Alex

Alexey Maslov · Oct 31, 2016 go to post

Hi Michael,

As I recall each SWD supports dedicated kind of hdd devices (different at the OS level). 

Alexey Maslov · Oct 31, 2016 go to post

Thank you, Ray.
As to our modest experience, WAN failure was the only type of disasters through two years of Krasnoyarsk Data Center of Regional HIS production life, so its potential network isolation does not seem to be non-realistic.
We hope that the mentioned enhancement would not come too late.

Alexey Maslov · Oct 31, 2016 go to post

Are there any plans to introduce in 2017.1 a feature of Quick Old Primary Switching Back which seems to be of great importance for the scenario of temporary move of the Primary role to Promoted Async DR? It is known as Prodlog 142446. In a few words:

After the promotion of DR Async without partners check, the old Primary / Backup members functionality would be likely restored only after rebuilt. Copying the ~1TB backup using long distance link can take many hours or even days, so it will be nice to track a point when the databases were last time "in sync" (while I'm not sure if this term can be used in the case of DR async). After that:
- discard the SETs/KILLs that could be made on old primary after this point
- demote (?) it to be a new DR async
- having the most recent journal on this new DR async, we can promote it to be a very new primary (returning it its "native" role).

Alexey Maslov · Oct 31, 2016 go to post

Thank you, Bob, you mostly answered my questions.

you would compare the name and date of the most recent journal file from the async to the most recent journal file on the DR you are going to promote to see if you can get more recent journal data, which may not be the most recent.

At the meantime we are (internally) discussing the worst case of complete isolation of main Data Centre, so both Member A and B can be not available. In this case the only thing we can do is to check if the DR Async we are going to promote has the more recent journal data among all other available Asyncs, right?

Alexey Maslov · Oct 31, 2016 go to post

The version of Caché is 2015.1.4 for Windows x64, on Core i5 based laptop. When I have a time gap, I'd re-run this test on more powerful server, while I don't expect a noticeable difference.

Let's try to estimate time for both operations:
1: set max(args(i))=i ~ time_to_find_args(i)_position_in_max_array + time_to_allocate_memory_for_max(args(i)) ~  O(ln(length(args(i))) + O(length(args(i)+length(i))) ~ O(3*ln(length(args(i)))
2: max<args(i)  ~ time_to_compare_max_and_args(i) ~ O(ln(length(args(i))))

So it seems that 2 should be ~3 times quicker than 1, but we don't know real coefficients which stand behind those O() estimations. I should confess that local array node allocation penalty turned to be higher than I expected.

This speed difference should be even more would args(i) values be strings rather than numbers.

Alexey Maslov · Oct 31, 2016 go to post

Bob,

I have a couple of questions on DR Async.

1) There is an option of DR Promotion and Manual Failover with Journal Data from Journal Files
( http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY… )
where one is advised to get journal files from one of failover members.
If both failover members are not available, is it possible to get journal files from another (DR or reporting) Async member?
If so, what preliminary configuration steps one should proceed on that member to allow this option in the case of disaster?


2) Another question is on journal files collection as well. You wrote: 

Asyncs receive journal data from the primary asynchronously, and as a result may sometimes be a few journal records behind

Is it true that Asyncs pulls journal data from Primary? If so, Primary is not aware of whether data was pulled by Async or not. Therefore, if Async was not getting journal data from Primary for several days (e.g. due to communication problems), the next unread journal file can be already purged from Primary's local storage.

Is it possible to recover from this situation without rebuilding Async? E.g., if the purged journals are available as a part of file system level backup, or those files are kept on another Async server, can it help?

==
Thanks...

Alexey Maslov · Oct 30, 2016 go to post

I've amended the testing result due to some inaccuracy found.

New results surprised me as I didn't expect to scan a local array about 10 times faster than to create a new one. 

Alexey Maslov · Oct 29, 2016 go to post

I agree, it's elegant, but not memory efficient design: the sorted array is built only to pick up its top node. Of course, arg lists are rarely long and yet another small array doesn't mean much, but in more generic case of searching the max element in array of sufficient size the following code should be twice more memory efficient and maybe a bit faster:

ClassMethod max(args...) {
  s max=$g(args(1)),im=1 for i=2:1:args s:max<$g(args(i)) max=args(i),im=i
  q $lb(max,im)
}

P.S. I was curious if my variant was really faster and ran some tests. Here are the results that were collected using an array filled with random whole numbers. An array was refilled on each test run. 

In the table below:
 min - the lower limit of numbers values in the array
 max- the upper limit of numbers values in the array
 n - the quantity of numbers in the array
 var - # of variant (1 - original, 2 - mine)
 dt - average run time (in seconds)
 dtfill - avg time of filling the array; just for info.

min       max       n         var  dt        dtfill
-10000000 10000000  10        1    0.000005  0.000002
-10000000 10000000  10        2    0.000001  0.000002
-10000000 10000000  100       1    0.000047  0.000012
-10000000 10000000  100       2    0.000004  0.000012
-10000000 10000000  1000      1    0.000425  0.000115
-10000000 10000000  1000      2    0.000031  0.000115
-10000000 10000000  10000     1    0.005828  0.002949
-10000000 10000000  10000     2    0.000554  0.002949
-10000000 10000000  100000    1    0.074641  0.031128
-10000000 10000000  100000    2    0.006824  0.031128
-10000000 10000000  1000000   1    1.194625  0.313878
-10000000 10000000  1000000   2    0.069191  0.313878
Alexey Maslov · Oct 29, 2016 go to post

Certainly yes: SQL quering of Audit database (%SYS.Audit) won't help if Auditing is switched off.