crash-consistent snapshot of the entire system, including all CACHE.DAT files, the manager directory, journals, and the WIJ.

Ray, may I ask you to clarify this a bit?

Is any snapshot of the entire system can be considered crash-consistent?

Several years ago our company (SP.ARM) had developed Krasnoyarsk Kray Regional HIS, which was finally deployed using haproxy to balance CacheActiveX traffic load among 7 ECP application servers. This kind of traffic is based on the same protocol as ODBC, so the behavior should be quite the same (long sessions via 1972/tcp). To achieve high availability of the proxy server, it was deployed in two similar copies connected with ucarp.

Our initial solution for load balancing and high availability of proxy was based on LVS+keepalived, but the engineers of maintainer company (Rostelecom) prefered to use haproxy+ucarp as they felt themselves more comfortable with them; we didn't mind.

Hi Michael,
Using Studio's keyboard heavily, I can add to your list:
Shift-F2 - jump to previous bookmark
F6, Shift-F6 - switch to next and previous opened routine/class window
Ctrl-F7 - compile current routine/class
F7 - [re]compile the whole project
Ctrl-F9, F9, Shift-F9 - deal with break points (similar logic as with bookmarks)
Ctrl-E, Ctrl-Shift-E - switch to full commands/functions names, switch back to shortcuts
Ctrl-Shift-O - open project
Ctrl-O - open routine/class/etc.

P.S. Have not tried Atelier yet.

Just to complete an exercise: 

--- inst-guid.sh ---
#!/bin/bash
csession CACHE <<EOF | grep -E [0-9A-F\-]{36}
write ##class(%SYS.System).InstanceGUID()
halt
EOF
--------------------

$ ./inst-guid.sh
8BCD407E-EE5E-11E4-B9C2-2EAEB3D6024F
$

If you are able to do it on Primary, it would be possible to write a line of code which sets the appropriate SQL Table privileges and insert it into ^ZMIRROR routine under NotifyBecomePrimary entry point (see http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...).

You may also want to try Monitoring for Disconnect Mode (D mode)
(http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=...).
This mode prevents a Caché process from silent halting on disconnect, which occurs by default when the socket device is the process $principal. D mode garranties that process's error trap code will handle <DISCONNECT> errors anyway.

Dynamic dispatch is a hit on speed anyway.

Sounds like yet another reason not to use it in production code.

Macros can help in many cases but using $method() and other flavors of indirect method calls.

1) The following:

USER>do a.Awtf()
 
    do $method(, "OnBefore" _ Method, Args...)
    ^
<METHOD DOES NOT EXIST>%DispatchMethod+2^Utils.Annotations1.1 *OnBeforenBeforenBeforenBeforenBeforenBeforenBeforenBeforenBeforenBeforenBeforenBefo,Utils.Annotations1
USER 58d1>

is not a great problem as can be easily corrected by adding a check whether a method named $e(method,2,*) really exists in the class.

2) The whole approach seems to be too complicated as you omitted an important step:
all calls of all app's methods should be switchable from "do a.methodName" to "do a.AmethodName" back and  forth. How to implement this?

You may also have a look at The Universal MUMPS(M) commander (Alt-NC) (http://minimdb.com/tools/altnc412.html).
It is similar to other commanders well known outside of Caché/M communities, such as Midnight Commander (mc, Linux), File and Archive Manager (far.exe, MS Windows), and of course their predecessor famous Norton Commander (nc, MS DOS) in many ways. Full screen editor which is included into commander is very simple and easy to learn. It is not so sophisticated as great programmers' tools of the past (such as DEC's EDT), but a bit smarter than notepad (MS Windows) as it is full functional without a mouse.

Pros: it runs from a command line, it does't force any extra tcp port to be opened, it supports as much terminal types as OS does, it has a non-% version so you don't need CACHESYS:RW privilege to load the code.

Cons: the screen design is pretty ancient, but should we expect too much from CHUI application which roots come from early 1990x?

As we say in this country, "all psychics are on vacations".
Did you notice any messages that may appear in console log or in backup log?
 

I agree, it's elegant, but not memory efficient design: the sorted array is built only to pick up its top node. Of course, arg lists are rarely long and yet another small array doesn't mean much, but in more generic case of searching the max element in array of sufficient size the following code should be twice more memory efficient and maybe a bit faster:

ClassMethod max(args...) {
  s max=$g(args(1)),im=1 for i=2:1:args s:max<$g(args(i)) max=args(i),im=i
  q $lb(max,im)
}

P.S. I was curious if my variant was really faster and ran some tests. Here are the results that were collected using an array filled with random whole numbers. An array was refilled on each test run. 

In the table below:
 min - the lower limit of numbers values in the array
 max- the upper limit of numbers values in the array
 n - the quantity of numbers in the array
 var - # of variant (1 - original, 2 - mine)
 dt - average run time (in seconds)
 dtfill - avg time of filling the array; just for info.

min       max       n         var  dt        dtfill
-10000000 10000000  10        1    0.000005  0.000002
-10000000 10000000  10        2    0.000001  0.000002
-10000000 10000000  100       1    0.000047  0.000012
-10000000 10000000  100       2    0.000004  0.000012
-10000000 10000000  1000      1    0.000425  0.000115
-10000000 10000000  1000      2    0.000031  0.000115
-10000000 10000000  10000     1    0.005828  0.002949
-10000000 10000000  10000     2    0.000554  0.002949
-10000000 10000000  100000    1    0.074641  0.031128
-10000000 10000000  100000    2    0.006824  0.031128
-10000000 10000000  1000000   1    1.194625  0.313878
-10000000 10000000  1000000   2    0.069191  0.313878

Using a repository is a good idea for sure, but what about a solution that can help even if an 'intruder' had bypassed it and changed a class, e.g., on production server? Here is one which answers who changed SomeClassName.CLS; this code can be executed in "%SYS" namespace using System Management Portal/SQL:

SELECT DISTINCT TOP 1 s.UTCTimeStamp, s.OSUsername, s.Username, s.Description
FROM %SYS.Audit as s
WHERE s.Event='RoutineChange'
AND s.Description LIKE '%SomeClassName.cls%'
ORDER BY s.UTCTimeStamp desc

It's easy to adapt it for searching the same info for MAC, INT and INC routines.
Enjoy!

Sometimes such strange results are caused by ignoring the fact that usually there are several levels of caching, from high to low:

- Caché global cache

- filesystem cache (on Linux/UNIX only, as Windows version uses direct i/o)

- hdd controller cache.

So even restarting Caché can be not enough to drop the cache for clear "cold" testing. The tester should be aware of data volume involved, it should be much more than hdd controller cache (at least). mgstat can help to figure this out, besides it can show when you start reading data mostly from global cache rather than from filesystem/hdd.

Google answered that there are 2 options to replace '<' in PS, please see https://blogs.technet.microsoft.com/heyscriptingguy/2011/07/16/working-around-legacy-redirection-issues-with-powershell/ .

We distribute a COS routine which runs after our application update is imported and does all the necessary job: checks if the TASKMGR's task already exists in schedule, and sets it up if not. Some tasks are the subject for manual setup as their settings may depend on local instance specific.

Fabian, yes, it would be interesting to hear more on your approach.

Recently I faced the similar problem: we were asked for a tool to estimate a size of each global from the top N biggest ones. Our solution is to calculate the global sizes on regular basis (using a Cache Manager's Task) and to transfer the results to external SNMP server (using our own customized MIB). Visualization is provided by SNMP server (we and our customer use Zabbix).

As to global size calculation speed, in our case it takes about 30 minutes for 1TB database. Only allocated space is estimated.

Not tested for speed, while I expect this version should be rather fast as it compares common parts of both references rather than individual suscripts. Enjoy!

tttcmp(fgname,tgname,bKill,nErrTotal,nErrTop) ; Compare [sub]array @fgname with [sub]array @tgname
;In:
; fgname - "original" [sub]array
; tgname - its copy to check with;
; bKill - kill @tgname if it matches to @fgname (default = 0)
; nErrTop - # of mismatches to find to stop comparison
;
;Out:
; returns 1 on full subscripts and data match, else - 0.
; ByRef nErrTotal - # of mismatches.
;
new x,y,xtop,ytop,i,flOK,flQ,xquit,yquit,nErr,xstart,ystart
set bKill=$get(bKill,0)
set nErrTop=$get(nErrTop,1)
set x=fgname,y=tgname write !,"Comparing original "_fgname_" with imported "_tgname_":"
set xstart=$length($name(@x,$qlength(x)))+$select($qlength(x):1,1:2)
set xtop=$select($qlength(x):$extract(x,1,$length(x)-1)_",",1:x)
set ystart=$length($name(@y,$qlength(y)))+$select($qlength(y):1,1:2)
set ytop=$select($qlength(y):$extract(y,1,$length(y)-1)_",",1:y)
set flOK=1,flQ=0,nErr=0,nErrTotal=0
for i=1:1 do  quit:flQ
. set x=$query(@x),xquit=x=""!(x'[xtop)
. set y=$query(@y),yquit=y=""!(y'[ytop)
. if xquit,yquit write " OK. i=",i set flQ=1 quit
. if xquit!yquit write " NO!!!: i=",i,$select(xquit:" "_fgname_" is shorter than "_tgname,1:" "_tgname_" is shorter than "_fgname) set nErrTotal=nErrTotal+1,flOK=0,flQ=1 quit
. if $extract(x,xstart,$length(x))'=$extract(y,ystart,$length(y)) write !,"!!! Ref NEQ: i=",i write !," x=",x,!," y=",y  set nErrTotal=nErrTotal+1,nErr=nErr+1,flOK=0 set:nErr'<nErrTop flQ=1 quit:flQ  ;!,$e(x,xstart,$l(x)),!,$e(y,ystart,$l(y)),
. if $get(@x)'=$get(@y) write !,"!!! Data NEQ: i=",i write !," *** x = ",x,!," x => ",@x,!," *** y = ",y,!," @y => ",@y set nErrTotal=nErrTotal+1,nErr=nErr+1,flOK=0 set:nErr'<nErrTop flQ=1 quit:flQ
. else  set nErr=0
if flOK,bKill write !,"Killing "_tgname_"..." kill @tgname
else  write !,"Not Killing "_tgname
quit flOK

When logging to file, I usually use a snippet as follows, as it keeps log file closed (and readable) almost always:

log(message)
  open tLogFile:"AW"
  use tLogFile
  write message,!
  close tLogFile
  quit

I'd prefer not to use %ALL at all for a couple of reasons:

- if the mapping is created programmatically, it is not a problem to create it for each namespace,

- if it's created manually, one day you may forget about it and loose some important data or some other usefull things stored in non-default DB for years... of course, all this stuff should be documented and re-checked, but for me it is easier to do it once (for my nsp mappings) than twice (for my and for %ALL).