Thank you for excellent articles, Murray.

We use slightly different approach for memory planning.

Our app mostly runs as a set of concurrent user sessions, one process per user. It's known that avg memory per process is 10Mb, we multiply it by 3*N_concurrent_users. The 1st multiplier (3) makes a gap for memory spikes. So, the result is a memory we leave for user processes.

We try to leave for Routine Buffer cache as much memory as possible, upto 1Gb.

The Global Buffer memory is usually calculated as a 30% of 3-years-old-database size for given kind of customer. Usually it comes to 24-64Gb global cache for medium to large size hospitals and provides thousands (or dozens of thousands) Rdratio. At whole, we usually get numbers that are close to your 60/40 proportion, while my globuff calculation method is not so presized as yours and I feel that I need a better calculation base for it.

I'd prefer not to use %ALL at all for a couple of reasons:

- if the mapping is created programmatically, it is not a problem to create it for each namespace,

- if it's created manually, one day you may forget about it and loose some important data or some other usefull things stored in non-default DB for years... of course, all this stuff should be documented and re-checked, but for me it is easier to do it once (for my nsp mappings) than twice (for my and for %ALL).

Hi Mark,

May I ask you to clarify a bit.
Am I right thinking that journal transfer is acknowledged only when it runs between failover nodes, so network latency is not a great problem for async communications? Of course, the bigger latency the bigger journal transfer delay, but it would not slow down the primary node operation.
E.g., if we have average 10-20 JrnWrts per second during each busy hour (with spikes up to 300), for right async WAN planning I should take in account that the latency <= 50 ms (1000/20=50) should be sufficient if we can live with DR async node that is usually about several seconds behind the primary?

As I can remind, shadowing allows cascading. Don't you think that cascading can be a good option for deploying of several async nodes at one (long distance) location as well?

Thanks,
Alex
 

Thank you for the article, Mark. I have a couple of questions on your "Option 3: Geographically Dispersed Deployments".

You placed two DR acync nodes at Data Center B. So, the data flow from Primary (at Data Center A) to DR async (at Data Center B) will be doubled. In general, WAN is not too quick, so it may increase latency. More important that mirrored databases at both DR async nodes may have different states (due to different latency), and it would be difficult to decide which of them should be promoted as a new Primary.
And how to catch up the database at the new Secondary, if we can't guarantee that it is in the same state as at the new Primary?

Well-well, Ed. The intention of following check up:

if zzzzzzZ'["zzzz" 

was to avoid logging my own variables which names were deliberately started with this nasty prefix. I agree that it's rather naive trick, but it worked.
As to original code, it was taken from the existing code base and complied our internal coding standards; I preferred to publish it "as is", having neither time nor will to re-test it.

Somewhen I used the following function to save stacked calls and all variables defined. Variables are not separated by stack levels, that seems to be the reason of $$getst() quickness.

 getst(zzzzgetvars,zzzzStBeg,zzzztemp) ; Save call stack in local or global array
 ; In:
 ; zzzzgetvars = 1 - save variables defined at the last stack level
 ; zzzzgetvars = 0 or omitted - don't save; default = 0
 ; zzzzStBeg - starting stack level for save; default: 1
 ; zzzztemp - where to save ($name).
 ; Out:
 ; zzzztemp    - number of stack levels saved 
 ; zzzztemp(1) - call at zzzzStBeg level
 ; zzzztemp(2) - call at zzzzStBeg+1 level
 ; ...
 ; zzzztemp(zzzztemp) - call at zzzzStBeg+zzzztemp-1 level
 ;
 ; Calls are saved in format:
 ; label+offset^rouname +CommandNumberInsideCodeLine~CodeLine w/o leading spaces"
 ; E.g.:
 ; zzzztemp(3) = First+2^%ZUtil +3~i x=""1stAarg"" s x=x+1 s c=$$Null(x,y)"
 ; Sample calls:
 ; d getst^%ZUtil(0,1,$name(temp)) ; save calls w/o variables in temp starting from level 1
 ; d getst^%ZUtil(1,4,$name(^zerr($i(^zerr)))) ; save calls with variables in ^zerr starting from level 4
 zzzzloop,zzzzzzZ,zzzzStEnd
 zzzzgetvars=$g(zzzzgetvars),zzzzStBeg=$g(zzzzStBeg,1) @zzzztemp @zzzztemp=0 zzzzStEnd=$STACK(-1)-2
 for zzzzloop=zzzzStBeg:1:zzzzStEnd @zzzztemp=@zzzztemp+1,@zzzztemp@(@zzzztemp)=$STACK(zzzzloop,"PLACE")_"~"_$zstrip($STACK(zzzzloop,"MCODE"),"<W") zzzzgetvars,(zzzzloop=zzzzStEnd) d
 . zzzzzzZ="" for  zzzzzzZ=$o(@zzzzzzZ) q:zzzzzzZ=""  if zzzzzzZ'["zzzz" @zzzztemp@(@zzzztemp,zzzzzzZ)=$g(@zzzzzzZ)
 . $ze'="" @zzzztemp@(@zzzztemp,"$ze")=$ze
 1

Just adding my 2c to "4. Tip For UNIX sites":
All necessary unix/linux packages should be installed before the first invocation of 

Do run^pButtons(<any profile>)

otherwise some commands may be missed in ^pButtons("cmds"). I've recently faced it at the server where sysstat wasn't installed: `sar -d` and `sar -u` commands were absent. If you decide to install it later (`sudo yum install sysstat`in my case), ^pButtons("cmds") would not be automatically updated without little help from you: just kill it before calling the run^pButtons().

This is actual at least for pButtons v.1.15c-1.16c and v.5 (which recently occurred on ftp://ftp.intersys.com/pub/performance/), in Caché 2015.1.2.

Murray, thank you for the series of articles.

A couple of questions I have.

1) Documentation (2015.1) states that Rdratio is a Ratio of physical block reads to logical block reads,
while one can see in mgstat log Rdratio values >> 1 (usually 1000 and more). Don't you think that the definition should be reversed?

2) You wrote that:
If you do see high PhyRds on your system there are a couple of strategies you can consider:
...
- Long running read only reports, batch jobs or data extracts can be run on a separate shadow server or asynchronous mirror to minimise impact on interactive users and to offload system resource use such as CPU and IOPS.

I heard this advice many times, but how to return report results back to primary member? (ECP) mounting of remote database that resides on primary member is prohibited on backup member, and vise versa. Or these restrictions does not apply to asynchronous members (never played with them yet)?
 

Hi Mark,

You are right, in our setup Cache' is able to remove/assign the VIP to whichever node is the primary mirror member. All production traffic goes through ECP, so the mirror's VIP is used to simplify some administrative tasks only (just to recognize primary node quicker).

Virtual router is a standard component of vCloud Director, so called Edge Gateway. It sits between a single external address and internal load balancer implemented as a couple of Linux based VMs, each of which has haproxy for balancing and ucarp for VIP/failover. Load balancer distributes incoming client connections among several ECP application servers (Linux based VMs as well). 

That's how it works, in short. If somebody is interested in more verbose description, I could write more about it.

Regards,
Alex

Hi All,

Last year we deployed cloud based Regional MIS of Krasnoyarsky Region of Russia. It was based on O7 National Cloud Platform which had VMWare vSphere inside. One of the Cloud components was a virtual router which allowed us to do NAT/PAT between external and internal networks. So, mirroring VIP was maintained in internal network and permanently mapped to external (fixed) IP without the need to remap it on mirror nodes role exchange.

This project was the real challenge for us not only due to its large scale (7000 named and 1800 concurrent users, > 600GB database), but mostly due to exciting technology mix we were to try and use: virtualization, ECP, Mirroring, application side load balancing and fault tolerance, etc.