Hi Alexey, good question. There is no one size fits all. My  aim is to highlight how external backups work so teams responsible can evaluate their best solution when talking to vendors. 

Third-party solutions will be a suite of management tools, not simply backup/restore, so there are many features to evaluate. For example, products that backup VMs will have features for change block tracking (CBT) so only changed blocks in the VM (not just changes to CACHE.DAT) are backed up. So incremental. But they also include many other features including replication, compression, deduplication, and data exclusion to manage what is backed up, when and what space is required. Snapshot solutions at the storage array level also have many similar functions. You can also create your own solutions integrating freeze/thaw, for example using LVM snapshots. 

Often a Caché application is only one of many applications and databases at a company. So usually the question is turned around to "can you backup <Caché Application x> with <vendor product y>". So now with knowledge of how to implement freeze/thaw you can advise the vendor of your Caché application requirements.

Addendum: I was reminded that there is an extra step when configuring external backups.

When the security configuration requires that the backup script supply Caché credentials, you can do this by redirecting input from a file containing the needed credentials. Alternatively, you can enable OS-level authentication and create a Caché account for the OS user running the script.

Please see the online documentation for full details

I wanted to blast out the polymetric dashboard to a bunch of my lab VMs this morning so I used ansible, I put code snippet in case its useful.

I admit this is a bit of a hack but shows some useful syntax.

polykit-install.yml

---
### Main Task - instal polymetric dashboard ###
- debug: msg="Start polykit-install.yml"

- name: local path on laptop for tar.gzip of dashboard kit relative to ansible root
  set_fact:
    local_package_path: ./z-packages

- debug: 
    msg: "{{ local_package_path }}"

# hack! ASSUME only one instance per host!!!! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ !!!!

- name: get instance name
  shell:
    ccontrol qlist | awk -F '^' '{print $1}'
  register: instance_name

- debug: var=instance_name.stdout    

- name: get instance path (add ./mgr )
  shell:
    ccontrol qlist | awk -F '^' '{print $2}'
  register: instance_path

- debug: var=instance_path.stdout    

- name: set base path
  set_fact:
    base_path: "{{ instance_path.stdout }}/mgr/polykit"

# hack! ASSUME only one instance per host!!!! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ !!!!


# Create install directory on target
- name: Create install directory
  file:
    dest="{{ base_path }}"
    mode={{ 777 }}
    state=directory

# Copy polykit file to install directory
- name: Copy polykit.tar.gz to install directory
  copy:
    src="{{ local_package_path }}/polykit-v1.0.0.tar.gz"
    dest="{{ base_path }}"
    remote_src=False

# Unarchive
- name: Unarchive polykit database
  unarchive:
    src="{{ base_path }}/polykit-v1.0.0.tar.gz"
    dest="{{ base_path }}"
    creates="{{ base_path }}/polykit-v1.0.0/DashboardInstaller.xml"
    remote_src=yes


# Run installer

# Build text file

- name: build csession file line at a time
  shell:
    echo "set status = \$System.OBJ.Load(\""{{ base_path }}/polykit-v1.0.0/DashboardInstaller.xml"\", \"ck\")" >"{{ base_path }}"/polykit-inst.txt

- shell:
    echo "set status = {{ '#' }}{{ '#' }}class(SYS.Monitor.DashboardInstaller).Install(\""{{ base_path }}/polykit-v1.0.0"\",0)" >> "{{ base_path }}"/polykit-inst.txt

- shell:
    echo "h">>"{{ base_path }}"/polykit-inst.txt

- shell:  
    cat "{{ base_path }}"/polykit-inst.txt
  register: polyinst_check
- debug: var=polyinst_check.stdout_lines

- name: Install dashboard
  shell:  
    cat "{{ base_path }}/polykit-inst.txt" | csession "{{ instance_name.stdout }}" -U %SYS
  register: polyinst_install
- debug: var=polyinst_install.stdout_lines

Hi Alexey, I have not seen Caché implement distributed databases as you describe at customers. I can see how flash solutions in storage IO is a way to increase IOPS and lower latency, so perhaps some way to moving the choke point on that  particular problem. Of course the problem has not gone away we just moved threshold.

Hi Anzelem, Obviously you had some problems. And the best solution in your case was to use a single instance. My experience is ECP is widely used where customers want a single database instance to scale beyond the resources of a single server. On a well configured solution the impact for a user , e.g. response time for a screen, should be negligible. I have also seen cases where ECP is chosen for HA.

Bottlenecks in the infrastructure (network, storage or application) will have an impact.  This is true for single instance or ECP configurations. As noted in the post ECP has strict requirements for network and and storage that will impact performance. There are also considerations in application design. 

It is true there are additional latencies when using a distributed architecture. Even on a well resourced set-up expect there to be some loss of overall efficiency when comparing processing on a single server vs distributed architecture -- Four ECP application servers (CPU + Memory size x) will not produce throughput equal to the total of four database servers (CPU + Memory size x) running separate instances of an application.  But as above this should not impact individual users experience.

Hi Timur, thanks for the comments and links. I agree, 3D XPoint is a case of waiting to see real performance when it's released. Even 10x lower latency is still a big jump - the figures in the post are what is publicly talked about by Micron now. My aim is to give people a heads up on what's coming and to look out for it (although vendors will be shouting it from the rooftops :) Hopefully we will have some real data and pricing soon.

Hi, back of the envelope logic is like this:

For the old server you have 8 cores. Assuming the workload does not change:

Each core on the new server is capable of about 25% more processing throughput. Or another way; each core of the old server is capable of of about 75% processing throughput of the new server. So roughly (8 *.75) old cores equates to about 6 cores on the new server.

You will have to confirm how your application behaves, but if you are using the calculation to work out how much you can consolidate applications on a new virtualized server you can get a good idea what to expect. If it is virtualized you can also right-size after monitoring to fine tune if you have to.

14 Nov 2016. Updated the table images - just some simple typos in the headings.

BTW Although I have not tried implementing - due to high number of Healthcare applications using Caché I thought it may be of interest that Paessler PRTG has developed new sensors for monitoring medical equipment that communicate via HL7 and DICOM.

https://www.paessler.com/blog/2016/04/13/all-about-prtg/ehealth-sensors-...

Hi, if the question is how to format then I use the three tildes on the line immediately before and after the block of code "~~~" eg. The following code block had the tildes fencing the code on the line immediately before and immediately after the block.

eg.

~~~
//Get PDF stream object previously saved
  Set pdfStreamContainer =
etc
etc
~~~

//Get PDF stream object previously saved
   Set pdfStreamContainer = ##Class(Ens.StreamContainer).%OpenId(context.StreamContainerID)
   Try {
     Set pdfStreamObj = pdfStreamContainer.StreamGet()
   }
   Catch {
      $$$TRACE("Error opening stream object ID = "_context.StreamContainerID)
      Quit
   }
 //Set PDF stream object into new OBX:5 segment
   Set status = target.StoreFieldStreamBase64(pdfStreamObj,"OBX("_count_"):ObservationValue(1)")
   Set ^testglobal("status",$H) = status

I note that the engine for the community website should automatically highlight known languages but you can override .

eg. On first line before "~~~javascript" or "~~~python"

var s = "JavaScript syntax highlighting";
alert(s);

s = "Python syntax highlighting"
print s

Hi, thanks for the comments. 

Your right I had a bad choice of words... For ECP systems sustained throughput average write response time is for all journals -- because they are on the same disk.

I really can't give guidance on specific IO metrics you will have to validate for you own systems.

Hi Alexey, the ECP post is up now.

https://community.intersystems.com/post/data-platforms-and-performance-p...