We are receiving more and more requests for VSS integration, so there may be some movement on it, however no guarantees or commitments at this time.  

In regards to the alternative as a crash consistent backup, yes it would be safe as long as the databases, WIJ, and journals are all included and have a consistent point-in-time snapshot.  The databases in the backup archive may be "corrupt", and not until after starting Caché for the WIJ and journals to be applied will it be physically accurate.  Just like you said - a crash consistent backup and the WIJ recovery is key to the successful recovery.  

I will post back if I hear of changes coming with VSS integration.

Hi Dean - thanks for the comment.  There are no changes required from a Caché standpoint, however Microsoft would need to add the similar functionality to Windows to allow for Azure Backup to call a script within the target Windows VM similar to how it is done with Linux.  The scripting from Caché would be exactly the same on Windows except for using .BAT syntax rather then Linux shell scripting once Microsoft provides that capability.  Microsoft may already have it this capability?  I'll have to look to see if they have extended it to Windows as well.

Mark B-

Hi Raymond,

Thank you for your question.  I can help with your question.  We have done a lot of testing with EC2, and the performance of an EC2 instance will vary based on an on-demand or reserved instances even of the same EC2 instance type.  In AWS a given EC2 instance type's reported number of vCPU is an individual thread on the processor as a "logical processor".  The OS (and Ensemble/HealthShare as well for that matter) will only see a given instance's number of vCPUs, and the OS will only schedule jobs on those as it sees them. Ensemble and HealthShare are process based - not thread based, so for an instance type of m4.large with 4 vCPUs will mean only 4 jobs in parallel will execute as a time.

In your specific case with the large amount of XSLT parsing and adjusting pool sizes, you will want to first determine if FIFO is a requirement, if so, then unfortunately you need to remain at a pool size of 1 to ensure FIFO.  However, if FIFO is not required in your production or a given Business Service/Process/Operation, you can adjust the pool sizes to values higher than 1 to manage the message queues.  Having a large pool size won't impact the performance or a single XSTL parse, however it will allow for more parallel messages and XSLT parsing.  If you see CPU utilization at 100% and the message queues continual grow, you may need a large EC2 instance type (and larger pool size) to accommodate the message rates.

I hope this helps.  

Kind regards,

Mark B-

Hi Mack,

I can help here.  The VMS/Itanium system you are migrating from is quite old, and has quite slow processors.  For something like this you can figure at least 4 of the McKinley cores (maybe more) to 1 single current model Intel Xeon E5 v4 series core.  I would look to using a server such as a single-socket system with an Intel Xeon E5-2667v4 processor and 64GB of RAM (more RAM doesn't hurt either).  The E5-2667v4 processor is a 8-core processor @ 3.2Ghz each which is far more CPU than you would need, however it's actually quite difficult to get a smaller server theses.  

For a workload like this, a virtual machine in vSphere, Hyper-V, or KVM would probably be more appropriate.

Also, I have a few comments on your current Caché configuration:

  • The amount of routine buffers configured you have configured (3584MB) exceeds the maximum allowed (max is only 1023MB).  You can confirm in your cconsole.log that startup actually reduced to the max value.  You will want to update your routine cache size to 1023MB so that it takes effect on the next Caché restart.
  • I see you have 512MB of 2KB database buffers allocated and 43496MB of 8KB buffers.  I would suggest removing the allocation of the 2KB buffers completely and just allow any 2KB databases you have to use the 8KB buffers.  That way you aren't artificially capping your database cache.
  • Speaking of 2KB databases, If you still actually have 2KB databases on your system, it is highly recommended to convert those to 8KB databases for data safety and performance reasons.  

Kind regards,

Mark B-

I will revise the post to be more clear that THP is enabled by default in 2.6.38 kernel but may be available in prior kernels and to reference your respective Linux distributions documentation for confirming and changing the setting.  Thanks for your comments.

Hi Alexander,

Thank you for you post.  We are only relying on what RH documentation is stating as to when THP was introduced to the main stream kernel (2.6.38) and enabled by default as noted in the RH post you referenced.  The option may have existed in previous kernels (although I would not recommending to try it), it may not have been enabled by default.  All the documentation I can find on THP support in RH references the 2.6.38 kernel where is was merged feature.

If you are finding it in previous kernels, confirm that THP are enabled by default or not.  That would be interesting to know.  Unfortunately there isn't much we can do other than to do the checks for enablement as mentioned in the post.  As the ultimate confirmation, RH and the other Linux distributions would need to update their documentation to confirm when this behavior was enacted in the respective kernel versions.  

As I mentioned in other comments, the use of THP is not necessarily a bad thing and won't cause "harm" to a system, but there may be performance impacts for applications that have a large amount of process creation as part of their application.

Kind regards,

Mark B-

Hi Alexey,

Thank you for your comment.  Yes, both THP and traditional/reserved Huge_pages can be used at the same time, however there is not benefit and in fact systems with many (thousands) of Caché processes, especially if there is a lot of process creation, has shown a performance penalty in testing.  The overhead of instantiating the THP for those processes at a high rate can be noticeable.  Your application may not exhibit this scenario and may be ok.  

The goal of this article is to provide guidance for those that may not know which is the best option to choose and/or point out that this is a change in recent Linux distributions.  You may find that THP usage is perfectly fine for your application.  There is no replacement for actual testing and benchmarking your application.  :)

Kind regards,

Mark B-

Hi Anzelem,

Here are the steps that need to be defined in your VCS cluster resource group with dependencies.

  • Remount the storage <— this is not new
  • Relocate the cluster IP <— this is not new
  • Simple VCS application/script agent to restart the ISC Agent < — THIS IS NEW
  • ISC VCS cluster agent to start Caché < — this is not new (make the previous step a dependency before executing)

The script to start the ISCAgent would be dependent on the storage being mounted in the first step. 

This should provide you with the full automation needed here.  Let me know if there any any concerns or problems with the above steps.


Mark B-

Thank you for your question.  It is recommended with any InterSystems 2014.1 product (including Caché, Ensemble, or HealthShare) version to remain using SMT4 (or SMT2).  Not until running a version based on 2015.1 or higher would SMT8 be advisable and provide any potential gain. 

Thank you for your comment.  You will need to establish you own monitoring and ultimately range of IO response times for your application using tools like iostat.  This article is used to give you a starting point for monitoring.  Your specific application may need higher or lower requirements.   

Using iostat, you want to continuously monitor storage device performance (specifically the iostat -x <device> <time between sample in seconds> <number of iterations> command) and monitor it for a particular range of time.  For example, if you want to only monitor during peak business hours from 8am-12pm.  What is mostly important is average response times - typically I like using iostat -x <devices> 2 1000 to report 1000 2-second samples.  This is useful when diagnosing a performance issue.  

To reduce the amount of data collected you can use a higher time between samples such as iostat -x <devices> 5 1000 for 5 second samples or even higher if you wish.  It's really a function of what reasons you are monitoring - if doing an in-depth performance analysis you would want a small time between samples to better observe spikes in response times, or if you are doing just daily statistic collection you could go for a higher time between samples.  The objective here is to get familiar with your specific application's needs and this article just provides a baseline for what is typical for most applications.

Kind regards,

Mark B-