Performance

Syndicate content 11 

This post will show you an approach to size shared memory requirements for database applications running on InterSystems data platforms including global and routine buffers, gmheap, and locksize as well as some performance tips you should consider when configuring servers and when virtualizing Caché applications. As ever when I talk about Caché I mean all the data platform (Ensemble, HealthShare, iKnow and Caché).


A list of other posts in this series is here

Last comment 22 August 2017
+ 18   2 11
5072

views

+ 18

rating

Your application is deployed and everything is running fine. Great, hi-five! Then out of the blue the phone starts to ring off the hook – it’s users complaining that the application is sometimes ‘slow’. But what does that mean? Sometimes? What tools do you have and what statistics should you be looking at to find and resolve this slowness? Is your system infrastructure up to the task of the user load? What infrastructure design questions should you have asked before you went into production? How can you capacity plan for new hardware with confidence and without over-spec'ing? How can you stop the phone ringing? How could you have stopped it ringing in the first place?

Last comment 30 September 2016
+ 17   1 18
2149

views

+ 17

rating

In the last post we scheduled 24-hour collections of performance metrics using pButtons. In this post we are going to be looking at a few of the key metrics that are being collected and how they relate to the underlying system hardware. We will also start to explore the relationship between Caché (or any of the InterSystems Data Platforms) metrics and system metrics. And how you can use these metrics to understand the daily beat rate of your systems and diagnose performance problems.

Last comment 24 March 2018
+ 15   0 16
2013

views

+ 15

rating

Globals, these magic swords for storing data, have been around for a while, but not many people can use them efficiently or know about this super-weapon altogether.

If you use globals for tasks where they truly shine, the results may be amazing, either in terms of increased performance or dramatic simplification of the overall solution (1, 2).

Globals offer a special way of storing and processing data, which is completely different from SQL tables. They were first introduced in 1966 in the M(UMPS) programming language, which was initially used in medical databases. It is still used in the same way, but has also been adopted by some other industries where reliability and high performance are top priorities: finance, trading, etc.

Later M(UMPS) evolved into Caché ObjectScript (COS). COS was developed by InterSystems as a superset of M. The original language is still accepted by developers' community and alive in a few implementations. There are several signs of activity around the web: MUMPS Google group, Mumps User's group), effective ISO Standard, etc.

Modern global based DBMS supports transactions, journaling, replication, partitioning. It means that they can be used for building modern, reliable and fast distributed systems.

Globals do not restrict you to the boundaries of the relational model. They give you the freedom of creating data structures optimized for particular tasks. For many applications reasonable use of globals can be a real silver bullet offering speeds that developers of conventional relational applications can only dream of.

Globals as a method of storing data can be used in many modern programming languages, both high- and low-level. Therefore, this article will focus specifically on globals and not the language they once came from.

Last comment 31 July 2017
+ 12   0 4
1224

views

+ 12

rating

This week I am going to look at CPU, one of the primary hardware food groups :) A customer asked me to advise on the following scenario; Their production servers are approaching end of life and its time for a hardware refresh. They are also thinking of consolidating servers by virtualising and want to right-size capacity either bare-metal or virtualized. Today we will look at CPU, in later posts I will explain the approach for right-sizing other key food groups - memory and IO.

So the questions are

Last comment 7 June 2017
+ 12   0 4
2702

views

+ 12

rating

There are often questions surrounding the ideal Apache HTTPD Web Server configuration for HealthShare.  The contents of this article will outline the initial recommended web server configuration for any HealthShare product. 

As a starting point, Apache HTTPD version 2.4.x (64-bit) is recommended.  Earlier versions such as 2.2.x are available, however version 2.2 is not recommended for performance and scalability of HealthShare.

+ 12   5 3
0

comments

404

views

+ 12

rating

In previous posts I have shown how it is possible to collect historical performance metrics using pButtons. I go to pButtons first because I know it is installed with every Data Platforms instance (Ensemble, Caché, …). However there are other ways to collect, process and display Caché performance metrics in real time either for simple monitoring or more importantly for much more sophisticated operational analytics and capacity planning. One of the most common methods of data collection is to use SNMP (Simple Network Management Protocol). SNMP a standard way for Caché to provide management and monitoring information to a wide variety of management tools. The Caché online documentation includes details of the interface between Caché and SNMP. While SNMP should 'just work' with Caché there are some configuration tricks and traps.

Last comment 17 March 2017
+ 9   1 2
2113

views

+ 9

rating

Myself and the other Technology Architects often have to explain to customers and vendors Caché IO requirements and the way that Caché applications will use storage systems. The following tables are useful when explaining typical Caché IO profile and requirements for a transactional database application with customers and vendors.  The original tables were created by Mark Bolinsky.

In future posts I will be discussing more about storage IO so am also posting these tables now as a reference for those articles. 



A list of other posts in this series is here
 


It is vital to have well set up storage such as a storage array to provide predictable disk IO performance, support high availability features and provide storage redundancy, scalability, and reliability for your applications

Last comment 13 November 2016
+ 8   1 2
1290

views

+ 8

rating

Note (Sept 2018): There have been big changes since this post first appeared, I suggest using the Docker Container version, the project and details for running as a container are still in the same place  published on GitHub so you can download, run - and modify if you need to

Last comment 11 May 2017
+ 8   1 5
882

views

+ 8

rating

Hyper-Converged Infrastructure (HCI) solutions have been gaining traction for the last few years with the number of deployments now increasing rapidly. IT decision makers are considering HCI when scoping new deployments or hardware refreshes especially for applications already virtualised on VMware. Reasons for choosing HCI include; dealing with a single vendor, validated interoperability between all hardware and software components, high performance especially IO, simple scalability by addition of hosts, simplified deployment and simplified management

Last comment 16 August 2017
+ 7   0 6
1804

views

+ 7

rating

A short post for now to answer a question that came up. In post two of this series I included graphs of performance data extracted from pButtons. I was asked off-line if there is a quicker way than cut/paste to extract metrics for mgstat etc from a pButtons .html file for easy charting in Excel.

See: - Part 2 - Looking at the metrics we collected

pButtons compiles data it collects into a single html file to make it easier to send to WRC and review the collated data. However, especially for pButtons run over long collection times like 24 hours, some of the time based data like mgstat, vmstat etc is easier to review graphically to look for trends or patterns

Last comment 4 November 2016
+ 7   0 5
837

views

+ 7

rating

No doubt bitmap indexing, if used with a suitable property, performs just impressive!
But there is a major limit: ID key has to be a positive integer.
For modern class definitions working with CacheStorage this is a default.

BUT: There are hundreds (thousands ?) old applications out in the field that
are still using composite ID keys.
Or - to name the origin - work on Globals with 2 subscript levels (or more).
They are by construction excluded from our "Bitmap Wonderland".

Of course, using the feature of multiple storage maps could possibly allow to escape - somehow.
But in those cases where you have many many GB of data stored in that way and
where you work with millions of lines of old  - often poorly documented - code.
You wouldn't think seriously more than 5 min. if you change the storage structure or not.
So the bitmap is not with you

Last comment 17 February 2018
+ 7   0 5
357

views

+ 7

rating

ˮ This is one of my articles which was never published in English. Let's fix it!

Hello! This article is about quite a practical way of developing InterSystems solutions without using the integrated tools like Studio or Atelier. All the code of the project can be stored in the form of "traditional" source code files, edited in your favorite development environment (for example, Visual Studio Code), indexed by any version control system and arbitrarily combined with many external tools for code analysis, preprocessing, packaging and so on

Last comment 2 March 2019
+ 6   5 3
719

views

+ 6

rating

Beginning - see Part 1.

 

3. Variants of structures when using globals

 

A structure, such as an ordered tree, has various special cases. Let's take a look at those that have practical value for working with globals.

 

 

 

 

3.1 Special case 1. One node without branches

 

Last comment 8 July 2017
+ 6   0 5
739

views

+ 6

rating

In the previous parts (1, 2) we talked about globals as trees. In this article, we will look at them as sparse arrays.

A sparse array - is a type of array where most values assume an identical value.

In practice, you will often see sparse arrays so huge that there is no point in occupying memory with identical elements. Therefore, it makes sense to organize sparse arrays in such a way that memory is not wasted on storing duplicate values.

In some programming languages, sparse arrays are part of the language - for example, in J, MATLAB. In other languages, there are special libraries that let you use them. For C++, those would be Eigen and the like.

Globals are good candidates for implementing sparse arrays for the following reasons:

Last comment 17 July 2017
+ 6   0 5
532

views

+ 6

rating

Quotes (1NF/2NF/3NF)ru:

Every row-and-column intersection contains exactly one value from the applicable domain (and nothing else).
The same value can be atomic or non-atomic depending on the purpose of this value. For example, “4286” can be
  • atomic, if its denotes “a credit card’s PIN code” (if it’s broken down or reshuffled, it is of no use any longer)
  • non-atomic, if it’s just a “sequence of numbers” (the value still makes sense if broken down into several parts or reshuffled)

This article explores the standard methods of increasing the performance of SQL queries involving the following types of fields: string, date, simple list (in the $LB format), "list of <...>" and "array of <...>".

+ 6   0 3
0

comments

361

views

+ 6

rating

Using the CSP Page Statistics

Application Performance Management

Introduction

A key part of Application Performance Management (APM) is recording the activity and performance of user activity. For many web applications the closest you can get to this is to record the CSP pages or CSP based services being dispatched.

If the pages or service names are meaningful and they indicate the business activity being performed the CSP page statistics can be very useful in building up a historical record of activity, performance and resource usage. This allows you to recognize trends in activity for planning purposes. It allows you to see gradually degrading performance or increasing resource usage so you can take action before you have a crisis. Also, if you do have a performance crisis you can look back to see what has changed since the performance was fine

Last comment 11 November 2017
+ 5   0 2
312

views

+ 5

rating

It has been noticed that some customers running JAVA programs (for example, FOP) on AIX would see the server eventually running low then out of memory. Customer would notice the system pages heavily and user experience becomes bad. And the server would crash when out of memory.

 

When the problem happens, we can see in ipcs a lot of shared memory segment marked for deletion (Capital D at the beginning of MODE section). This means they will not disappear until the last process attached to the segment detaches it.

 

The problem can be easily recreated using the following command from UNIX prompt on a server running AIX 7, for example:

 

/usr/java6_64/jre/bin/java -Xrs -classpath /scratch3/gcooper/ggc170refp750/dev/java/lib/JDK16/cachegateway.jar:/scratch3/gcooper/ggc170refp750/dev/java/lib/JDK16/cachejdbc.jar:/usr/java6_64/jre/bin com.intersys.gateway.JavaGateway 62001 2>&1

+ 5   1 1
0

comments

874

views

+ 5

rating

In this short article we talk about how to get Yape running in a docker container to avoid having to setup python on your machine.

It's been a while since the last article in this series, so let's recap quickly.

We talked about using matplotlib to create a basic graph. Afterwards we introduced dynamic graphs using bokeh.
In the 3rd part we talked about generating heatmaps using monlbl data

Last comment 7 August 2017
+ 5   0 3
433

views

+ 5

rating

In last week's discussion we created a simple graph based on the data input from one file. Now, as we all know, sometimes we have multiple different datafiles to parse and correlate. So this week we are going to load additional perfmon data and learn how to plot that into the same graph.
Since we might want to use our generated graphs in reports or on a webpage, we'll also look into ways to export the generated graphs

+ 5   0 3
0

comments

494

views

+ 5

rating

There are three things most important to any SQL performance conversation:  Indices, TuneTable, and Show Plan.  The attached PDFs includes historical presentations on these topics that cover the basics of these 3 things in one place.  Our documentation provides more detail on these and other SQL Performance topics in the links below.  The eLearning options reinforces several of these topics.  In addition, there are several Developer Community articles which touch on SQL performance, and those relevant links are also listed.

There is a fair amount of repetition in the information listed below.  The most important aspects of SQL performance to consider are:

  1. The types of indices available
  2. Using one index type over another
  3. The information TuneTable gathers for a table and what it means to the Optimizer
  4. How to read a Show Plan to better understand if a query is good or bad
Last comment 18 January 2019
+ 5   2 2
240

views

+ 5

rating

This is a follow-up to Murray's article for scripting pButtons data extraction on Unix/MacOS:
    Extracting pButtons data to CSV for UNIX/MacOS

Extracting pButtons data to a CSV file on Windows

PowerShell scripting was used in this article because it is available by default on Windows platforms.

The script, Extract_pButtons.ps1, will search the given pButtons file until it finds the marker for the beginning of the section. The section will be printed line-by-line, from the header to the line before the section end marker

+ 4   0 4
0

comments

401

views

+ 4

rating

I was benchmarking the populate utils when I noticed something strange. Consider this simple benchmarking method:

ClassMethod runBench(count As %Integer = 300000)
{
        s types=["name","ssn","company","title","phone","city","street","zip","mission","state","color","product","string","integer","float"]
        //s types=["name","ssn"]
        s iter=types.%GetIterator()
        while iter.%GetNext(.key,.value){
            s start=$P($ZTS,",",2)
            s opt={}
            s opt.type=value
            for i=1:1:count{
                s res=##class(DataGen.Generator).genVal(opt)            }
            s end=$P($ZTS,",",2)
            s total=end-start
            w value_$C(9)_" total time:"_total_$C(9)_" rate:  "_$NORMALIZE(count/total,2)_" per second",!
        }
}

With my genVal method being this

+ 4   0 2
0

comments

256

views

+ 4

rating

practical guide to using the tools PERFMON and MONLBL.

Introduction

When investigating performance problems, I often use the utilities ^PERFMON and ^%SYS.MONLBL to identify exactly where in the application pieces of code are taking a long time to execute. In this short paper I will describe an approach that first uses ^PERFMON to identify the busiest routines and then uses ^%SYS.MONLBL to analyze those routines in detail to show which lines are the most expensive.

The details of ^PERFMON and ^%SYS.MONLBL are individually described in detail in the Caché documentation, but here I summarize when, why and how you can use them together to find the cause of a performance problem in an application

Last comment 29 December 2017
+ 4   0 11
547

views

+ 4

rating

Continuing on with providing some examples of various storage technologies and their performance profiles, this time we looked at the growing trend of leveraging internal commodity-based server storage, specifically the new HPE Cloudline 3150 Gen10 AMD processor-based single socket servers with two 3.2TB Samsung  PM1725a NVMe drives.  

Last comment 29 October 2018
+ 4   0 2
330

views

+ 4

rating