Open Exchange App Yape - Yet another pButtons extractor (and automatically create charts)

Note (June 2019): A lot has changed, for the latest details go here

Note (Sept 2018): There have been big changes since this post first appeared, I suggest using the Docker Container version, the project and details for running as a container are still in the same place  published on GitHub so you can download, run - and modify if you need to.

Working with customers on performance reviews, capacity planning and trouble-shooting I am often unpacking and reviewing Caché and Operating System metrics from pButtons. Rather than wade through html files cutting and pasting sections to be graphed in excel  I published a post a while ago with a utility to unpack pButtons metrics  written using unix shell, perl and awk scripting. While this is a useful time-saver its not the full story… I also use scripts to automatically chart the metrics for quick review and inclusion in reports. However these charting scripts are not easily maintained and become especially messy when there is site specific configuration needed, for example list of disks for iostat or windows perfmon, so I never publicly published the graphing utilities. But I am happy to say there is a much simpler solution now.

Serendipity occurred when I was at a customer site looking at system performance with Fabian and he showed me what he was doing using useful Python modules for charting . This is a much more flexible and maintainable solution than the scripts I was using. And the ease of integrating Python modules for file management and charting, including interactive html you can share, means the output can be much more useful. Taking Fabians posts as a base I wrote Yape to quickly and simply extract then chart customer pButtons files in several formats. The project is  published on GitHub so you can download, run and modify if you need to.

Overview

At the moment this process has two steps.

Step 1. extract_pButtons.py

Extract interesting sections from pButtons and write to .csv files for opening with excel or processing with charting with graph_pButtons.py.

Step 2. graph_pButtons.py

Chart files created at step 1. Currently output can be line or dot charts as .png or asinteractive .html with options to pan, zoom, print, etc.

Readme.md on GitHub has details on how to set up and run the two python scripts and will be the most up to date reference.

Other Notes

As an example: Using the options for adding prefixes to the output and input directories you can easily walk a directory with a set (e.g. a weeks) pButtons html files and output to a separate directory for each pButtons file.

for i in `ls *.html`; do ./extract_pButtons.py $i -p ${i}_; done

for i in `ls *.html`; do ./graph_pButtons.py ./${i}_metrics -p ${i}_; done

In the short term while I continue with the series on Caché capacity planning and performance I will use the charts created with these utilities.

I have tested on OSX, but not on Windows. You should be able to install and run Python on windows, please leave feedback on your experience with Windows. I suspect you will have to make changes to the file path slashes for example.

Note: Until a few weeks ago I had never written anything in Python, so if you are a Python expert there may be some things in the code that are not best practice. However I use the scripts nearly every day so I will continue to make improvements. I expect my Python skills will improve — but please feel free to ‘educate’ me if you some see something that should be corrected!

If you find the scripts useful let me know and check back regularly to get new features and updates.

Comments

If you are watching this page; I have checked in some changes on GitHub to make it easier to chart multiple days e.g. a weeks worth of pButtons at a time.

Murray,

This looks great.  However I am having a problem getting this to work with a recent pButtons report I received.  The error is a follows.  Any thoughts?

docker container run --rm -v "$(pwd)":/data yape/yape --mgstat --vmstat -c /data/dbmirror2-c_CACHE_20190131_161057_24hours.html
INFO:root:Profile run "24hours" started at 16:10:57 on Jan 31 19.
Traceback (most recent call last):
  File "/usr/local/bin/yape", line 11, in <module>
    load_entry_point('yape', 'console_scripts', 'yape')()
  File "/src/yape/yape/command_line.py", line 6, in main
    yape.yape2()
  File "/src/yape/yape/main.py", line 248, in yape2
    parsepbuttons(args.pButtons_file_name, db)
  File "/src/yape/yape/parsepbuttons.py", line 382, in parsepbuttons
    StartDateStr = datetime.strptime(StartDateStr, "%b %d %Y").strftime(
  File "/usr/local/lib/python3.7/_strptime.py", line 577, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/local/lib/python3.7/_strptime.py", line 359, in _strptime
    (data_string, format))
ValueError: time data 'Jan 31 19' does not match format '%b %d %Y'
 

Looks like a locale problem.

Add to the beginning of the script:

import locale
locale.setlocale(locale.LC_ALL, 'en_US.utf8')

Hi Rich, the short story is that I have fixed this an pushed to GitHub. A new container version will appear very soon.

The problem is caused by an unexpected date format in the "Profile" section of the pButtons HTML file.  Year format yy instead of expected yyyy. 

As a workaround, you can open the HTML file in a text editor and edit the date manually to be the expected format. eg

Profile run "24hours_5" started at 00:00:00 on Jun 08 19.

change to:

Profile run "24hours_5" started at 00:00:00 on Jun 08 2019.

Thanks for bringing this to my attention.!