#Monitoring

So if you are following from the previous post or dropping in now, let's segway to the world of eBPF applications and take a look at Parca, which builds on our brief investigation of performance bottlenecks using eBPF, but puts a killer app on top of your cluster to monitor all your iris workloads, continually, cluster wide!  

Continous Profiling with Parca, IRIS Workloads Cluster Wide

I attended Cloud Native Security Con in Seattle with full intention of crushing OTEL day, then perusing the subject of security applied to Cloud Native workloads the following days leading up to CTF as a professional excercise. This was happily upended by a new understanding of eBPF, which got my screens, career, workloads, and atitude a much needed upgrade with new approaches to solving workload problems. 

So I made it to the eBPF party and have been attending clinic after clinic on the subject ever since, here I would like to "unbox" eBPF as a technical solution, mapped directly to what we do in practice (even if its a bit off), and step through eBPF through my experimentation on supporting InterSystems IRIS Workloads, particularly on Kubernetes, but not necessarily void on standalone workloads.

eBee Steps with eBPF and InterSystems IRIS Workloads

We are constantly running into issues where there are billions of Orphaned messages in our system that cause problems, and we have to manually run a cleanup to fix performance issues.

 In the following article about orphaned messages... https://community.intersystems.com/post/ensemble-orphaned-messages it mentions either programmatically eliminating the Orphaned messages or using a Utility like Demo.Util.CleanupSet in ENSDEMO.

Welcome to the next chapter of my CI/CD series, where we discuss possible approaches toward software development with InterSystems technologies and GitLab. Today, we continue talking about Interoperability, specifically monitoring your Interoperability deployments. If you haven't yet, set up Alerting for all your Interoperability productions to get alerts about errors and production state in general.

Inactivity Timeout is a setting common to all Interoperability Business Hosts. A business host has an Inactive status after it has not received any messages within the number of seconds specified by the Inactivity Timeout field. The production Monitor Service periodically reviews the status of business services and business operations within the production and marks the item as Inactive if it has not done anything within the Inactivity Timeout period. The default value is 0 (zero). If this setting is 0, the business host will never be marked Inactive, no matter how long it stands idle.

Hi Community, 

Watch this video to learn about the Monitoring and Alerting Capabilities of InterSystems IRIS.

🗣  Presenter: @Mark Bolinsky, Principal Technology Architect, InterSystems

This demo was prepared for one of our past online developer roundtables. We encourage you to ask your specific questions about this topic in the comments section, and we will invite our experts to answer them!

Useful Links:

 
Have written several worthless Conky's in my day, and this one is no exception, but it was fun.

Hello developers, 

Our project was designed to optimize patient clinical outcomes by reducing hospitalization time and supporting the development of resident and novice physicians. Additionally, it contributes to lowering financial waste in the healthcare system by improving the monitoring of pregnant patients, thereby decreasing risks and enhancing their safety.

Using the most accessible tool, the smartphone, was the obvious choice to make patients' lives easier.

Howdy all,

I'm very new to intersystems, therefore I'm seeing some suggestions here. 
My query -  I'd like to setup monitoring for IRIS local database size monitoring using SolarWinds. Preferably, using REST API connections, could someone help with this. Thank you in advance. 

Hi Developers,

Join us at the upcoming Developer Roundtable on April 25th at 9 am ET | 3 pm CET. 📍
We will have 2 topics covered by the invited experts and open discussion as always.

Tech Talks:
➡ Practical Usage of Embedded Python - by Stefan Wittmann Product Manager, InterSystems

▶ Recording: 

Spoilers: Daily Integrity Checks are not only a best practice, but they also provide a snapshot of global sizes and density. 
Update 2024-04-16:  As of IRIS 2024.1, Many of the below utilities now offer a mode to estimate the size with <2% error on average with orders of magnitude improvements in performance and IO requirements. I continue to urge regular Integrity Checks, however there are situations where more urgent answers are needed.

• EstimatedSize^%GSIZE- Runs %GSIZE in estimation mode.   
• ##class(%Library.GlobalEdit).GetGlobalSize(directory, globalname, .allocated, .used.

With System Alerting and Monitoring (SAM) being deprecated in the near future..

• What is everyone's go-to for Monitoring IRIS? 
• What is readily available?
• What is the cost surrounding it?

Just trying to get ideas floating around of what we might need to start looking at to satisfy IT leadership.

Thanks

Scott

In The Business Process and the Business Operation, I am using the following code to get the value of TimeCreated and TimeProcessed 

BP:

%Ensemble("Process").%PrimaryRequestHeader.TimeCreated

%Ensemble("Process").%PrimaryRequestHeader.TimeProcessed

BO:

..%RequestHeader.TimeCreated

..%RequestHeader.TimeProcessed

But I am trying to use ..%RequestHeader.TimeCreated in Business Service it is not storing any value.

How to get the value of TimeCreated and TimeProcessed in Business Service?

In the Business Operation, how do we get to know which source is currently sending primary request if there are multiple requests coming at the same time?

Hi Developers,

Our first Online Developer Roundtable of 2024 will take place on March 5th at 9 am ET | 3 pm CET. 

Tech talks:

1. ObjectScript Unit Testing Tools, Techniques and Best Practices - by @Timothy Leavitt , Development Manager, Application Services, InterSystems
2. Monitoring and Alerting Capabilities of InterSystems IRIS - by  @Mark Bolinsky, Principal Technology Architect, InterSystems Mark's presentation is rescheduled for the roundtable in April.

▶ Update: watch the recording of the roundtable below:

Hi developers!

Today I would like to address a subject that has given me a hard time. I am sure this must have been the case for quite a number of you already (so-called “the bottleneck”). Since this is a broad topic, this article will only focus on identifying incoming HTTP requests that could be causing slowness issues. I will also provide you with a small tool I have developed to help identify them.

Our software is becoming more and more complex, processing a large number of requests from different sources, be it front-end or third-party back-end applications. To ensure optimal performance, it is essential to have a logging system capable of taking a few key measurements, such as the response time, the number of global references and the number of lines of code executed for each HTTP response. As part of my work, I get involved in the development of EMR software as well as incident analysis.  Since user load comes mostly from HTTP requests (REST API or CSP application), the need to have this type of measurement when generalized slowness issues occur has become obvious.

Hello,

In response to the infrastructure needs of our company's service, I've created a small API that sends SNMP queries to InterSystems to visualize relevant data for retrieval when the infrastructure implements monitoring.

However, I'm experiencing a timeout issue when attempting to collect information using an SNMP walk. Here is the code for my API's SNMP service:

importfrom"net-snmp"()

I do have the 'session created' console.log, so I am able to successfully create the SNMP session. However, I encounter this error: 'error: RequestTimedOutError: Request timed out.

I tried to monitoring IRIS with Zabbbix Server. Snmpwalk from Zabbix to IRIS success and got data from IRIS. But in message.log appears many warning message if snmpwalk running. 

111823"3"

I tried set ^SYS("MONITOR","SNMP","DEBUG")=1, but nothing anomaly  in log SNMP.log

Could you help me please. Thanks in advance

I need to develop a tool to help to get what data is being consumed by a certain process, in order to get all data used to build an automated test scenario.

For example, some user process will pull data from ^GLOBAL(1)="dataString", ^GLOBAL(2)="dataString2", ^GLOBAL1(1)="data1String", ^GLOBAL2(4)="data2String4". Amidst all other data on these Globals, I will ignore everything that was not used in the user process, and get the specific keys used on it.

It appears to be working correctly, mirroring data etc. but the status on the Primary Cache DB is showing the wrong status for the DR DB.

 re-configured Cache Mirroring BUt  still have the same issue afterwards.

The GUI display remains the same regardless.

What's the way to get healthcheck info for an IRIS (container) running without PWS?

Is there any built-in functionality to send out an email or a text message when a mirror member fails?

Hello InterSystems Community Members,

I hope you are all doing well. I am currently facing an issue while trying to set up the SNMP subagent functionality for my InterSystems Cache installation.

I am using InterSystems Cache for Windows (AMD64) version 5.2.4 (Build 809_0_9006U). The SNMP subagent functionality requires the iscsnmp.dll dynamic library, which I have been unable to locate in my installation directory.

I have checked my installation directory thoroughly, typically located at C:\InterSystems\Cache, and its subdirectories, but I couldn't find the iscsnmp.dll file.

As the iscsnmp.dll

When installing IRIS, all the system AUDIT events are not enabled.

What is the fastest way to activate all events?

  System > Security Management > System Audit Events

Hello, 

Our team is working on building dashboard for internal reference and monitoring. 

We would like to have details like Interface Name, Current Status, Last Messages Processed at, IP & Port, Serve/Instance/Production Environment name etc. 

If there is any built-in service which we can utilize or any pre-compiled code that we can utilize to build such dashboard. 

At this moment want to keep it basic, but moving forward will enhance with more advance features. 

Please suggest, any help will be appreciated. 

Thanks,

Yash 

In the Windows Ressource Manager I can observe multiple parallel processes coming from cache.exe with read operations to journaling files.

All except one of these processes have the same reads(Byte/s). The processes point to different journal files and constantly read between 200 and 3000 Bytes/s.

The corresponding process via PID in the management portal of Caché shows the process %SYS.Monitor.Control.1. In 3 days of uptime on the server it has run 181.632.583 commands and modified 32.140.642 globals. 

These processes are not writing to the journal at any point.

My employer set up a web-based HL7 interface monitor dashboard that will display all Ensemble components (Service/Process/Operation) in a Production, their status, and the support information embedded in each interfaces listing on the Monitor.  Please see 3 screenshots.  

This is part of the URL that we go to when accessing this Web based Monitor:  ......57772/csp/healthshare/monitor/Rush.Monitor.Web.Home.cls

I've gone into Studio and I find the "Rush" 'folder' and the "Monitor" 'folder' but then the only thing to navigate to is "WebStudio" and I can never find the "Home.

I'm using the EnumerateJobStatus query of class Ens.Util.Statistics to obtain the LastActivity value of a Business Host.

I would expect that this would return the timestamp of the last message received by the BH, understanding that any connect/disconnect activity would reset that timer. However, the time returned appears to actually be the time at which Ens.MonitorService generated the alert and is not directly related to anything that happened in the BH itself.

Enterprise Monitor is a component of Ensemble and can help organizations monitor multiple productions running on different namespaces within the same instance or namespaces running on multiple instances.

Documentation can be found at:

http://docs.intersystems.com/ens20161/csp/docbook/DocBook.UI.Page.cls?KEY=EMONITOR_all#EMONITOR_enterprise

In Ensemble 2016.1 there were changes made to make this utility work with HealthShare environments.

This article will:

• Show how to set up Enterprise Monitor for HealthShare sites
• Show some features of Enterprise Monitor
• Show some features of Enterprise Message Viewer

For this article, I used the following version of HealthShare:

Cache for Windows (x86-64) 2016.1 (Build 656U) Fri Mar 11 2016 17:42:42 EST [HealthShare Modules:Core:14.02.2415 + Linkage Engine:14.02.2415 + Patient Index:14.02.2415 + Clinical Viewer:14.02.2415 + Active Analytics:14.02.2415]

InterSystems Production Monitor is a page in Management Portal to display monitoring information for the current running production. I like the page as it comes, but one size does not fit all.

On May 13, 2022 I read this post on Developer Community:

https://community.intersystems.com/post/creating-custom-monitoring-page

I agree with Mark that Production Monitor is complex. I liked the idea to create a nice clean clear monitor page.

I began to work on a Cache Server Page utilizing class methods to provide the production data.

I shared my first attempt with Mark.

Released with no formal announcement in IRIS preview release 2019.4 is the /api/monitor service exposing IRIS metrics in Prometheus format. Big news for anyone wanting to use IRIS metrics as part of their monitoring and alerting solution. The API is a component of the new IRIS System Alerting and Monitoring (SAM) solution that will be released in an upcoming version of IRIS.

However, you do not have to wait for SAM to start planning and trialling this API to monitor your IRIS instances.