Monitoring InterSystems IRIS with Prometheus and GrafanaContestant

Monitoring your IRIS deployment is crucial. With the deprecation of System Alert and Monitoring (SAM), a modern, scalable solution is necessary for real-time insights, early issue detection, and operational efficiency. This guide covers setting up Prometheus and Grafana in Kubernetes to monitor InterSystems IRIS effectively. 

This guide assumes you already have an IRIS cluster deployed using the InterSystems Kubernetes Operator (IKO), which simplifies deployment, integration and mangement.

Why Prometheus and Grafana?

Prometheus and Grafana are widely adopted tools for cloud-native monitoring and visualization. Here’s why they are a fit:

• Scalability: Prometheus handles large-scale data ingestion efficiently.
• Alerting: Customizable alerts via Prometheus Alertmanager.
• Visualization: Grafana offers rich, customizable dashboards for Kubernetes metrics.
• Ease of Integration: Seamlessly integrates with Kubernetes workloads.

Prerequisites

Before starting, ensure you have the following:

• Basic knowledge of Kubernetes and Linux
• kubectl and helm installed.
• Familiarity with Prometheus concepts (refer to the Prometheus documantion for more information).
• A deployed IRIS instance using the InterSystems Kubernetes Operator (IKO), refer to another article here.  

Step 1: Enable Metrics in InterSystems IRIS

InterSystems IRIS exposes metrics via /api/monitor/ in the Prometheus format. Ensure this endpoint is enabled:

1. Open the Management Portal.
2. Go to System Administration > Security > Applications > Web Applications.
3. Ensure /api/monitor/ is enabled and accessible by Prometheus. You can check its status by navigating to the Management Portal, going to System Administration > Security > Applications > Web Applications, and verifying that the endpoint is listed and enabled.

Verify its availability by accessing:

http://<IRIS_HOST>:<PORT>/api/monitor/metrics

Step 2: Deploy Prometheus Using Helm

Deploying Prometheus using Helm provides an easy-to-manage monitoring setup. We will use the kube-prometheus-stack chart that includes Prometheus, Alertmanager, and Grafana.

1. Prepare the configuration: Create a values.yaml file with the following settings:

   prometheus:
     prometheusSpec:
       additionalScrapeConfigs:
         - job_name: 'intersystems_iris_metrics'
           metrics_path: '/api/monitor/metrics'
           static_configs:
             - targets:
                 - 'iris-app-compute-0.iris-svc.commerce.svc.cluster.local:80' # Replace with your IRIS service
   
         # To scrape custom metrics from the REST API created in IRIS
         - job_name: 'custom_iris_metrics'
           metrics_path: '/web/metrics'
           static_configs:
             - targets:
                 - 'commerce-app-webgateway-0.iris-svc.commerce.svc.cluster.local:80'
           basic_auth:
             username: '_SYSTEM'
             password: 'SYS'

   • Explanation:
     • iris-app-compute-0.iris-svc.commerce.svc.cluster.local:80: The format of the target should follow this convention: <pod-name>-iris-svc.<namespace>.svc.cluster.local:80. Replace <pod-name> with your IRIS pod, specify whether you want to scrape compute or data pods, and adjust the namespace as needed.
     • basic_auth** section**: If authentication is required to access the IRIS metrics endpoint, provide the necessary credentials.
2. Add the Helm repository:

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update

    
3. Install Prometheus using Helm:

   helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring --create-namespace -f values.yaml

   
4. Verify the deployment:

   kubectl get pods -n monitoring

    

Step 3: Custom Metrics with REST API

You can create a custom metrics CSP page that serves your application metrics. In this guide, I provide an example of a simple CSP page that extracts system metrics from IRIS itself, but you can totally build your own CSP page with your own custom metrics—just make sure they are in the Prometheus format.

Class CustomMetrics.REST Extends %CSP.REST
{ Parameter HandleCorsRequest = 1; ClassMethod Metrics() As %Status
{
    Try {
        Do %response.SetHeader("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
        New $Namespace Set $Namespace = "%SYS"
        Set ref = ##class(SYS.Stats.Dashboard).Sample()
        Write "# HELP iris_license_high Peak number of licenses used", $CHAR(10)
        Write "# TYPE iris_license_high gauge", $CHAR(10)
        Write "iris_license_high ", ref.LicenseHigh, $CHAR(10)
        Write "# HELP iris_active_processes Number of active processes", $CHAR(10)
        Write "# TYPE iris_active_processes gauge", $CHAR(10)
        Write "iris_active_processes ", ref.Processes, $CHAR(10)
        Write "# HELP iris_application_errors Number of application errors", $CHAR(10)
        Write "# TYPE iris_application_errors counter", $CHAR(10)
        Write "iris_application_errors ", ref.ApplicationErrors, $CHAR(10)
        Return $$$OK
    } Catch ex {
        Do %response.SetHeader("Content-Type", "text/plain")
        Write "Internal Server Error", $CHAR(10)
        Do $System.Status.DisplayError(ex.AsStatus())
        Return $$$ERROR($$$GeneralError, "Internal Server Error")
    }
} 
XData UrlMap
{
<Routes>
    <Route Url="/metrics" Method="GET" Call="Metrics"/>
</Routes>
} 
}

Deploy this as a REST service under a new web application called metrics in IRIS, and add its path to Prometheus for scraping.

Step 4: Verify Prometheus Setup

1. Open the Prometheus UI (http://<PROMETHEUS_HOST>:9090).
2. Go to Status > Targets and confirm IRIS metrics are being scraped.

Step 5: Access Grafana

With Prometheus scraping IRIS metrics, the next step is to visualize the data using Grafana.

1. Retrieve the Grafana service details:

kubectl get svc -n monitoring

If you’re using an ingress controller, you can access Grafana using the configured hostname (e.g., http://grafana.example.com). Otherwise, you can use the following options:

1. Port Forwarding: Use kubectl port-forward to access Grafana locally:

   kubectl port-forward svc/monitoring-grafana -n monitoring 3000:80

   Then, access Grafana at http://localhost:3000.
2. NodePort or ClusterIP: Refer to the NodePort or ClusterIP service details from the command output to connect directly.

Step 6: Log In to Grafana

Use the default credentials to log in:

• Username: admin
• Password: prom-operator (or the password set during installation).

Step 7: Import a Custom Dashboard

I’ve created a custom dashboard specifically tailored for InterSystems IRIS metrics, which you can use as a starting point for your monitoring needs. The JSON file for this dashboard is hosted on GitHub for easy access and import: Download the Custom Dashboard JSON

To import the dashboard:

1. Navigate to Dashboards > Import in Grafana.
2. Paste the URL of the JSON file into the Import via panel JSON field or upload the file directly.
3. Assign the dashboard to a folder and Prometheus data source when prompted.

Once imported, you can edit the panels to include additional metrics, customize the visualizations, or refine the layout for better insights into your IRIS environment.

Conclusion

By following this guide, we've successfully set up Prometheus to scrape InterSystems IRIS metrics and visualize them using Grafana. Additionally, you can explore other monitoring tools such as Loki to also monitor logs efficiently and configure alerts using Alertmanager or external services like PagerDuty and Slack. If you have any questions or feedback, feel free to reach out!