New post
Question
Michael Jobe · Jan 26, 2021

Bug in SAM Prometheus metrics endpoint

#Monitoring #InterSystems IRIS

The current version of SAM creates Prometheus metric endpoints which appear to be handled correctly by the current prometheus scraper, however the metrics do not confirm to the current prometheus standard.  The standard states: 

  • Prometheus' text-based format is line oriented. Lines are separated by a line feed character (\n). The last line must end with a line feed character. Empty lines are ignored.

Link is here: Prometheus format

In the current output on the

<hostname>:8080/api/monitor/metrics

endpoint the last metric does not contain a newline causing parsers to fail.  Adding a newline to this should correct the issues.

Product version: IRIS 2020.1
0
0 307
Discussion (9)1
Log in or sign up to continue
Michael Jobe · Jan 26, 2021

is this the right place to post this?  I'm looking for assistance and want to make sure I am posting in the right place.  Thanks.

0 0
Luca Ravazzolo · Jan 27, 2021

Thanks @Michael Jobe - I thought we implemented as described as we have had people using it in AWS with CloudWatch and others and I have not heard any issue before. Let me verify this.

0 0
Michael Jobe · Jan 31, 2021

Any thoughts?  I'm pretty sure that it is not up to spec, although some parsers will be fine.  To be clear there needs to be a newline following each key value pairing per the Prometheus spec. 

0 0
Luca Ravazzolo · Feb 1, 2021

Hi @Michael Jobe that last "\n" was indeed missing. We have fixed it and are re-spinning...

Thanks for your diligence.

May I ask you what product(s) you are using and how this caused you issues?

Thanks

0 0
Michael Jobe · Feb 1, 2021

Great!  Thanks for the update, I will wait for the update and test.  The product I am using is the New Relic Prometheus Open Metric Integration.  I work for New Relic and I have run across several customers leveraging Cache or IRISand wanted to have a solution to ingest the metrics needed for Cache/IRIS observability.

1 0
Luca Ravazzolo · Feb 1, 2021

@Michael Jobe thanks for highlighting this issue and for being candid. I've always liked your product. I hope our customers will be able to leverage your tool without any issue from now on. Thanks again.

0 0
Michael Jobe · Feb 3, 2021

Absolutely, as I am seeing more Cache' and IRIS instances pop up in customers, I'm excited to explore this more.  Feel free to reach out I'm happy to test or be a guinea pig on anything.  More than happy to help or collaborate anyway I can :). BTW, I'm seeing lots of community solutions for Cache' / Prometheus but it seems the SAM docs only refer to IRIS, is it not planned for Cache'?

0 0
Luca Ravazzolo · Feb 4, 2021

Hi @Michael Jobe,

Thanks for your availability. We are working on SAM v2.0 and we would appreciate your feedback.

SAM is not planned to be backported to support Caché at the moment. Many customers have already migrated to IRIS and many are planning to do so.

0 0