Question
Michael Jobe · Jan 26, 2021

Bug in SAM Prometheus metrics endpoint

The current version of SAM creates Prometheus metric endpoints which appear to be handled correctly by the current prometheus scraper, however the metrics do not confirm to the current prometheus standard.  The standard states: 

  • Prometheus' text-based format is line oriented. Lines are separated by a line feed character (\n). The last line must end with a line feed character. Empty lines are ignored.

Link is here: Prometheus format

In the current output on the

<hostname>:8080/api/monitor/metrics

endpoint the last metric does not contain a newline causing parsers to fail.  Adding a newline to this should correct the issues.

Product version: IRIS 2020.1
0
0 307
Discussion (9)1
Log in or sign up to continue

is this the right place to post this?  I'm looking for assistance and want to make sure I am posting in the right place.  Thanks.

Thanks @Michael Jobe - I thought we implemented as described as we have had people using it in AWS with CloudWatch and others and I have not heard any issue before. Let me verify this.

Any thoughts?  I'm pretty sure that it is not up to spec, although some parsers will be fine.  To be clear there needs to be a newline following each key value pairing per the Prometheus spec. 

Hi @Michael Jobe that last "\n" was indeed missing. We have fixed it and are re-spinning...

Thanks for your diligence.

May I ask you what product(s) you are using and how this caused you issues?

Thanks

Great!  Thanks for the update, I will wait for the update and test.  The product I am using is the New Relic Prometheus Open Metric Integration.  I work for New Relic and I have run across several customers leveraging Cache or IRISand wanted to have a solution to ingest the metrics needed for Cache/IRIS observability.

@Michael Jobe thanks for highlighting this issue and for being candid. I've always liked your product. I hope our customers will be able to leverage your tool without any issue from now on. Thanks again.

Absolutely, as I am seeing more Cache' and IRIS instances pop up in customers, I'm excited to explore this more.  Feel free to reach out I'm happy to test or be a guinea pig on anything.  More than happy to help or collaborate anyway I can :). BTW, I'm seeing lots of community solutions for Cache' / Prometheus but it seems the SAM docs only refer to IRIS, is it not planned for Cache'?

Hi @Michael Jobe,

Thanks for your availability. We are working on SAM v2.0 and we would appreciate your feedback.

SAM is not planned to be backported to support Caché at the moment. Many customers have already migrated to IRIS and many are planning to do so.