Question
· Feb 25, 2022

How to configure SAM to monitor containers or local instances ?

Monitoring IRIS through SAM (sam:1.0.0.115)

While I've used to make it work in the past, I'm not able any more to fix the already faced issue : "State: Unreachable"

Despite all the ip-addresses attempts (or hostname = host.docker.internal)  and while "/api/monitor/metrics" is running well...

 

Could someone provide good practices to make it work cleanly ?

Product version: IRIS 2022.1
$ZV: IRIS for UNIX (Ubuntu Server LTS for x86-64 Containers) 2022.1 (Build 114U) Mon Jan 31 2022 01:58:06 EST
Discussion (13)1
Log in or sign up to continue

It's not a big issue, to make it work. Just keep in mind that both containers, with SAM and monitored system, have to be in the same docker's network. If you run both images in with docker-compose, they by default will get the same network, so, in most cases, you may use IRIS service's name from docker-compose.yml as a hostname, and it will be able to connect to it internally inside that network.

If you need access from dockerized SAM to a dockerized IRIS, both working in different networks, you have to publish ports for IRIS, so, it should be available from the host, but from the docker, your SAM has to know the host's address and host.docker.internal is a way, to give it to him. And port should be the same as published when docker container is running.

When docker container is started with -p param, which says how to publish the port, possible to say, which IP to select, or select all  for instance -p 0.0.0.0:52273:52273

docker ps, may help with understanding how the container is available

8d8977f9d5f8   intersystemsdc/iris-community:preview                          "/tini -- /iris-main…"   6 days ago       Up 3 days (healthy)       0.0.0.0:1972->1972/tcp, 2188/tcp, 53773/tcp, 0.0.0.0:52773->52773/tcp, 54773/tcp                           iris

if it shows something like 127.0.0.1:52773->52773, it will only be available as localhost or 127.0.0.1 and will not be accessible from another container and network, because of host.docker.internal will refer to another ip

Could you explain, how you running all the parts?

Thanks Dmitry for your reply.

Actually, I know all of this ; that's why I don't understand why it's not working any more...

  1. gh repo clone intersystems-community/sam
  2. cd sam
  3. tar xvzf sam-1.0.0.115-unix.tar.gz
  4. cd sam-1.0.0.115-unix
  5. ./start.sh

Then I create a cluster + a target on my local instance (non-container) :

iris list irishealth                

Configuration 'IRISHEALTH'

directory:    /Users/guilbaud/is/irishealth

versionid:    2021.2.0.649.0

datadir:      /Users/guilbaud/is/irishealth

conf file:    iris.cpf  (SuperServer port = 61773, WebServer = 52773)

status:       running, since Fri Feb 25 15:35:32 2022

state:        ok

product:      InterSystems IRISHealth

I check that /api/monitor/metrics runs well :

curl http://127.0.0.1:52773/api/monitor/metrics -o metrics

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100 17634  100 17634    0     0  14174      0  0:00:01  0:00:01 --:--:-- 14383

 

127.0.0.1 will not work, due to it's still different host, and in case of this sam bundle, it should point to sam's iris. And will not work it, because, it should be available for prometheus container, not just for IRIS container for sam. So, you have to use, it's name as a host iris

And to access, some IRIS external to docker, you may use host.docker.internal hostname, or docker.for.mac.localhost with Docker on macOS

So, the first line is points to SAM's IRIS, and second to an IRIS outside

I think host.docker.internal was added in some recent versions of Docker, so, probably check the version of Docker

I'm running the very last version of Docker : 4.5.0 (74594)

 
 docker version
docker logs sam-prometheus-1

level=info ts=2022-02-28T16:14:51.209Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.1, branch=HEAD, revision=ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)"

level=info ts=2022-02-28T16:14:51.210Z caller=main.go:334 build_context="(go=go1.13.9, user=root@806b02dfe114, date=20200326-16:18:19)"

level=info ts=2022-02-28T16:14:51.210Z caller=main.go:335 host_details="(Linux 5.10.76-linuxkit #1 SMP Mon Nov 8 10:21:19 UTC 2021 x86_64 aa300025820c (none))"

level=info ts=2022-02-28T16:14:51.210Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"

level=info ts=2022-02-28T16:14:51.210Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"

level=info ts=2022-02-28T16:14:51.215Z caller=web.go:514 component=web msg="Start listening for connections" address=0.0.0.0:9090

level=info ts=2022-02-28T16:14:51.214Z caller=main.go:667 msg="Starting TSDB ..."

level=info ts=2022-02-28T16:14:51.225Z caller=head.go:575 component=tsdb msg="replaying WAL, this may take awhile"

level=info ts=2022-02-28T16:14:51.227Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0

level=info ts=2022-02-28T16:14:51.227Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=1.4381ms

level=info ts=2022-02-28T16:14:51.228Z caller=main.go:683 fs_type=EXT4_SUPER_MAGIC

level=info ts=2022-02-28T16:14:51.228Z caller=main.go:684 msg="TSDB started"

level=info ts=2022-02-28T16:14:51.228Z caller=main.go:788 msg="Loading configuration file" filename=/config/isc_prometheus.yml

ts=2022-02-28T16:14:51.234Z caller=dedupe.go:112 component=remote level=info remote_name=43bf89 url=http://iris:52773/api/sam/private/db/write msg="starting WAL watcher" queue=43bf89

ts=2022-02-28T16:14:51.234Z caller=dedupe.go:112 component=remote level=info remote_name=43bf89 url=http://iris:52773/api/sam/private/db/write msg="replaying WAL" queue=43bf89

level=info ts=2022-02-28T16:14:51.239Z caller=main.go:816 msg="Completed loading of configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:14:51.239Z caller=main.go:635 msg="Server is ready to receive web requests."

ts=2022-02-28T16:14:59.846Z caller=dedupe.go:112 component=remote level=info remote_name=43bf89 url=http://iris:52773/api/sam/private/db/write msg="done replaying WAL" duration=8.6116179s

level=info ts=2022-02-28T16:36:46.583Z caller=main.go:788 msg="Loading configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:36:46.592Z caller=main.go:816 msg="Completed loading of configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:37:14.966Z caller=main.go:788 msg="Loading configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:37:14.974Z caller=main.go:816 msg="Completed loading of configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:37:37.074Z caller=main.go:788 msg="Loading configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:37:37.081Z caller=main.go:816 msg="Completed loading of configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:37:53.001Z caller=main.go:788 msg="Loading configuration file" filename=/config/isc_prometheus.yml

level=info ts=2022-02-28T16:37:53.007Z caller=main.go:816 msg="Completed loading of configuration file" filename=/config/isc_prometheus.yml
docker exec -ti -u root sam-prometheus-1 sh

/prometheus # wget http://172.20.10.3:50004/api/monitor/metrics -O iris-a-metrics

Connecting to 172.20.10.3:50004 (172.20.10.3:50004)

saving to 'iris-a-metrics'

iris-a-metrics       100% |*************************************************************************************************|  8584  0:00:00 ETA

'iris-a-metrics' saved

/prometheus # wget http://172.20.10.3:50014/api/monitor/metrics -O iris-b-metrics

Connecting to 172.20.10.3:50014 (172.20.10.3:50014)

saving to 'iris-b-metrics'

iris-b-metrics       100% |*************************************************************************************************|  7901  0:00:00 ETA

'iris-b-metrics' saved

/prometheus # wget http://172.20.10.3:52773/api/monitor/metrics -O iris-health-metrics

Connecting to 172.20.10.3:52773 (172.20.10.3:52773)

saving to 'iris-health-metrics'

iris-health-metrics  100% |*************************************************************************************************| 24567  0:00:00 ETA

'iris-health-metrics' saved

/prometheus # wget http://172.20.10.3:52774/api/monitor/metrics -O iris-metrics

Connecting to 172.20.10.3:52774 (172.20.10.3:52774)

saving to 'iris-metrics'

iris-metrics         100% |*************************************************************************************************|  6110  0:00:00 ETA

'iris-metrics' saved

After a reboot I'm now again able to reach 1 local instance (out of 2) and 0 containers + the IRIS-SAM instance.

 

 
docker-compose.yml
 
docker ps
 

 isc_prometheus.yml