Monitor Docker containers using SAM and cAdvisor

cAdvisor (short for container Advisor) analyzes and exposes resource usage and performance data from running containers. cAdvisor exposes Prometheus metrics out of the box. 

https://prometheus.io/docs/guides/cadvisor/

Prometheus is integrated in SAM. This makes it possible to leverage the cAdvisor metrics and expose them via Prometheus and Grafana.

Since cAdvisor listens on port 8080, which conflicts with the Nginx port, you can choose to change the Nginx port to accommodate that.

Configuration Steps:

1. Change nginx port.

modify nghix.conf:

    server {
        listen 9991; 

This allows you to access cAdvisor UI via http://server:8080/, which comes with many example dashboards.

2. Configure docker-compose to add cAdvisor container:

in docker-compose.yml, add the following:

  cadvisor:
    image: google/cadvisor:latest
    ports:
      - 8080:8080   
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro

3. Configure prometheus to add job for cAdvisor:

modify isc_prometheus.yml and add:

- job_name: cadvisor
  scrape_interval: 5s
  static_configs:
  - labels:
      cluster: "1"
      group: node 
    targets:
    - cadvisor:8080

You're done! To make sure prometheus is pulling cAdvisor metrics, go to prometheus UI http://server:9090/, under Status->Targets, you should see the cAdvisor endpoint and status.

you can download some excellent pre-built dashboards with cAdvisor metrics, just need to add the cluster parameter in each query.