Simple Prometheus setup on Docker Compose

Steps to configure Prometheus, Alertmanager and Node Exporter in a simple Docker Compose configuration

Introduction

Prometheus is an open-source monitoring and alerting toolkit used to collect time-series metrics from various sources in real-time. It employs a multi-dimensional data model with labels for efficient querying and analysis. The tool features PromQL, a powerful query language for extracting insights from the data, and supports both push and pull data collection models. Prometheus allows defining alerting rules to trigger alerts when certain conditions are met, and it integrates with a wide range of technologies through its exporter ecosystem.

Prometheus is extensively utilized by DevOps and SRE professionals in Cloud Native environments, particularly those leveraging technologies like Docker or Kubernetes, to comprehensively monitor their entire applications. It relies on two essential components:

• Node Exporter: It is designed to collect and expose various system-level metrics from a target node or machine. It runs as a service on the node and provides valuable information about the node’s CPU usage, memory consumption, disk utilization, network statistics, and other crucial system-level data. Node Exporter allows Prometheus to scrape these metrics using the pull model and store them as time-series data.
• Alertmanager: It is responsible for handling and managing alerts generated by Prometheus based on predefined alerting rules. Alertmanager takes care of deduplicating, grouping, and routing alerts to various alert notification channels like email, Slack, PagerDuty, etc. It ensures that each alert is sent to the right receivers and prevents overwhelming users with redundant notifications.

In this post, we will walk through a step-by-step example that demonstrates how to set up Prometheus, Node Exporter, and Alert Manager in three phases. All code samples can be found here and it is based on the book Prometheus: Up & Running, 2nd Edition.

Diagram of the architecture of this orchestration

Pre-requisites

Consult the official Docker documentation to set up Docker and Docker Compose on your system.

• Install Docker
• Install Docker Compose

Step 1: Setup Prometheus

The step 1 involves setting up the container orchestration responsible for deploying Prometheus and the subsequent components in the upcoming phases. To achieve this, we create a docker-compose.yaml file that will deploy all the necessary components.

version: '3'
services:
  prometheus:
    image: prom/prometheus
    volumes:
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
    networks:
      - localprom
    ports:
      - 9090:9090
networks:
  localprom:
    driver: bridge

This docker-compose file sets up the following components:

• A Prometheus container utilizing the official image from Docker Hub, named prometheus. This container will be provided with a prometheus.yml configuration file and will be bind to port 9090, the standard port for Prometheus.
• Additionally, we create a network named localprom that connects all the containers in this configuration. This network allows for smooth communication between the containers and assigns domain names, making the entire setup easier to configure.

The prometheus.yml file will include the basic setup to monitor the Prometheus container itself.

global:
  scrape_interval: 10s
scrape_configs:
 - job_name: prometheus
   static_configs:
    - targets:
       - prometheus:9090

We can read the content of this file as the following:

• In the global configuration section, the scrape_interval parameter defines the frequency at which our Prometheus container will poll the various targets for monitoring data. In this instance, it is set to 10 seconds.
• In the scrape_configs section, we have a list of jobs along with their respective configurations. For the prometheus job, we specify the target as prometheus:9090 in the format <server>:<port>. Here, we use the domain name of our Prometheus container as the server, but it could also be an IP address resolving to the container's host or simply localhost for this specific example.

To execute this example, simply run the following command. Please note that if this is your first time running the example, it will take some time to start as it needs to download the Prometheus Docker image.

docker compose up -d

Now you can go to your browser and open http://localhost:9090 and see your Prometheus server up and running for the first time!

Prometheus server dashboard

When you are ready to proceed with step 2, you can shut down your setup by executing the following command.

docker compose down

Step 2: Setup Node Exporter

In this step, we will enable a container running Node Exporter and include its target in the Prometheus configuration file. To achieve this, we will add a new service using the official Node Exporter image to the docker-compose.yaml.

version: '3'
services:
  prometheus:
    image: prom/prometheus
    volumes:
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
    networks:
      - localprom
    ports:
      - 9090:9090
  node-exporter:
    image: prom/node-exporter
    networks:
      - localprom
    ports:
      - 9100:9100
networks:
  localprom:
    driver: bridge

We will create a new node-exporter service, running on port 9100. Additionally, we will include the target for Node Exporter in the prometheus.yml configuration file in a manner similar to what we did for the Prometheus server in step 1.

global:
  scrape_interval: 10s
scrape_configs:
 - job_name: prometheus
   static_configs:
    - targets:
       - prometheus:9090
 - job_name: node
   static_configs:
    - targets:
       - node-exporter:9100

Once the changes are saved, proceed to restart your Docker Compose orchestration.

docker compose up -d

If you open http://localhost:9090, you will be able to view various metrics related to the host where your Node Exporter container is running.

Prometheus dashboard displaying Node Exporter’s memory trends

Additionally, you can access the Node Exporter dashboard by opening http://localhost:9100 in your web browser.

Node Exporter server dashboard

Once you are done, you can turn down again your setup.

docker compose down

Step 3: Setup Alertmanager

For the final step, we will configure an Alertmanager server and set up alert rules in Prometheus that will be forwarded to Alertmanager. To do this, let’s make one last modification to our docker-compose.yaml file, including the Alertmanager container and the necessary configurations.

version: '3'
services:
  prometheus:
    image: prom/prometheus
    volumes:
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
      - "./rules.yml:/etc/prometheus/rules.yml"
    networks:
      - localprom
    ports:
      - 9090:9090
  node-exporter:
    image: prom/node-exporter
    networks:
      - localprom
    ports:
      - 9100:9100
  alert-manager:
    image: prom/alertmanager
    volumes:
      - "./alertmanager.yml:/alertmanager/alertmanager.yml"
    networks:
      - localprom
    ports:
      - 9093:9093
networks:
  localprom:
    driver: bridge

In this last version of the docker-compose.yaml file we can see some changes:

• We have added a new rule.yml file to the Prometheus container, which will be responsible for configuring the alert rules using PromQL. For simplicity and testing purposes, we won't delve into detailed configurations. Instead, we will set up a basic alert rule that triggers when an instance is down for at least 1 minute.

groups:
 - name: example
   rules:
   - alert: InstanceDown
     expr: up == 0
     for: 1m

• We we added the alert-manager service using the official Alertmanager image, which is bound to port 9093. This service includes an alertmanager.yml configuration file, facilitating the setup of alert notifications. For now, we won't provide specific configuration details; instead, we will configure a dummy email notification. To receive actual email notifications, you will need to set up an SMTP server and configure a valid email address as the receiver.

global:
  smtp_smarthost: 'smtp:25'
  smtp_from: 'email@example.com'
route:
  receiver: example-email
  group_by: [alertname]
receivers:
 - name: example-email
   email_configs:
    - to: 'email@example.com'

One last change you need to do is configure in your prometheus.yml file the Alertmanager server target, the alert rules you just defined and an evaluation interval for the alerting rules.

The final adjustment required is to configure the Alertmanager server target, incorporate the newly defined alert rules, and set an evaluation interval for the alerting rules in your prometheus.yml file.

global:
  scrape_interval: 10s
  evaluation_interval: 10s
rule_files:
  - rules.yml
alerting:
  alertmanagers:
   - static_configs:
     - targets:
        - alert-manager:9093
scrape_configs:
 - job_name: prometheus
   static_configs:
    - targets:
       - prometheus:9090
 - job_name: node
   static_configs:
    - targets:
       - node-exporter:9100

Once all the changes are saved, proceed to restart your docker compose orchestration.

docker compose up -d

You can access your newly created Alermanager server in http://localhost:9093.

Alertmanager dashboard

For testing purposes, you can stop the Node Exporter container, and after one minute, the alert will trigger. The Alertmanager dashboard will display the triggered alert.

Alertmanager dashboard showing the InstanceDown alert

Once you are done, you can turn down again your setup.

docker compose down

Conclusion

In this brief guide, we provided straightforward instructions on configuring a small Prometheus setup with several essential components using Docker and Compose. It is important to note that this setup is best suited for development and testing purposes and may not be ideal for production environments. For production-grade setups, it is recommended to use Kubernetes or other suitable orchestration tools.

I hope this guide proves helpful and serves as a stepping stone into the realm of Cloud Native monitoring.