Monitoring Airflow Metrics

Table of Contents

1. Prerequisites

2. StatsD

3. Prometheus

4. Grafana

5. GitHub Repository

statsd_on = False
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
nc -l -u localhost 8125
Metrics scraping using StatsD
docker run --name=prom-statsd-exporter \
-p 9123:9102 \
-p 8125:8125/udp \
-v $PWD/mapping.yml:/tmp/mapping.yml \
prom/statsd-exporter \
--statsd.mapping-config=/tmp/mapping.yml \
--statsd.listen-udp=:8125 \
level=info ts=2021-10-04T04:38:30.408Z caller=main.go:358 msg="Accepting Prometheus Requests" addr=:9102
docker run --name=prometheus \
-p 9090:9090 \
-v $PWD/prometheus.yml:prometheus.yml \
prom/prometheus \
--config.file=prometheus.yml \
--log.level=debug \
--web.listen-address=:9090 \
level=info ts=2021-10-02T21:09:59.717Z caller=main.go:794 msg="Server is ready to receive web requests."
docker run -d --name=grafana -p 3000:3000 grafana/grafana
New Dashboard
Edit Panel
rate(airflow{host="airflowStatsD", instance="host.docker.internal:9123", job="airflow", metric="scheduler_heartbeat"}[1m])
Metrics Browser Command
Metric Plot given a title and axis name

GitHub Repository



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store