Airflow Error Tracking using Sentry

Table of Contents

Install sentry dependency for airflow:

  • You can install sentry’s dependency in your virtual environment using the following command:
pip install ‘apache-airflow[sentry]’

Set Up your Account on sentry.io.

  • While the installation finishes, go to sentry.io and sign up for a free account.
  • After signing up, create a new project, give that project any name, like “airflow_error_mgmt” select any team name, and select Python as the language.
New Project in Sentry
  • We will be then taken to a page that has a block of code like this:
DSN Key partially hidden from our side
  • Copy the DSN key given in the block of code.

Set Up airflow.cfg

  • Next, in the airflow.cfg file in the airflow directory, search for sentry and we will see two parameters:
  1. sentry_on
  2. sentry_dsn
airflow.cfg file
  • So, sentry has been set up on our system.
  • Now, to make sure that Sentry is working well with airflow, we need some errors.

DAG File

  • Let’ create a simple error by commenting out any of the import statements (don’t comment out the import DAG statement, your DAG file will never be recognized by the scheduler)
  • In the given file, we have commented out the import statement of the Python Operator.

Error Tracking using Sentry

  • Now, activate the airflow scheduler, webserver, and workers. The webserver will throw an error on the CLI and UI.
Webserver Error
  • If we go to sentry’s dashboard and click on the issues tab, we should see that sentry has caught an error.
Sentry Dashboard reporting Error
  • It also mentions the error that has occurred, the part of the code that caused the error, and the number of events that have taken place.
  • You will also receive an email on your registered email account for sentry.
Sentry Email Error Reporting
  • So Sentry is useful for remote tracking of your data pipelines and other systems.
  • You can also explore the various elements provided by sentry to troubleshoot and test your pipeline.

What’s Next?

One of India’s leading institutions providing world-class Data Science & AI programs for working professionals with a mission to groom Data leaders of tomorrow!

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Where To Determine Movies And Tv Shows Filming This Columbus Day

Hosting a Dynamic Content Website with CloudFront, Route 53, and ALB in AWS

what i have been waiting for has come !

Monitoring Gin and GORM with OpenTelemetry

gRPC load balancing — Service Meshes

Optimize AWS Lambdas

Basics of Data Base Management Systems

My Journey: How to Create a Filter in #MEM

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
INSAID

INSAID

One of India’s leading institutions providing world-class Data Science & AI programs for working professionals with a mission to groom Data leaders of tomorrow!

More from Medium

Data pipeline automation using Airflow DAGs in Docker containers

How to build a data lake from scratch — Part 2: Connecting the components

How to install Apache Airflow on k8s in 30 minutes

Apache Hop, installation on Windows 10/11