Sign in

By Hiren Rupchandani & Mukesh Kumar

Although the Celery executor is the most preferred for data pipelines in Airflow, it has some notable drawbacks:


By Hiren Rupchandani & Mukesh Kumar


By Hiren Rupchandani & Mukesh Kumar

Sentry.io is an open-source full-stack error tracking sysem that supports a wide range of servers, browsers, desktops, and mobile languages and frameworks including PHP, Node.js, Python, etc.

The system is used by Dropbox, AirBnB, PayPal, Uber, Reddit, Mozilla, Slack, and Microsoft to monitor thousands…


By Hiren Rupchandani & Mukesh Kumar

We can finally use the Celery Executor that can enable us to make production-ready data pipelines. We can run a lot of tasks in parallel with the help of celery workers across different servers.

Although we will demonstrate a simple working only on our…


By Hiren Rupchandani & Mukesh Kumar

Now that we have achieved parallel execution of tasks in airflow, it is time to move further ahead with Celery Executor to build production ready pipelines. …


By Hiren Rupchandani & Mukesh Kumar

In our last article, we saw a high-level overview of the various executors in airflow. Now, we will see how a local executor can be used to achieve parallelism in airflow. So, let’s get started…

Setup for Local Executor


By Hiren Rupchandani & Mukesh Kumar

Airflow executors are the mechanism that handles the running of tasks. Airflow can only have one executor configured at a time; they can be set in the airflow.cfg file in the airflow directory.

Primarily, there are two types of executors in Airflow — local…


By Hiren Rupchandani & Mukesh Kumar

In the previous articles, we set up airflow in Windows (using WSL), Ubuntu, and MacOS operating systems. It’s finally time to write our first DAG in airflow!

In this article, we will guide you through some important concepts that you need to keep in…


By Hiren Rupchandani & Mukesh Kumar

In the previous articles, we saw a high-level overview of data and data pipelines, followed by an introduction to apache airflow.

Continuing the series, in this article, we will install Apache Airflow on a MacOS machine using a virtual environment.

It is assumed that…


By Hiren Rupchandani & Mukesh Kumar

In the previous articles, we saw a high-level overview of data and data pipelines, followed by an introduction to apache airflow.

Continuing the series, in this article, we will install Apache Airflow on a Ubuntu machine using a virtual environment.

The following steps were…

INSAID

One of India’s leading institutions providing world-class Data Science & AI programs for working professionals with a mission to groom Data leaders of tomorrow!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store