Executors in Airflow

Table of Contents

Sequential Executor

Sequential Executor

Local Executor

Local Executor

Celery Executor

Celery Executor

Kubernetes Executor

Kubernetes Executor

CeleryKubernetes Executor

  1. The number of tasks needed to be scheduled at the peak exceeds the scale that your Kubernetes cluster can comfortably handle
  2. A relatively small portion of your tasks requires runtime isolation.
  3. You have plenty of small tasks that can be executed on Celery workers but you also have resource-hungry tasks that will be better to run in predefined environments.
CeleryKubernetes Executor

Dask Executor

Dask Executor

What’s Next

One of India’s leading institutions providing world-class Data Science & AI programs for working professionals with a mission to groom Data leaders of tomorrow!

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

What is the Linux Permissions?

An Invalid Comparison: Terraform v. CloudFormation et al.

31 Microservices Interview Questions For SDE II

Kotlin Basic: Data Class Vs Class

Active Directory Enumaration

Install latest GO in AWS Cloud9 (Ubuntu Server 18.04 LTS)

An introduction to the accuRx Data Stack

Raise your hand if you want to quit drinking coffee. Why, though???

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
INSAID

INSAID

One of India’s leading institutions providing world-class Data Science & AI programs for working professionals with a mission to groom Data leaders of tomorrow!

More from Medium

Airflow: Create Custom Operator from MySQL to PostgreSQL

Airflow on Kubernetes Cluster: Using your own Python modules on Spark Jobs

Apache Airflow: DAG Structure and Data Pipeline

How to install Apache Airflow on k8s in 30 minutes