Set Up Celery, Flower, and RabbitMQ for Airflow

Table of Contents

1. Setting up a RabbitMQ Server

2. Setting up Celery and Flower

3. DAGRun

4. What’s Next?

Setting up a RabbitMQ Server

(airflow_env) username@desktopname:~$ sudo apt-get install erlang
(airflow_env) username@desktopname:~$ sudo apt-get install rabbitmq-server
(airflow_env) username@desktopname:~$ sudo service rabbitmq-server start
OUTPUT:
* Starting RabbitMQ Messaging Server rabbitmq-server
(airflow_env) username@desktopname:~$ service --status-all
OUTPUT:
[ - ] ...
[ - ] rabbitmq-server
[ - ] ...
(airflow_env) username@desktopname:~$ sudo rabbitmq-plugins enable rabbitmq_management
RabbitMQ Server Home
(airflow_env) username@desktopname:~$ sudo rabbitmqctl add_user admin admin
(airflow_env) username@desktopname:~$ sudo rabbitmqctl set_user_tags admin administrator
(airflow_env) username@desktopname:~$ sudo rabbitmqctl set_permissions -p / admin "." "." ". "
RabbitMQ Login with admin

Setting up Celery and Flower

(airflow_env) username@desktopname:~$ sudo pip3 install apache-airflow[celery]
(airflow_env) username@desktopname:~$ airflow celery flower
OUTPUT:

[2021-09-05 19:07:36,724] {command.py:135} INFO - Visit me at http://0.0.0.0:5555
[2021-09-05 19:07:36,733] {command.py:142} INFO - Broker: amqp://admin:**@localhost:5672//
[2021-09-05 19:07:36,735] {command.py:143} INFO - Registered tasks:
.
.
.
Celery Flower Server Dashboard
sql_alchemy_conn = mysql+pymysql://sql_username:sql_password@localhost/airflow_dbexecutor = CeleryExecutorbroker_url = amqp://admin:admin@localhost/result_backend = db+mysql+pymysql://sql_username:sql_password@localhost/airflow_db
(airflow_env) username@desktopname:~$ sudo service mysql start
(airflow_env) username@desktopname:~$ sudo service rabbitmq-server start
(airflow_env) username@desktopname:~$ airflow db init
(airflow_env) username@desktop_name:~/mnt/c/Users/username/Documents/airflow$ airflow webserver
(airflow_env) username@desktop_name:~/mnt/c/Users/username/Documents/airflow$ airflow scheduler
(airflow_env) username@desktop_name:~/mnt/c/Users/username/Documents/airflow$ airflow celery worker
(airflow_env) username@desktop_name:~/mnt/c/Users/username/Documents/airflow$ airflow celery flower
Celery Flower with a Celery Worker’s Status
RabbitMQ Server Queues during execution
Airflow Webserver UI

What’s next

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store