Kubernetes Executor in Airflow

Table of Contents

1. Prerequisites

2. Process

3. Install Kubernetes and Helm

4. Configure and Install Airflow

5. DAGRun

6. References

7. What’s Next?

Prerequisites

Process

Install Kubernetes and Helm

choco install minikube -y
minikube start
kubectl get nodes
OUTPUT:
NAME STATUS ROLES AGE VERSION
minikube Ready master 2d
choco install kubernetes-helm

Configure and Install Airflow

path: "/Users/username/Documents/etl-series/dags"
helm install airflow stable/airflow -f chapter2/airflow-helm-config-kubernetes-executor.yaml --version 7.2.0
helm list
export POD_NAME=$(kubectl get pods --namespace default -l "component=web,app=airflow" -o jsonpath="{.items[0].metadata.name}")
echo
http://127.0.0.1:8080
kubectl port-forward --namespace default $POD_NAME 8080:8080

DAGRun

kubectl get pods

References

What’s Next?

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store