Airflow with Docker and Kubernetes

Docker packages Airflow and its dependencies into isolated containers that run identically everywhere. Kubernetes orchestrates those containers across multiple machines for production-grade scalability. Most serious Airflow deployments use one or both technologies.

Why Run Airflow in Docker?

Problem Without Docker:
  Developer A installs Python 3.9, Airflow 2.7, PostgreSQL 14
  Developer B installs Python 3.11, Airflow 2.9, PostgreSQL 15
  → "Works on my machine" — different setups cause different bugs

With Docker:
  Everyone uses the same container image
  Same Python, same Airflow, same libraries
  → Identical behavior on every machine, every environment

Docker Concepts in Plain English

Running Airflow With Docker Compose

The official Docker Compose setup starts all Airflow components (webserver, scheduler, worker, database, Redis) with one command.

Step 1: Download the Compose File

curl -LfO 'https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml'

Step 2: Create Required Directories and the .env File

mkdir -p ./dags ./logs ./plugins ./config
echo "AIRFLOW_UID=$(id -u)" > .env

Step 3: Initialize the Database

docker compose up airflow-init

Step 4: Start All Services

docker compose up -d

Containers Started:
┌────────────────────────────────────────────┐
│  airflow-webserver   → port 8080           │
│  airflow-scheduler   → reads dags/         │
│  airflow-worker      → runs tasks          │
│  airflow-triggerer   → deferred tasks      │
│  postgres            → metadata database   │
│  redis               → task queue (Celery) │
└────────────────────────────────────────────┘

Open http://localhost:8080 and log in with airflow / airflow.

Stopping All Services

docker compose down

Stopping and Deleting All Data

docker compose down --volumes --rmi all

Adding Python Packages to Docker

Create a custom Docker image that extends the official Airflow image with your extra packages:

# Dockerfile
FROM apache/airflow:2.9.2

# Install extra pip packages
RUN pip install --no-cache-dir \
    pandas \
    apache-airflow-providers-postgres \
    apache-airflow-providers-amazon

Build and use your custom image:

docker build -t my-airflow:latest .

# In docker-compose.yaml, replace the image line:
# image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.9.2}
# with:
# image: my-airflow:latest

Alternatively, use a requirements.txt file and mount it:

# docker-compose.yaml (override section)
x-airflow-common:
  &airflow-common
  build:
    context: .
    dockerfile: Dockerfile

Kubernetes Concepts in Plain English

Real World Analogy: Kubernetes is like a shipping port manager.
───────────────────────────────────────────────────────────────
Docker Container = one shipping container (self-contained cargo)
Kubernetes Pod   = a truck carrying one or more containers
Node             = a dock (physical or virtual machine)
Cluster          = the entire port (all docks together)
Kubernetes       = the port manager who assigns trucks to docks,
                   replaces broken trucks, and scales capacity
───────────────────────────────────────────────────────────────

Deploying Airflow on Kubernetes With Helm

Helm is the Kubernetes package manager. The official Apache Airflow Helm chart sets up the entire Airflow infrastructure in a Kubernetes cluster with one command.

Step 1: Add the Helm Repository

helm repo add apache-airflow https://airflow.apache.org
helm repo update

Step 2: Create a Namespace

kubectl create namespace airflow

Step 3: Install the Chart

helm install airflow apache-airflow/airflow \
  --namespace airflow \
  --set executor=KubernetesExecutor \
  --set images.airflow.tag=2.9.2

Step 4: Access the Web UI

kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow

KubernetesExecutor: One Pod Per Task

With KubernetesExecutor, each task runs in its own Kubernetes Pod:

Task Queue → Scheduler → Kubernetes API → Pod created
                                              │
                                        Task runs inside Pod
                                              │
                                        Task finishes
                                              │
                                        Pod deleted

Benefits:

• Full isolation — one task's crash cannot affect another
• Custom resources — each task specifies how much CPU and RAM it needs
• Infinite scale — Kubernetes spins up as many pods as needed
• No idle workers — pods exist only while tasks are running

Setting Resources Per Task

from kubernetes.client import models as k8s

task = PythonOperator(
    task_id="heavy_computation",
    python_callable=run_heavy_job,
    executor_config={
        "pod_override": k8s.V1Pod(
            spec=k8s.V1PodSpec(
                containers=[
                    k8s.V1Container(
                        name="base",
                        resources=k8s.V1ResourceRequirements(
                            requests={"cpu": "2", "memory": "4Gi"},
                            limits={"cpu": "4",   "memory": "8Gi"},
                        ),
                    )
                ]
            )
        )
    },
)

DAG Distribution: Syncing DAG Files to Workers

In distributed deployments, all workers need access to the same DAG files. Three common approaches:

Architecture Summary: Docker vs Kubernetes

Docker Compose (single machine):
  Good for: local dev, small teams, staging environments
  Limits: only scales vertically (bigger machine)
  Setup complexity: low

Kubernetes + Helm (multi-machine cluster):
  Good for: production, large teams, enterprise workloads
  Limits: requires Kubernetes knowledge
  Setup complexity: medium-high
  Reward: unlimited horizontal scaling + high availability