Airflow Executors and Scalability

An Executor controls how Airflow runs tasks. The default executor runs tasks one by one on a single machine. Production systems use distributed executors that run many tasks at the same time across multiple machines.

What Does an Executor Do?

Airflow Architecture:
┌──────────────────────────────────────────────────────┐
│                                                      │
│  Scheduler                                           │
│  ─────────                                           │
│  Reads DAG files, decides which tasks are ready      │
│  Hands tasks to the Executor                         │
│         │                                            │
│         ▼                                            │
│  Executor                                            │
│  ────────                                            │
│  Receives tasks from Scheduler                       │
│  Decides HOW and WHERE to run them                   │
│  Reports results back to Scheduler                   │
│         │                                            │
│         ▼                                            │
│  Worker(s)                                           │
│  ─────────                                           │
│  Actually execute the task code                      │
│                                                      │
└──────────────────────────────────────────────────────┘

The Four Main Executors

1. SequentialExecutor (Default for Development)

Runs one task at a time on the same machine as the Scheduler. Simple to set up but cannot run tasks in parallel.

Task Queue:  [task_a] → [task_b] → [task_c] → [task_d]
             (one runs, others wait)

Use SequentialExecutor only for local testing and learning. Never use it in production.

2. LocalExecutor (Single Machine, Parallel)

Runs tasks in parallel using Python sub-processes. All tasks run on the same machine as the Scheduler.

Task Queue:  [task_a] [task_b] [task_c]
                ↓        ↓        ↓
            Process1  Process2  Process3  ← run simultaneously
            (same machine)

LocalExecutor is good for small to medium workloads on a single powerful server. Set it in airflow.cfg:

[core]
executor = LocalExecutor

LocalExecutor requires a PostgreSQL or MySQL metadata database — SQLite does not support the parallel writes it needs.

3. CeleryExecutor (Distributed, Multi-Machine)

Distributes tasks to worker machines using a message broker (Redis or RabbitMQ). Each worker machine runs independently and picks tasks from a shared queue.

Central Setup:
┌──────────────┐     ┌──────────────────────────┐
│  Scheduler   │────▶│  Message Broker (Redis)  │
└──────────────┘     └──────────────────────────┘
                               │
              ┌────────────────┼────────────────┐
              ▼                ▼                ▼
         [Worker 1]       [Worker 2]       [Worker 3]
         Server A          Server B         Server C
         Runs task_a       Runs task_b      Runs task_c

CeleryExecutor scales horizontally — add more worker machines when you need more capacity.

# Install Celery support
pip install apache-airflow[celery]

# airflow.cfg
[core]
executor = CeleryExecutor

[celery]
broker_url = redis://localhost:6379/0
result_backend = db+postgresql://user:pass@host/airflow

Start a Celery worker on each worker machine:

airflow celery worker

4. KubernetesExecutor (Container-Per-Task)

Runs each task in its own Kubernetes Pod. The Pod starts when the task is queued and shuts down when the task finishes.

When task_a is queued:
  Scheduler → Kubernetes API → Pod created → task_a runs → Pod deleted

When task_b is queued simultaneously:
  Scheduler → Kubernetes API → Pod created → task_b runs → Pod deleted

Benefits: complete isolation between tasks, no shared state, scales to thousands of tasks, and uses exactly the resources each task needs.

[core]
executor = KubernetesExecutor

[kubernetes]
namespace = airflow
worker_container_repository = apache/airflow
worker_container_tag = 2.9.2

Choosing the Right Executor

Task Pools: Controlling Concurrency

A Pool sets a maximum number of tasks that can run simultaneously for a specific resource. This prevents overloading a database or API with too many concurrent requests.

Scenario: A database can handle 10 connections at a time.
You have 50 tasks that all query the database.

Without a Pool:  All 50 tasks try to connect simultaneously → database overloads.
With a Pool:     Only 10 tasks run at once → database stays healthy.

Create a pool in Admin → Pools in the UI. Set the name (e.g., database_pool) and slots (e.g., 10).

Assign a task to a pool:

query_task = PythonOperator(
    task_id="query_db",
    python_callable=run_query,
    pool="database_pool",
    pool_slots=1,  # this task uses 1 of the 10 slots
)

Concurrency Settings in airflow.cfg

[core]
# Max tasks running across ALL DAGs at once
parallelism = 32

# Max tasks running in one DAG at once
dag_concurrency = 16

# Max simultaneous DAG runs for one DAG
max_active_runs_per_dag = 16

Tune these numbers based on your machine's CPU and RAM. More concurrency is not always faster — it depends on how much work each task does.

Scaling Diagram: Small vs Large Deployment

SMALL (LocalExecutor):
┌────────────────────────────┐
│  Single Server             │
│  ┌──────────┐ ┌─────────┐  │
│  │Scheduler │ │Webserver│  │
│  └──────────┘ └─────────┘  │
│ ┌────────────────────────┐ │
│ │  Local Workers (fork)  │ │
│ └────────────────────────┘ │
└────────────────────────────┘

LARGE (CeleryExecutor):
┌──────────┐  ┌─────────┐  ┌───────┐
│Scheduler │  │Webserver│  │ Redis │
└────┬─────┘  └─────────┘  └───┬───┘
     │                         │
     │    ┌────────────────────┘
     │    ▼
     │  ┌──────────┐  ┌──────────┐  ┌──────────┐
     └─▶│ Worker 1 │  │ Worker 2 │  │ Worker 3 │
        └──────────┘  └──────────┘  └──────────┘
        (Add more workers as load grows)