Airflow Tasks and Dependencies

Tasks are the individual units of work inside a DAG. Dependencies tell Airflow which tasks must finish before others can start. Setting dependencies correctly is the heart of building reliable pipelines.

What Is a Task?

A task is one step in your workflow. It is created by giving an operator a unique task_id and the details it needs to do its job. Every task belongs to one DAG.

Task Anatomy:
┌──────────────────────────────────────────┐
│  task_id     : "download_sales_data"     │  ← unique name inside the DAG
│  operator    : PythonOperator            │  ← what type of work
│  callable    : download_from_api()       │  ← the actual function
│  retries     : 3                         │  ← optional settings
│  owner       : "data_team"               │
└──────────────────────────────────────────┘

Setting Dependencies: The >> and << Operators

Right Shift >> (Downstream)

task_a >> task_b means "task_a runs first, then task_b."

task_a >> task_b >> task_c

[task_a] → [task_b] → [task_c]

Left Shift << (Upstream)

task_b << task_a means the same thing — task_a must finish before task_b. Most people prefer >> because it reads left to right like English.

set_upstream and set_downstream Methods

You can also use method calls instead of operators:

task_b.set_upstream(task_a)    # same as task_a >> task_b
task_a.set_downstream(task_b)  # same as task_a >> task_b

Parallel Tasks (Fan-Out)

Multiple tasks can run at the same time if they do not depend on each other. This speeds up your pipeline.

task_a >> [task_b, task_c, task_d]

         [task_a]
        /    |    \
  [task_b] [task_c] [task_d]

Here, task_b, task_c, and task_d all start at the same time after task_a finishes.

Converging Tasks (Fan-In)

Multiple tasks can all feed into one final task. The final task waits for all of them to finish.

[task_b, task_c, task_d] >> task_e

  [task_b] [task_c] [task_d]
        \      |      /
           [task_e]

Diamond Pattern (Fan-Out Then Fan-In)

The most common real-world pattern combines fan-out and fan-in:

task_a >> [task_b, task_c] >> task_d

       [task_a]
      /         \
  [task_b]   [task_c]
      \         /
       [task_d]

Real-world example: Extract data from two different sources in parallel (task_b and task_c), then merge and load the results (task_d).

Task Trigger Rules

By default, a task runs only if all its upstream tasks succeed. Trigger rules let you change this behavior.

task_c = PythonOperator(
    task_id="always_run_cleanup",
    python_callable=cleanup,
    trigger_rule="all_done",  # runs even if upstream tasks failed
)

Practical Trigger Rule Example

Scenario: You always want to send a notification email,
whether the pipeline succeeded or failed.

[download_data] → [process_data]
                         |
                         ↓ (all_done)
                  [send_notification]

Set trigger_rule="all_done" on send_notification. It runs no matter what happened upstream.

Task Groups (Organizing Complex DAGs)

When a DAG has many tasks, group related ones together using TaskGroup. This collapses them into a single box in the UI, making the graph easier to read.

from airflow.utils.task_group import TaskGroup

with dag:
    with TaskGroup("extract_phase") as extract:
        get_sales = PythonOperator(task_id="get_sales", ...)
        get_users = PythonOperator(task_id="get_users", ...)

    with TaskGroup("transform_phase") as transform:
        clean_sales = PythonOperator(task_id="clean_sales", ...)
        clean_users = PythonOperator(task_id="clean_users", ...)

    extract >> transform

UI Graph View:
┌────────────────┐     ┌──────────────────┐
│  extract_phase │ ──▶ │  transform_phase │
│ [get_sales]    │     │ [clean_sales]    │
│ [get_users]    │     │ [clean_users]    │
└────────────────┘     └──────────────────┘

Viewing Dependencies in the UI

Open any DAG in the Airflow UI and click Graph View. Airflow draws your entire task dependency map as a flowchart. Hover over any task box to see its upstream and downstream connections. Click a task box to see its status, logs, and options.

Full Example: ETL Pipeline With Dependencies

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.empty import EmptyOperator
from datetime import datetime

def extract_sales(): print("Extracting sales...")
def extract_users(): print("Extracting users...")
def transform():     print("Transforming data...")
def load():          print("Loading to warehouse...")
def notify():        print("Sending notification...")

with DAG("etl_pipeline", start_date=datetime(2024, 1, 1),
         schedule="@daily", catchup=False) as dag:

    start = EmptyOperator(task_id="start")

    get_sales = PythonOperator(task_id="extract_sales", python_callable=extract_sales)
    get_users = PythonOperator(task_id="extract_users", python_callable=extract_users)
    transform = PythonOperator(task_id="transform",     python_callable=transform)
    load      = PythonOperator(task_id="load",          python_callable=load)
    notify    = PythonOperator(task_id="notify",        python_callable=notify,
                               trigger_rule="all_done")
    end = EmptyOperator(task_id="end")

    start >> [get_sales, get_users] >> transform >> load >> notify >> end

         [start]
        /        \
[get_sales] [get_users]
        \        /
       [transform]
           |
          [load]
           |
        [notify]   ← runs even if load fails
           |
          [end]