Airflow Variables and Connections

Variables store configuration values your DAGs read at runtime. Connections store credentials for external systems like databases and APIs. Both keep sensitive information and environment-specific settings out of your code.

Why Not Hard-Code Values in DAGs?

BAD — hard-coded credentials in code:
──────────────────────────────────────────────
def upload_to_db():
    conn = psycopg2.connect(
        host="prod-db.company.com",
        user="admin",
        password="SuperSecret123",   # ← visible to anyone who reads the file
        dbname="sales"
    )
──────────────────────────────────────────────

GOOD — credentials stored in Airflow Connections:
──────────────────────────────────────────────
def upload_to_db():
    hook = PostgresHook(postgres_conn_id="prod_sales_db")  # ← name only, no password
    conn = hook.get_conn()
──────────────────────────────────────────────

Hard-coded values create security risks. They also break when you switch between development and production environments. Variables and Connections solve both problems.

Airflow Variables

What Are Variables?

Variables are simple key-value pairs stored in Airflow's database. Use them for values that change between environments or that you want to update without editing DAG code.

Variable Key         │ Variable Value
─────────────────────┼─────────────────────────────────
report_output_path   │ /data/reports/
alert_email          │ team@company.com
max_rows_per_batch   │ 5000
environment          │ production

Creating Variables in the UI

Go to Admin → Variables in the Airflow UI. Click the plus (+) button. Enter a Key, a Value, and an optional Description. Click Save.

Creating Variables via CLI

airflow variables set report_output_path "/data/reports/"
airflow variables set alert_email "team@company.com"
airflow variables get report_output_path
airflow variables delete report_output_path

Reading Variables in a DAG

from airflow.models import Variable

def generate_report():
    output_path = Variable.get("report_output_path")
    email = Variable.get("alert_email")
    print(f"Saving report to {output_path}")
    print(f"Alerting {email}")

Variables With Default Values

Pass a default_var so the task does not fail if the variable is missing:

batch_size = Variable.get("max_rows_per_batch", default_var=1000)

JSON Variables

Store complex settings as JSON and deserialize them automatically:

# In the UI, set key="db_config" and value:
# {"host": "db.company.com", "port": 5432, "dbname": "sales"}

config = Variable.get("db_config", deserialize_json=True)
print(config["host"])   # → db.company.com
print(config["port"])   # → 5432

Performance Warning

Do not call Variable.get() at the top level of a DAG file (outside a function). Airflow parses DAG files frequently. Top-level Variable calls hit the database every parse, which slows Airflow down. Always read variables inside task functions.

Airflow Connections

What Are Connections?

A Connection stores the address, port, username, and password for an external system. You give it a unique name (conn_id). Your DAG code references only the name — never the actual credentials.

Connection Object:
┌────────────────────────────────────────────┐
│  Conn ID   : prod_postgres_db              │
│  Conn Type : Postgres                      │
│  Host      : db.company.com                │
│  Schema    : sales                         │
│  Login     : airflow_user                  │
│  Password  : ••••••••••••                  │
│  Port      : 5432                          │
└────────────────────────────────────────────┘

Creating Connections in the UI

Go to Admin → Connections. Click the plus (+) button. Fill in the form fields. The Conn Type dropdown gives you a list of supported systems (Postgres, MySQL, S3, HTTP, SFTP, and many more). Click Save.

Creating Connections via CLI

airflow connections add \
  --conn-id "prod_postgres_db" \
  --conn-type "postgres" \
  --conn-host "db.company.com" \
  --conn-schema "sales" \
  --conn-login "airflow_user" \
  --conn-password "SuperSecret123" \
  --conn-port 5432

Using a Connection in a DAG

from airflow.providers.postgres.hooks.postgres import PostgresHook

def load_data():
    hook = PostgresHook(postgres_conn_id="prod_postgres_db")
    hook.run("INSERT INTO sales SELECT * FROM staging_sales;")

The Hook fetches the connection details from Airflow's database automatically. The password never appears in your code.

Testing a Connection

In the UI, go to Admin → Connections. Click the pencil icon next to any connection. Scroll to the bottom of the form and click Test Connection. Airflow checks if it can reach the host and authenticate.

Environment Variables for Connections

You can also define connections through operating system environment variables. This is popular in Docker and Kubernetes deployments:

# Format: AIRFLOW_CONN_{CONN_ID_UPPERCASE}
export AIRFLOW_CONN_PROD_POSTGRES_DB="postgresql://airflow_user:SuperSecret123@db.company.com:5432/sales"

Airflow reads environment variables automatically. No UI setup needed.

Variables vs Connections: When to Use Each

Secrets Backend (Enterprise Alternative)

For production environments, many teams store secrets in dedicated vaults instead of Airflow's own database. Airflow supports plugging in these secret backends directly:

• HashiCorp Vault — popular open-source secrets manager
• AWS Secrets Manager — managed service from Amazon
• Google Secret Manager — managed service from Google
• Azure Key Vault — managed service from Microsoft

When a secrets backend is configured, Airflow looks there first before checking its own database for connections and variables.