dbt CI/CD Pipeline

A CI/CD pipeline for dbt automatically tests every code change before it reaches production and deploys passing changes without manual steps. CI stands for Continuous Integration — catching errors early. CD stands for Continuous Deployment — shipping verified changes automatically. Together they mean no bad code reaches your dashboards undetected.

The Problem Without CI/CD

Without CI/CD:
  Developer changes fct_orders.sql
  Runs dbt run locally, looks fine
  Merges to main branch
  Production job runs that night
  Broken model fails at 02:00
  CEO opens dashboard at 08:00 -- no data
  On-call engineer debugged the issue for 2 hours

With CI/CD:
  Developer changes fct_orders.sql
  Opens pull request
  CI pipeline runs automatically on the PR branch
  fct_orders test fails in CI
  PR is blocked, developer fixes the issue
  Production never sees the broken model

CI/CD Pipeline Overview

Developer writes code | v Push to feature branch | v Open Pull Request (PR) | v [CI job triggers automatically] dbt deps dbt build --select state:modified+ | Pass ---> PR can be reviewed and merged Fail ---> PR blocked, developer fixes and pushes again | v Merge to main branch | v [CD job triggers automatically] dbt deps dbt build --target prod | v Production warehouse updated | v Dashboard refreshed with new data

CI Pipeline with GitHub Actions

# .github/workflows/dbt_ci.yml
name: dbt CI

on:
  pull_request:
    branches: [main]

jobs:
  dbt-ci:
    runs-on: ubuntu-latest

    env:
      SNOWFLAKE_ACCOUNT:  ${{ secrets.SNOWFLAKE_ACCOUNT }}
      SNOWFLAKE_USER:     ${{ secrets.SNOWFLAKE_USER }}
      SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}
      DBT_SCHEMA:         dbt_ci_${{ github.run_id }}

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dbt
        run: pip install dbt-snowflake

      - name: Install packages
        run: dbt deps

      - name: Download production artifacts
        run: |
          dbt run-operation download_artifacts
          # or copy from S3 / dbt Cloud API

      - name: Run changed models and tests
        run: |
          dbt build \
            --select state:modified+ \
            --defer \
            --state ./prod-artifacts \
            --target ci

Slim CI with --defer and --state

Running the entire project on every pull request wastes time and money. Slim CI runs only the models that changed and their downstream children:

dbt build --select state:modified+ --defer --state ./prod-artifacts

state:modified+   run changed models AND their downstream dependencies
--defer           for models NOT in the selection, read from production
--state           folder containing production manifest.json for comparison

Example with 200 models, only 3 changed:

Without slim CI: rebuild all 200 models (45 minutes)
With slim CI:    rebuild 3 changed + 8 downstream models (4 minutes)

Obtaining Production Artifacts

The --state flag needs the production manifest.json to compare against. Get it from:

Option 1: dbt Cloud API

curl -H "Authorization: Token $DBT_API_TOKEN" \
  "https://cloud.getdbt.com/api/v2/accounts/$ACCOUNT_ID/runs/latest/artifacts/manifest.json" \
  -o prod-artifacts/manifest.json

Option 2: S3 Bucket

aws s3 cp s3://my-bucket/dbt-artifacts/manifest.json prod-artifacts/

CD Pipeline: Deploy to Production

# .github/workflows/dbt_deploy.yml
name: dbt Deploy

on:
  push:
    branches: [main]

jobs:
  dbt-deploy:
    runs-on: ubuntu-latest

    env:
      SNOWFLAKE_ACCOUNT:  ${{ secrets.SNOWFLAKE_ACCOUNT_PROD }}
      SNOWFLAKE_USER:     ${{ secrets.SNOWFLAKE_USER_PROD }}
      SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD_PROD }}

    steps:
      - uses: actions/checkout@v4

      - name: Install dbt
        run: pip install dbt-snowflake

      - name: Install packages
        run: dbt deps

      - name: Run full production build
        run: dbt build --target prod

      - name: Generate docs
        run: dbt docs generate

      - name: Upload artifacts to S3
        run: |
          aws s3 cp target/manifest.json s3://my-bucket/dbt-artifacts/
          aws s3 cp target/catalog.json  s3://my-bucket/dbt-artifacts/
          aws s3 sync target/ s3://my-bucket/dbt-docs/

CI/CD with dbt Cloud (Simpler Setup)

dbt Cloud handles CI/CD with minimal configuration:

Step 1: Create a CI job in dbt Cloud
  Commands: dbt build --select state:modified+ --defer --state ./prod-artifacts
  Trigger: on pull request open

Step 2: Create a Production deploy job
  Commands: dbt deps, dbt build
  Trigger: after merge to main (via webhook or schedule)

Step 3: Connect dbt Cloud to GitHub
  dbt Cloud posts CI results directly to the GitHub PR
  No YAML files needed

Branch Strategy

main         production-ready code, auto-deploys to prod
feature/*    developer branches, CI tests on PR
hotfix/*     urgent fixes, tested in CI before merging to main

What to Test in CI

Always test in CI:
  dbt build --select state:modified+   (changed models and their children)

Also consider testing:
  +exposure:*   (all models feeding exposures)
  tag:critical  (models tagged as business-critical)

CI/CD Checklist

[ ] GitHub Actions (or GitLab CI) workflow file created
[ ] Warehouse credentials stored as encrypted secrets
[ ] Production manifest.json accessible to CI runner
[ ] Slim CI uses --select state:modified+ --defer --state
[ ] CD job runs dbt build --target prod on merge to main
[ ] dbt docs generate runs after production build
[ ] Artifacts uploaded to storage for next CI comparison
[ ] Failure notifications sent to Slack or email

Previous lesson

Back to course

Next lesson