Databricks Connect

Databricks Connect lets developers write and run Spark code from a local editor while the heavy work happens on Databricks compute. You use familiar tools such as VS Code, PyCharm, or a terminal, and the platform processes the data. Debugging, testing, and refactoring feel like regular software development.

The Remote Kitchen Analogy

Imagine a chef who writes recipes at home and sends them to a large professional kitchen. The kitchen cooks the meal with powerful equipment and sends back a small tasting sample. The chef keeps a comfortable desk, and the kitchen handles the heavy cooking. Databricks Connect works the same way. Your laptop holds the code, and the cluster does the processing.

HOW DATABRICKS CONNECT WORKS
──────────────────────────────────────────────────────────
Your laptop (editor + Python code)
        │  sends the query plan
        ▼
Databricks compute (Spark processes the data)
        │  returns results
        ▼
Your laptop (prints, tests, or saves a small result)

Why Developers Use It

Write code in a full-featured editor with autocomplete and refactoring tools.
Set breakpoints and step through code line by line.
Run unit tests from a local test runner.
Reuse the same project in Git and continuous integration pipelines.
Work with real data without copying it to a laptop.

What You Need

A Databricks workspace with compute, such as serverless compute or a supported cluster.
Python installed on your computer.
Authentication, such as a configuration profile created by the CLI.
The databricks-connect package that matches your compute version.

Version matching matters. Choose a package version that fits the Databricks Runtime of your cluster, and follow the compatibility table on the official documentation page.

Set Up a Project

# Create an isolated environment
python -m venv .venv
source .venv/bin/activate

# Install the package (pick the version that matches your compute)
pip install "databricks-connect==YOUR.VERSION.*"

# Sign in once and save a profile
databricks auth login --host https://YOUR-WORKSPACE-HOST

The virtual environment keeps project libraries separate from other Python work on your computer.

Write Code That Runs Remotely

from databricks.connect import DatabricksSession

spark = DatabricksSession.builder.getOrCreate()

orders = spark.table("main.shop.orders")

summary = (orders
    .filter("amount > 100")
    .groupBy("city")
    .count())

summary.show(10)

The code looks identical to notebook code. The DatabricksSession object replaces the notebook's built-in spark variable. Spark sends the plan to the remote compute and returns only the rows that you display.

Debug Like a Software Engineer

Place a breakpoint inside a function, run the file in debug mode, and inspect variables. A small DataFrame converted with toPandas lets you examine values in the editor. Avoid converting huge tables, because the result must fit in your laptop's memory.

DEBUG LOOP
──────────────────────────────────────────────────────────
Write function ──► set breakpoint ──► run in debug mode
       ▲                                     │
       └──── fix the bug ◄── inspect values ◄┘

Test Your Transformations

Wrap business logic in small functions that accept and return DataFrames. Test each function with a tiny sample DataFrame.

# transform.py
from pyspark.sql.functions import col

def keep_large_orders(df, minimum):
    return df.filter(col("amount") >= minimum)

# test_transform.py
from transform import keep_large_orders

def test_keep_large_orders(spark):
    data = [(1, 50), (2, 150), (3, 300)]
    df = spark.createDataFrame(data, ["order_id", "amount"])
    result = keep_large_orders(df, 100)
    assert result.count() == 2

A test runner such as pytest executes these tests. Continuous integration systems repeat them on every pull request.

Databricks Connect Compared with Notebooks

Need	Notebook	Databricks Connect
Quick exploration and charts	Excellent	Limited
Large codebase with many modules	Harder	Excellent
Step-through debugging	Basic	Full editor support
Automated testing	Limited	Strong

Limits to Know

Only the Spark work runs remotely. Plain Python code, such as file operations and local libraries, runs on your laptop. Some Spark features, such as low-level RDD operations, are unavailable in the Spark Connect design that powers the tool. Check the documentation for the current list of supported features.

Example: Data Cleaning Library

A data team builds a library of cleaning functions for customer tables. Developers write the functions in VS Code and test them with a small sample. A final run against the real customer table confirms the result on actual data. After a pull request review, the team packages the library and deploys it through a bundle.

CLEANING LIBRARY PROJECT
──────────────────────────────────────────────────────────
VS Code ──► unit tests ──► real-data check via Databricks Connect
                                    │
                         pull request ──► review ──► deploy

Key Points

Databricks Connect runs local Spark code on remote Databricks compute.
Developers keep their favorite editors, debuggers, and test tools.
DatabricksSession replaces the notebook spark variable.
Package and compute versions must match.
Small functions and sample DataFrames make unit testing simple.
Notebooks suit exploration, and Databricks Connect suits larger software projects.

Previous lesson

Back to course

Next lesson