Databricks CLI and API

The workspace interface suits exploration, and automation needs code. The Databricks command-line interface, the REST API, and the software development kits let programs and scripts control the platform. Engineers create clusters, start jobs, manage files, and read results without a single click.

The Remote Control Analogy

A television has buttons on its front panel, and a remote control sends the same commands from across the room. The Databricks screen resembles the front panel. The command line and the API act as remote controls that send the same instructions from a terminal, a script, or another application.

THREE WAYS TO CONTROL DATABRICKS
──────────────────────────────────────────────────────────
┌─────────────┐   ┌──────────────┐   ┌───────────────────┐
│ CLI         │   │ REST API     │   │ SDKs              │
│ Terminal    │   │ HTTP calls   │   │ Python, Java, Go  │
│ commands    │   │ from any tool│   │ libraries         │
└──────┬──────┘   └──────┬───────┘   └─────────┬─────────┘
       └─────────────────┼─────────────────────┘
                         ▼
                 Databricks workspace

Authentication: Proving Your Identity

Every request needs credentials. Developers commonly use one of two methods.

OAuth Sign-In

OAuth opens a browser window, and you sign in with your normal account. The tool stores a short-lived token for you. This method suits people working at a keyboard.

Personal Access Token or Service Principal

A token is a long secret string that acts like a password. Automated jobs use a service principal with its own credentials, so they do not rely on a person's account. Treat tokens as secrets, store them in a vault, and rotate them regularly.

Install and Configure the CLI

Follow the official installation page for your operating system, because commands differ across Windows, macOS, and Linux. Check the installation with the version command.

databricks --version

# Sign in and save a named profile
databricks auth login --host https://YOUR-WORKSPACE-HOST --profile dev

Profiles live in a file named .databrickscfg in your home folder. A profile holds the workspace address and the credential settings. Several profiles let you switch between development and production.

[dev]
host = https://YOUR-DEV-WORKSPACE-HOST

[prod]
host = https://YOUR-PROD-WORKSPACE-HOST

Everyday CLI Commands

Command	What It Does
databricks clusters list	Show clusters in the workspace
databricks jobs list	Show jobs and their identifiers
databricks workspace list /Users	List folders and notebooks
databricks fs ls dbfs:/	List files in the file system
databricks catalogs list	Show Unity Catalog catalogs
databricks bundle deploy	Deploy a project bundle

Add --profile prod to any command to target another workspace. Add --help to see the options for a command.

The REST API

The REST API exposes the platform through web addresses. A program sends an HTTP request with a token and receives data in JSON format. Any language that can send HTTP requests can use it.

REQUEST AND RESPONSE
──────────────────────────────────────────────────────────
Program ──► GET https://HOST/api/2.1/jobs/list
            Header: Authorization: Bearer YOUR_TOKEN
        ◄── JSON list of jobs

import requests

url = "https://YOUR-WORKSPACE-HOST/api/2.1/jobs/list"
headers = {"Authorization": "Bearer YOUR_TOKEN"}

response = requests.get(url, headers=headers)
for job in response.json().get("jobs", []):
    print(job["job_id"], job["settings"]["name"])

The Python SDK

The SDK wraps the REST API in friendly Python objects. The code reads like normal Python, and the library handles authentication, retries, and paging for you.

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

for cluster in w.clusters.list():
    print(cluster.cluster_name, cluster.state)

# Start an existing job
run = w.jobs.run_now(job_id=123456789)
print("Started run:", run.run_id)

The client reads credentials from environment variables or from the .databrickscfg profile, so the code itself never contains a secret.

Choosing the Right Tool

Situation	Best Fit
Quick checks from a terminal	CLI
Shell scripts and deployment steps	CLI
Python automation with logic	Python SDK
Another language or a web service	REST API

Example: Nightly Cleanup Script

A platform team wants to find clusters that run for too long. A Python script uses the SDK to list clusters, checks the start time of each running cluster, and sends a message to the owner when the cluster exceeds a limit. A scheduler runs the script every night with a service principal.

CLEANUP SCRIPT FLOW
──────────────────────────────────────────────────────────
Scheduler ──► Python script ──► list clusters
                                     │
                         running longer than 12 hours?
                                     │
                    yes ──► notify the owner ──► log the result

Safe Automation Habits

• Use service principals for scheduled scripts.
• Give each credential the minimum permissions.
• Test destructive commands in a development workspace first.
• Keep scripts in Git and review changes.
• Read the official API reference for current endpoints, because versions evolve.

Key Points

• The CLI, REST API, and SDKs automate everything the workspace screen can do.
• OAuth suits interactive work, and service principals suit automated jobs.
• Named profiles in .databrickscfg switch between workspaces.
• The REST API returns JSON over HTTP, and the SDK wraps it in Python objects.
• Secrets belong in vaults and environment variables, not in code.
• Scripts in Git give teams repeatable, reviewable automation.