Databricks Security

Data platforms hold valuable and sometimes sensitive information, so security shapes every design choice. Databricks protects data through several layers: who can sign in, what each person can do, how the network connects, and how data stays encrypted. Understanding these layers helps you build a platform that people trust.

The Office Building Analogy

Consider a modern office building. A guard at the entrance checks identity badges. Keycards open only the floors that each employee needs. Cameras and logs record who entered which room. A vault protects the most valuable documents. Databricks follows the same layered thinking.

SECURITY LAYERS
──────────────────────────────────────────────────────────
Layer 1  IDENTITY       Who are you?          (sign-in, SSO, MFA)
Layer 2  ACCESS         What may you do?      (permissions, groups)
Layer 3  NETWORK        Where do you connect? (private links, IP lists)
Layer 4  DATA           Is the data protected?(encryption, masking)
Layer 5  AUDIT          What happened?        (logs and lineage)

Identity: Proving Who You Are

Organizations connect Databricks to a central identity provider such as Microsoft Entra ID, Okta, or Google. Single sign-on lets employees use one company login. Multi-factor authentication adds a second proof, such as a phone prompt. SCIM provisioning creates and removes Databricks users automatically when HR systems change.

Service principals are special identities for programs and automated jobs. A scheduled pipeline runs as a service principal, so it never depends on a person's personal account.

Access Control: What You Can Do

Unity Catalog manages permissions for catalogs, schemas, tables, volumes, and models. Administrators grant privileges to groups instead of individuals, which keeps rules easy to maintain.

-- Allow the analyst group to read one table
GRANT SELECT ON TABLE main.sales.orders TO `analysts`;

-- Allow the engineering group to create tables in a schema
GRANT CREATE TABLE ON SCHEMA main.sales TO `engineers`;

-- Remove a permission
REVOKE SELECT ON TABLE main.sales.orders FROM `interns`;

Permissions flow downward. A user needs access to the catalog and the schema before the table permission takes effect.

PERMISSION PATH
──────────────────────────────────────────────────────────
USE CATALOG main ──► USE SCHEMA sales ──► SELECT on orders
   (door 1)             (door 2)             (door 3)
All three doors must open before the user reads a single row.

Fine-Grained Protection

Row Filters

A row filter shows each person only the rows they may see. A regional manager sees orders for their own region, and the rest stay hidden.

Column Masks

A column mask hides or scrambles sensitive values. A support agent sees the last four digits of a phone number, and a compliance officer sees the full number.

COLUMN MASK EXAMPLE
──────────────────────────────────────────────────────────
Stored value:       9876543210
Support agent sees: ******3210
Compliance sees:    9876543210

Managing Secrets

Passwords, keys, and tokens never belong inside notebook code. Databricks secret scopes store them safely, and notebooks read them at run time. The secret value stays hidden in notebook output.

db_password = dbutils.secrets.get(scope="finance-scope", key="warehouse-password")

Compute Controls

Cluster policies restrict what users can create. An administrator can limit machine sizes, require auto termination, and force tags for cost tracking. Policies keep experiments safe and affordable.

Network Protection

• IP access lists allow sign-in only from approved office or VPN addresses.
• Private connectivity keeps traffic on the cloud provider's private network instead of the public internet.
• Secure cluster connectivity removes public IP addresses from compute resources.

Encryption

Data stays encrypted when it rests in cloud storage and when it travels across the network. Organizations that need extra control can supply their own encryption keys through the cloud provider's key service.

Auditing and Lineage

Audit logs record actions such as sign-ins, permission changes, and table reads. System tables expose this activity for analysis with SQL. Lineage shows which upstream tables feed each downstream table, dashboard, or model. A security team uses lineage to trace the impact of a sensitive column.

Example: Hospital Patient Data

A hospital stores patient visit records in a lakehouse. Doctors see full records for their own patients. Researchers see a de-identified version without names or phone numbers. Billing staff see cost columns only. One table serves all three groups through row filters and column masks.

ONE TABLE, THREE VIEWS
──────────────────────────────────────────────────────────
patient_visits table
   │
   ├── Doctor group      → own patients, all columns
   ├── Research group    → all patients, names and phones masked
   └── Billing group     → all patients, cost columns only

Security Habits for Every User

• Request the minimum access that your task requires.
• Avoid sharing personal access tokens.
• Rotate secrets on a regular schedule.
• Review group memberships every quarter.

Key Points

• Databricks protects data through identity, access, network, data, and audit layers.
• Single sign-on, multi-factor authentication, and service principals secure identities.
• Unity Catalog grants permissions to groups with GRANT and REVOKE statements.
• Row filters and column masks protect sensitive content inside shared tables.
• Secret scopes keep passwords and keys out of notebook code.
• Audit logs and lineage show who did what and where data flows.