ADE Key Vault and Managed Identity

Pipelines need passwords, connection strings, and keys. Writing these secrets inside code or pipeline settings invites trouble, since anyone who reads the code reads the secret. Azure Key Vault stores secrets safely. Managed identities let Azure services prove who they are without any password at all.

The Hotel Analogy

A hotel keeps its master keys in a locked safe at the front desk. A guest receives a keycard that opens only one room for a limited period. Key Vault plays the role of the safe. Managed identity plays the role of a staff badge that the hotel system recognizes automatically. Nobody photocopies the master key.

What Key Vault Stores

Item	Meaning	Data Engineering Example
Secret	Any small piece of sensitive text	A database password or an API token
Key	A cryptographic key	A customer-managed key that encrypts storage
Certificate	A security certificate with its private key	A certificate for secure partner connections

What a Managed Identity Is

A managed identity is an automatically managed account that Azure creates for a service such as Azure Data Factory, Databricks, or a Function App. Azure rotates the credentials behind the scenes. The owner never sees or handles a password.

• System-assigned: Tied to one resource and deleted with it. Choose it for simple, single-service cases.
• User-assigned: A standalone identity that several resources share. Choose it when many services need the same permissions.

How the Pieces Work Together

Azure Data Factory (has a managed identity)
        |
        | 1. Asks Microsoft Entra ID for a token as itself
        v
Microsoft Entra ID  ---- gives a short-lived token ----+
                                                        |
        +-----------------------------------------------+
        |
        | 2. Presents the token to Key Vault
        v
Azure Key Vault  (checks: does this identity hold a permitted role?)
        |
        | 3. Returns the secret value
        v
Azure SQL Database connection opens (password never stored in the pipeline)

Scenario: Sunrise Hotels Data Pipeline

Sunrise Hotels, a fictional chain, loads booking data from an on-premises database into the lake every night. The database password once lived inside a pipeline setting, and a copy of that setting appeared in a shared document. The team fixed the problem in four steps.

1. Create a Key Vault named sunrise-data-kv and store the password as a secret named bookingdb-password.
2. Give the Data Factory managed identity the Key Vault Secrets User role on that vault.
3. Change the linked service to read the password from Key Vault instead of holding it directly.
4. Rotate the old password, since the leaked copy remains valid until replaced.

Access Control with Roles

Azure role-based access control decides who can do what. Give each identity the smallest role that completes the job.

Role	Allows
Key Vault Secrets User	Read secret values
Key Vault Secrets Officer	Create, update, and delete secrets
Key Vault Administrator	Manage everything in the vault
Storage Blob Data Contributor	Read and write files in storage, often assigned to a managed identity

Passwordless Access to Storage and SQL

Managed identity removes secrets entirely for many connections. A Data Factory identity with the Storage Blob Data Contributor role reads and writes Data Lake Storage Gen2 with no key. Azure SQL Database supports Microsoft Entra authentication, so an identity connects once an administrator creates a matching database user for it.

Databricks Secret Scopes

Databricks notebooks need secrets too. A secret scope backed by Key Vault lets notebook code fetch a value by name. Databricks hides the value in notebook output and replaces it with the word REDACTED.

db_password = dbutils.secrets.get(scope="kv-scope", key="bookingdb-password")

Protection Features Worth Enabling

• Soft delete recovers deleted secrets during a retention period.
• Purge protection blocks permanent deletion until the retention period ends.
• Private endpoints keep vault traffic off the public internet.
• Diagnostic logs record every access for later review.

Common Mistakes

• Placing passwords in notebooks, pipeline parameters, or source control.
• Granting broad administrator roles to every pipeline.
• Skipping secret rotation after a leak.
• Sharing one vault across development and production.

Key Points

• Key Vault keeps secrets, keys, and certificates in one guarded place.
• Managed identities let Azure services sign in without passwords.
• Pipelines fetch secrets at run time through role-based access.
• Least privilege, soft delete, and logging protect the whole setup.