DevOps Secrets Management

A secret is any piece of data that grants access to something private. Passwords, API keys, database credentials, certificates, and tokens all count as secrets. Secrets management is the practice of storing, sharing, and rotating these values safely.

Why Secrets Need Special Care

Picture a hotel that gives every guest a key card. The card opens only one room and stops working after checkout. Nobody writes room codes on the front door. Software secrets deserve the same treatment. A leaked password in a public code repository can let strangers reach your servers within minutes.

Where Secrets Leak

• Passwords typed directly into source code.
• Config files committed to Git.
• Secrets printed in pipeline logs.
• Chat messages and emails that share keys.
• Container images that contain baked-in credentials.

Git remembers every change forever. Deleting a secret from a later commit does not remove it from history, so treat any leaked secret as stolen and replace it right away.

The Secrets Vault Idea

 Application                Secrets Vault               Database
     |                           |                          |
     |-- 1. "Who am I" (login) ->|                          |
     |<- 2. Short-lived token ---|                          |
     |-- 3. "Give DB password" ->|                          |
     |<- 4. Password (expires) --|                          |
     |------------------ 5. Connect with password --------->|

The application never stores the password. The vault checks identity, hands out the secret for a short time, and records who asked.

Common Tools

Environment Variables

An environment variable passes a value to a program at start time. This method keeps secrets out of the code file.

export DB_PASSWORD="read-from-vault"
python app.py

Environment variables improve on hard-coded values but still carry risks. Crash reports and debug pages can print them. A vault gives stronger control.

Kubernetes Secrets

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
stringData:
  username: appuser
  password: change-me

Kubernetes stores Secret values in an encoded form, not an encrypted form, unless the cluster enables encryption at rest. Anyone with read access to the Secret object can decode the value. Limit access with role-based rules and enable encryption.

Core Principles

Least Privilege

Give each application only the secrets it needs. A web server rarely needs the backup system password.

Rotation

Change secrets on a schedule. Rotation limits the time a stolen secret stays useful. Vaults can automate this task.

Short-Lived Credentials

A dynamic secret exists for minutes or hours and expires on its own. Vault can create a fresh database user for each application start and delete the user later.

Audit Trails

Record every access. Logs show who read a secret, when, and from where.

Scanning for Leaks

Secret scanners such as Gitleaks and TruffleHog search code and history for keys. Run a scanner in the pipeline and as a pre-commit check, so leaks stop before they reach the shared repository.

Steps After a Leak

• Revoke the exposed secret at once.
• Create a new secret and update the applications.
• Check the access logs for unknown activity.
• Add a scanner rule to prevent a repeat.

Encryption in Transit and at Rest

Encryption turns readable data into scrambled data that only a key can unlock. Encryption in transit protects data moving across a network, and TLS provides it for web traffic. Encryption at rest protects data sitting on a disk or in a database. Cloud key services, such as AWS KMS, store the master keys and handle the locking and unlocking. Applications never see the master key itself.

 Secret --> [ Encrypt with data key ] --> Stored ciphertext
                    ^
              Data key is locked by a master key inside the key service

Secrets in CI/CD Pipelines

Pipelines need credentials to deploy code. Pipeline tools store these values as masked variables, and masked values appear as stars in logs. A stronger method uses OpenID Connect. The pipeline proves its identity to the cloud provider and receives a temporary credential, so no long-lived key sits in the settings.

 Pipeline job --(identity token)--> Cloud provider
                                         |
 Pipeline job <--(temporary key, 1 hour)-+

Secrets in GitOps Workflows

GitOps keeps every setting in Git, which creates a puzzle for secrets. Three tools solve it.

• Sealed Secrets encrypts a secret so that only the target cluster can decrypt it. The encrypted file is safe to commit.
• SOPS encrypts only the values inside a YAML file and leaves the keys readable.
• External Secrets Operator keeps the real value in a vault and copies it into the cluster at run time.

Secret Sprawl and Ownership

Large teams collect hundreds of secrets over the years. Keep an inventory that lists each secret, its owner, its purpose, and its rotation date. Remove secrets that no application uses. An owner for each secret makes rotation and cleanup possible.

Break-Glass Access

A break-glass account gives an engineer emergency access when normal login systems fail. Store its credentials in a sealed place, alert the security team on every use, and rotate the credentials after each emergency.

Key Points

• Secrets never belong in code or Git history.
• A vault issues secrets on request and logs every access.
• Rotation and short lifetimes shrink the damage from theft.
• Automated scanning catches mistakes early.