Level 3 · 30 min

Secrets Management: vaults, rotation, leakage

Secrets — DB passwords, API keys, signing keys, TLS private keys — are the credentials your services use to authenticate to other systems. They are the prime target of attackers because one stolen API key can read an entire database. The discipline is keeping them out of source, distributing them safely at runtime, rotating them automatically, and detecting leakage fast.

Where Secrets Must Not Live

Anti-patterns that still ship to production weekly: secrets in source repos (yes, in .env files committed by accident); secrets in CI/CD logs (a curl -H "Authorization: $TOKEN" with set -x enabled prints the token); secrets in container image layers (ENV DB_PASSWORD=... in a Dockerfile is baked into every layer of the image, viewable by anyone with image pull access); secrets in error messages (a stack trace including the connection string); secrets in client-side code (any API key shipped to a browser is public, no exceptions). The git history pitfall: deleting a secret in a follow-up commit does not remove it from history — git filter-repo / BFG is required, and any clone made before the rewrite still has it. Once a secret has been pushed to a public repo for any duration, treat it as compromised: rotate immediately, do not just delete the commit.

Vaults: Architecture and Mechanisms

A vault (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) is a system of record for secrets with the following primitives: (1) Encryption at rest using a KMS root key (HSM-backed in cloud variants); the vault itself only ever sees decrypted secrets in memory while serving a request. (2) Identity-based access — services authenticate to the vault using their cloud workload identity (IAM role, GCP service account, Kubernetes service account token via OIDC) and receive short-lived credentials, no long-lived ''master secret'' required. (3) Audit log of every read, every write, every policy change — write-only to a separate destination (an S3 bucket the vault can append-only to). (4) Dynamic secrets: Vault generates per-request DB credentials with a 1-hour lease — instead of all services sharing one DB password forever, each request gets its own credentials that expire. Compromise of one service ' s in-memory secrets is bounded in time. (5) Transit (encryption-as-a-service): the vault performs encrypt/decrypt without exposing the key — useful for application-level field encryption (PII) without giving the app the key.

Rotation, Leak Detection, Least Privilege

Static secrets become weaker the longer they live: every backup, every log line, every memory dump is a potential leak vector. Automated rotation: DB credentials rotated by Vault on every lease (1 hour); JWT signing keys rotated quarterly (with key-id support so old tokens still verify until expiry); third-party API keys rotated quarterly via vendor APIs where supported. Pre-commit secret scanning is now table stakes: gitleaks or trufflehog as a pre-commit hook, GitHub secret scanning (auto-detects 100+ secret types, can revoke automatically with partner integrations like AWS), npm/PyPI package scanning. Response to a leak: rotate immediately, audit the secret''s usage in the audit log to determine blast radius, do not just rewrite git history (the secret has been seen). Least privilege: every secret has a single tenant (one service per secret, not ''shared service-X password''); environments fully isolated (dev / staging / prod have separate secrets — leaking dev does not leak prod); read-only services do not get secrets that allow writing; ''break-glass'' admin access is logged and time-limited (24h), with quarterly review of who used it.

Key Takeaways

Secrets in source repos are compromised the moment they''re pushed. Rotate; do not just delete the commit. Pre-commit scanning prevents most accidents.
Use a vault with workload-identity authentication and dynamic, short-lived credentials. Long-lived static credentials are the exception, not the default.
Separate secrets per environment and per service. Audit logs of every read are non-negotiable — without them, you can not bound the blast radius after a leak.

Code example

# .gitignore — at the top of every repo
.env
.env.*
!.env.example  # keep the template, never the real values
*.pem
*.key
secrets/

# pre-commit hook (.pre-commit-config.yaml)
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.0
    hooks:
      - id: gitleaks

# Vault Agent example (Kubernetes) — sidecar fetches secret, app reads from disk
apiVersion: v1
kind: Pod
metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "orders-service"
    vault.hashicorp.com/agent-inject-secret-db: "database/creds/orders-readwrite"
    vault.hashicorp.com/agent-inject-template-db: |
      {'{- with secret "database/creds/orders-readwrite" -}'}
      DB_USER="{'{.Data.username}'}"
      DB_PASS="{'{.Data.password}'}"
      {'{- end -}'}
# Vault generates a per-Pod, 1h-lease DB credential
# When the lease expires (or the Pod dies), the credential is revoked

# Application — read from injected file, no secret in env vars
Properties props = new Properties();
props.load(new FileInputStream("/vault/secrets/db"));
String user = props.getProperty("DB_USER");
String pass = props.getProperty("DB_PASS");