We’ve all done it. Committed an API key to git. Hardcoded a database password “just for testing.” Posted a screenshot with credentials visible in the corner. The security community has a name for this: Tuesday.

But secrets management doesn’t have to be painful. Let’s walk through the progression from “please no” to “actually reasonable” in handling sensitive credentials.

The Hierarchy of Secrets (From Worst to Best)

Level 0: Hardcoded in Source

1
2
3

# Don't do this. Ever.
db_password = "hunter2"
api_key = "sk-live-definitely-real-key"

This is how breaches happen. Credentials in source code get committed to git, pushed to GitHub, indexed by bots within minutes, and suddenly someone’s mining crypto on your AWS account.

Even private repos aren’t safe. Employees leave, repos get misconfigured, access tokens leak. Treat source code as public by default.

Level 1: Environment Variables

1
2

export DATABASE_URL="postgres://user:pass@localhost/db"
export API_KEY="sk-live-..."

Better! At least credentials aren’t in your codebase. But environment variables have problems:

  • They show up in process listings (ps aux)
  • They get logged by accident (printenv in a debug statement)
  • They’re visible to any process running as the same user
  • Child processes inherit them by default

Environment variables are fine for development. For production, we can do better.

Level 2: Secret Files

1
2
3
4
5

# config/secrets.yaml (gitignored)
database:
  password: "..."
api:
  key: "..."

Better separation, but now you’re managing files across servers. How do you deploy them? How do you rotate them? Who has access?

This works for small teams with simple deployments. It becomes a nightmare at scale.

Level 3: Encrypted Secret Files

1
2
3

# SOPS with age encryption
sops --encrypt secrets.yaml > secrets.enc.yaml
git add secrets.enc.yaml  # Safe to commit!

Now we’re cooking. Tools like SOPS, age, or even git-crypt let you encrypt secrets before committing them.

Benefits:

  • Secrets live in version control (audit trail!)
  • Only authorized keys can decrypt
  • Works with existing deployment pipelines

The catch: key management. Someone still needs access to the decryption key. You’ve moved the problem, not solved it.

Level 4: Secret Management Services

1
2
3
4
5

# AWS Secrets Manager
import boto3

client = boto3.client('secretsmanager')
secret = client.get_secret_value(SecretId='prod/db/password')

Cloud providers offer dedicated secret storage:

  • AWS Secrets Manager - Automatic rotation, IAM integration
  • GCP Secret Manager - Similar features, GCP-native
  • Azure Key Vault - Microsoft’s offering

Benefits:

  • Access controlled via IAM/RBAC
  • Audit logs for every access
  • Automatic rotation capabilities
  • No secrets on disk

Cost: You’re locked into a provider and paying per-secret. Fine for most production workloads.

Level 5: HashiCorp Vault (and Friends)

1
2

# Read a secret from Vault
vault kv get -field=password secret/prod/database

Vault is the Swiss Army knife of secrets management:

  • Dynamic secrets: Generate credentials on-demand that auto-expire
  • Encryption as a service: Encrypt data without handling keys
  • Multiple auth methods: LDAP, OIDC, cloud IAM, certificates
  • Policy-based access: Fine-grained control over who gets what

The dynamic secrets feature is killer. Instead of a static database password that lives forever, Vault creates a temporary user that exists for 30 minutes. If credentials leak, they’re already dead.

The downside: complexity. Vault needs to be deployed, managed, backed up, and made highly available. It’s infrastructure you’re taking on.

Practical Patterns

The “Twelve-Factor” Approach

For most applications, a hybrid approach works well:

  1. Development: .env files (gitignored) with direnv
  2. CI/CD: Secrets from your CI platform (GitHub Actions secrets, etc.)
  3. Production: Cloud secret manager or Vault
1
2
3
4
5

# Dockerfile - don't bake in secrets
FROM python:3.11
COPY . .
# Secrets come from environment at runtime, not build time
CMD ["python", "app.py"]

Secret Injection Patterns

Sidecar pattern (Kubernetes):

1
2
3
4

# Vault Agent injector adds secrets as files
annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/agent-inject-secret-db: "secret/data/db"

Init container pattern:

1
2
3
4

initContainers:
  - name: fetch-secrets
    image: vault-agent
    command: ["fetch-and-write-secrets.sh"]

External Secrets Operator:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
spec:
  secretStoreRef:
    name: aws-secrets-manager
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: prod/database

Rotation Strategy

Secrets should rotate regularly. Here’s a simple pattern:

  1. Generate new credential (new API key, new DB user)
  2. Deploy to secret store (both old and new valid)
  3. Roll out applications (they pick up new credential)
  4. Revoke old credential (after grace period)

For databases, consider short-lived credentials. Vault’s database secrets engine can create users that expire in hours, not months.

What I Actually Recommend

For most teams:

  1. Start with cloud provider secrets (Secrets Manager, etc.). It’s cheap, it works, it’s managed.

  2. Use SOPS for configuration files that need to live in git. Encrypt with KMS or age keys.

  3. Never put secrets in Docker images. Ever. Even in private registries. Build secrets should use multi-stage builds with secret mounts.

  4. Add secret scanning to CI. Tools like trufflehog, gitleaks, or GitHub’s built-in scanning catch accidents before they become incidents.

  5. Graduate to Vault when you need dynamic secrets or complex policies. Don’t start there unless you have the ops capacity.

The Human Element

Tools only go so far. The biggest secret leaks I’ve seen came from:

  • Screenshots posted to Slack with terminals open
  • Credentials shared in email “just this once”
  • .env files in downloads folders, synced to cloud storage
  • Logs that captured request bodies with API keys

Build a culture where secrets are treated seriously. Make it easy to do the right thing. Have a clear “I leaked a secret” playbook so people report issues instead of hiding them.

The goal isn’t perfect security (impossible). It’s reducing blast radius and making rotation painless when (not if) something leaks.

Secrets management is one of those things that’s easy to get wrong and painful to fix after the fact. A little investment upfront saves a lot of 3 AM incident calls later.

Start simple, automate rotation early, and remember: if a secret is convenient to use, it’s probably not secret enough.