Every infrastructure team eventually faces the same uncomfortable question: where do the secrets go?

API keys, database passwords, TLS certificates, OAuth tokens — they all need to live somewhere. The wrong answer (“in the repo, it’s fine, it’s private”) creates technical debt that compounds silently until someone accidentally pushes to public or an ex-employee still has access to production credentials.

The Anti-Patterns First

Environment variables everywhere. Yes, the twelve-factor app says config comes from the environment. But “the environment” doesn’t mean a .env file committed to git. Environment variables are runtime config, not secret storage.

Shared credentials. One API key for the whole team. Nobody knows who’s using it. Can’t revoke without breaking everyone. Audit logs are useless because every request looks the same.

Secrets in CI/CD config files. GitHub Actions secrets are fine. Hardcoded values in your workflow YAML are not. The distinction matters when you’re doing code review at 2am.

Patterns That Actually Work

1. External Secret Stores

HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager — pick one. The specific choice matters less than having a single source of truth.

1
2
3
4
5
6
7
8

# Instead of this:
export DATABASE_URL="postgres://user:hardcoded_password@host/db"

# Do this:
export DATABASE_URL=$(aws secretsmanager get-secret-value \
  --secret-id prod/database/url \
  --query SecretString \
  --output text)

The secret never touches disk. It’s fetched at runtime, logged in audit trails, and can be rotated without redeploying.

2. Short-Lived Credentials

Static credentials are a liability. The longer they exist, the more likely they leak.

AWS STS temporary credentials, Vault dynamic secrets, and OIDC federation all solve this differently but with the same goal: credentials that expire before anyone can abuse them.

1
2
3
4

# Vault dynamic database credentials
path "database/creds/readonly" {
  capabilities = ["read"]
}

Your app requests credentials, gets a username/password valid for 1 hour, uses them, forgets them. If they leak, they’re already expired by the time anyone notices.

3. Envelope Encryption

Sometimes you need to store encrypted secrets in git — Terraform state, Kubernetes sealed secrets, SOPS-encrypted config files. The pattern is always the same: encrypt the data with a data key, encrypt the data key with a master key stored elsewhere.

1
2

# SOPS with AWS KMS
sops --encrypt --kms arn:aws:kms:us-east-1:123:key/abc config.yaml > config.enc.yaml

The encrypted file is safe to commit. Decryption requires access to the KMS key, which has its own IAM policies and audit logs.

4. Secret Zero Problem

Every secrets management solution has a bootstrap problem: how does the application authenticate to the secret store in the first place?

For cloud-native deployments, the answer is usually workload identity — IAM roles for EC2/ECS/Lambda, Workload Identity for GKE, Managed Identity for Azure. The cloud provider handles the initial authentication.

For on-prem or hybrid, you’re looking at Vault’s AppRole, machine certificates, or (pragmatically) a single bootstrap secret stored more carefully than the rest.

The Migration Path

You don’t fix secrets management in a sprint. The realistic path:

  1. Audit. Find every secret in your codebase. Tools like trufflehog and gitleaks help.
  2. Centralize. Pick a secret store. Migrate the most critical secrets first (database credentials, API keys with billing implications).
  3. Rotate. Assume everything that’s been in git is compromised. Generate new credentials.
  4. Automate. Make the secure path the easy path. If fetching from Vault is harder than hardcoding, people will hardcode.

The Human Element

Technical solutions only work if the team uses them. A few things that help:

  • Make local development easy. A separate secrets file that’s gitignored, or a local Vault instance, or direnv loading from 1Password CLI. Whatever removes friction.
  • Document the “why.” People skip security measures when they don’t understand the risk. “Because compliance” is less motivating than “because this credential can delete production data.”
  • Rotate regularly. Not because the old credentials are compromised, but because rotation proves your automation works. The worst time to discover your rotation process is broken is during an incident.

Secrets management isn’t glamorous work. Nobody gets excited about credential rotation policies. But the alternative — a 3am incident response where you’re trying to figure out which of 47 services has the leaked API key — is worse.

Start with one secret store. Migrate incrementally. Make the secure path the default path. Future you will be grateful.