We’ve all done it. That first deployment where the database password lives in a .env file. The API key hardcoded “just for testing.” The SSH key committed to the repo because you were moving fast.

Environment variables as secrets storage is the gateway drug of bad security practices. Let’s talk about what actually works.

The Problem with Environment Variables

Environment variables seem safe. They’re not in the code, right? But consider:

1
2
3
4
5
6
7
8
9

# Anyone with shell access can see them
printenv | grep -i password

# Process listings can expose them
ps auxe | grep myapp

# Crash dumps include them
# Logging frameworks often dump them
# Child processes inherit them

They’re also static. Rotating a secret means restarting services. In a world of zero-trust security and compliance requirements, that’s not acceptable.

The Secrets Management Hierarchy

From “please don’t” to “production-ready”:

Level 0: Hardcoded (Don’t)

1
2

# Dear god no
DB_PASSWORD = "hunter2"

This is in your git history forever. Even if you delete it, git log -p remembers.

Level 1: Environment Variables (Minimum)

1

export DB_PASSWORD="actually_secret"

Better than hardcoded, but still problematic. Use this for local development only.

Level 2: Encrypted Files (Getting Warmer)

Tools like sops, age, or ansible-vault:

1
2
3
4
5

# Encrypt secrets
sops -e secrets.yaml > secrets.enc.yaml

# Decrypt at deploy time
sops -d secrets.enc.yaml | kubectl apply -f -

The encryption key still needs to live somewhere, but at least secrets aren’t plaintext in your repo.

Level 3: Secret Managers (Production-Ready)

This is where you should be:

HashiCorp Vault:

1
2
3
4
5

# Store a secret
vault kv put secret/myapp/db password="s3cur3"

# Application retrieves it at runtime
vault kv get -field=password secret/myapp/db

AWS Secrets Manager:

1
2
3
4

import boto3

client = boto3.client('secretsmanager')
secret = client.get_secret_value(SecretId='myapp/database')

Azure Key Vault, GCP Secret Manager: Similar patterns.

What Makes Secret Managers Worth It?

1. Dynamic Secrets

Vault can generate database credentials on the fly:

1
2

vault read database/creds/my-role
# Returns a username/password that expires in 1 hour

No more shared credentials. No more “who has the production password.” Each service gets unique, time-limited credentials.

2. Audit Logging

Every secret access is logged:

1
2
3
4
5
6

{
  "time": "2026-03-03T04:00:00Z",
  "auth": {"entity_id": "app-server-01"},
  "request": {"path": "secret/data/myapp/db"},
  "response": {"succeeded": true}
}

When security asks “who accessed what,” you have answers.

3. Rotation Without Restarts

Update a secret in Vault, and applications can pick up the new value without redeployment. Most client libraries support watching for changes.

4. Access Policies

1
2
3
4
5
6
7

# Only production servers can read production secrets
path "secret/data/prod/*" {
  capabilities = ["read"]
  allowed_parameters = {
    "role" = ["production"]
  }
}

Developers can’t accidentally (or intentionally) pull production credentials to their laptop.

Practical Implementation Pattern

Here’s how I typically structure secrets in a Kubernetes environment:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# External Secrets Operator syncs from Vault/AWS/etc
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: myapp-secrets
  data:
    - secretKey: db-password
      remoteRef:
        key: secret/data/myapp/db
        property: password

The application sees a regular Kubernetes secret. The sync is automatic. Rotation happens in Vault, propagates to the cluster, and the app reloads.

The Migration Path

You don’t have to boil the ocean. Start here:

  1. Audit: Where do secrets currently live? Environment variables, config files, committed code?

  2. Centralize: Pick one secret manager. Migrate the most sensitive secrets first (database credentials, API keys for payment processors).

  3. Automate: Set up CI/CD to pull from the secret manager, not from environment variables or encrypted files.

  4. Rotate: Implement rotation policies. Even if you start with 90-day rotation, that’s better than never.

  5. Monitor: Alert on unusual secret access patterns.

Common Mistakes

“We’ll just use Kubernetes secrets”: They’re base64 encoded, not encrypted at rest by default. Anyone with cluster access can read them.

“Vault is overkill for our small team”: AWS Secrets Manager and similar cloud-native options have minimal operational overhead. You don’t need to run infrastructure.

“We rotate secrets annually”: If a secret is compromised, you’ve got up to a year of exposure. Aim for 30-90 days minimum.

“Our developers need production secrets for debugging”: No, they need read-only replicas with sanitized data. Production secrets should flow one direction: into production.

The Security Mindset Shift

Secrets management isn’t about preventing breaches (though it helps). It’s about limiting blast radius.

When (not if) something goes wrong:

  • How many systems are affected?
  • How long were compromised credentials valid?
  • Can you prove who accessed what?

Good secrets management turns “catastrophic breach” into “contained incident.”

Start with your most critical secrets. Get one thing right before expanding. Perfect is the enemy of deployed.