Secrets Management: Keeping Credentials Out of Your Code

Hardcoded credentials in your repository are a security incident waiting to happen. One leaked .env file, one accidental commit, and your database is exposed to the internet.

Let’s do secrets properly.

The Basics

What’s a Secret?

Anything that grants access:

Database passwords
API keys
OAuth tokens
TLS certificates
SSH keys
Encryption keys

Where Secrets Don’t Belong

1
2
3
# ❌ Never do this
DATABASE_URL = "postgres://admin:supersecret123@db.prod.internal/myapp"
AWS_SECRET_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

Also bad:

.env files committed to git
Docker image layers
CI/CD logs
Chat messages
Wikis or documentation

Secret Storage Options

Environment Variables

Simple, but limited:

1
2
3
4
5
6
7
# Set in shell
export DATABASE_PASSWORD="supersecret"

# Or in systemd unit
[Service]
Environment="DATABASE_PASSWORD=supersecret"
EnvironmentFile=/etc/myapp/secrets

1
2
import os
password = os.environ.get("DATABASE_PASSWORD")

Pros: Simple, works everywhere Cons: Visible in process listings, no rotation, no audit

HashiCorp Vault

The gold standard for secrets management:

1
2
3
4
5
6
7
8
# Start dev server
vault server -dev

# Store a secret
vault kv put secret/myapp/database password="supersecret"

# Retrieve it
vault kv get -field=password secret/myapp/database

Application integration:

1
2
3
4
5
6
7
8
9
import hvac

client = hvac.Client(url='https://vault.internal:8200')
client.token = os.environ['VAULT_TOKEN']

secret = client.secrets.kv.v2.read_secret_version(
    path='myapp/database'
)
password = secret['data']['data']['password']

Vault features:

Dynamic secrets (generate on-demand)
Automatic rotation
Audit logging
Fine-grained access policies
Encryption as a service

AWS Secrets Manager

Managed service, tight AWS integration:

1
2
3
4
5
6
7
8
9
# Store a secret
aws secretsmanager create-secret \
  --name prod/myapp/database \
  --secret-string '{"username":"admin","password":"supersecret"}'

# Retrieve it
aws secretsmanager get-secret-value \
  --secret-id prod/myapp/database \
  --query SecretString --output text

1
2
3
4
5
6
7
import boto3
import json

client = boto3.client('secretsmanager')
response = client.get_secret_value(SecretId='prod/myapp/database')
secrets = json.loads(response['SecretString'])
password = secrets['password']

Built-in rotation for:

RDS databases
DocumentDB
Redshift
Custom Lambda functions

AWS Parameter Store

Lighter weight, lower cost:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Store (encrypted)
aws ssm put-parameter \
  --name /prod/myapp/database_password \
  --value "supersecret" \
  --type SecureString

# Retrieve
aws ssm get-parameter \
  --name /prod/myapp/database_password \
  --with-decryption \
  --query Parameter.Value --output text

Kubernetes Secrets

Native K8s solution:

1
2
3
4
5
6
7
8
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: YWRtaW4=  # base64 encoded
  password: c3VwZXJzZWNyZXQ=

1
2
3
4
5
6
7
# Use in pod
env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: password

Warning: K8s secrets are base64 encoded, not encrypted. Use:

Sealed Secrets (Bitnami)
External Secrets Operator
Vault integration

Secret Injection Patterns

Sidecar Pattern

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# Vault Agent sidecar
containers:
  - name: vault-agent
    image: vault:latest
    args:
      - agent
      - -config=/etc/vault/config.hcl
    volumeMounts:
      - name: secrets
        mountPath: /secrets
        
  - name: app
    image: myapp:latest
    volumeMounts:
      - name: secrets
        mountPath: /secrets
        readOnly: true

Vault Agent writes secrets to shared volume; app reads them.

Init Container Pattern

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
initContainers:
  - name: fetch-secrets
    image: amazon/aws-cli
    command:
      - sh
      - -c
      - |
        aws secretsmanager get-secret-value \
          --secret-id prod/myapp/database \
          --query SecretString --output text > /secrets/db.json
    volumeMounts:
      - name: secrets
        mountPath: /secrets

External Secrets Operator

Sync cloud secrets to Kubernetes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: prod/myapp/database
        property: password

Secret Rotation

Why Rotate?

Limits exposure window if compromised
Compliance requirements
Principle of least privilege over time

AWS RDS Automatic Rotation

1
2
3
4
aws secretsmanager rotate-secret \
  --secret-id prod/myapp/database \
  --rotation-lambda-arn arn:aws:lambda:...:function:SecretsManagerRDSRotation \
  --rotation-rules AutomaticallyAfterDays=30

Vault Dynamic Secrets

Vault generates credentials on-demand with automatic expiration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Configure database backend
vault write database/config/mydb \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@db:5432/myapp" \
  allowed_roles="readonly"

# Create role
vault write database/roles/readonly \
  db_name=mydb \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';" \
  default_ttl="1h" \
  max_ttl="24h"

# App requests credentials (fresh each time)
vault read database/creds/readonly

Each credential is unique and auto-expires.

Application Considerations

Apps must handle rotation gracefully:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
class DatabaseConnection:
    def __init__(self, secret_id):
        self.secret_id = secret_id
        self.refresh_credentials()
    
    def refresh_credentials(self):
        self.credentials = get_secret(self.secret_id)
        self.connection = create_connection(self.credentials)
    
    def execute(self, query):
        try:
            return self.connection.execute(query)
        except AuthenticationError:
            # Credential rotated, refresh and retry
            self.refresh_credentials()
            return self.connection.execute(query)

CI/CD Secrets

GitHub Actions

1
2
3
4
5
6
7
8
jobs:
  deploy:
    steps:
      - name: Deploy
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: ./deploy.sh

Better: Use OIDC for temporary credentials:

1
2
3
4
5
6
7
8
9
permissions:
  id-token: write
  contents: read

steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789:role/github-actions
      aws-region: us-east-1

GitLab CI

1
2
3
4
5
deploy:
  script:
    - ./deploy.sh
  variables:
    DATABASE_PASSWORD: $DATABASE_PASSWORD  # From CI/CD settings

Detection and Prevention

Git Hooks (Pre-commit)

1
2
3
4
5
6
7
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

Gitleaks

1
2
3
4
5
# Scan repository
gitleaks detect --source . --verbose

# In CI
gitleaks detect --source . --redact --exit-code 1

TruffleHog

1
2
3
4
5
# Scan git history
trufflehog git file://. --only-verified

# Scan GitHub org
trufflehog github --org=mycompany

Emergency Response

When a secret is leaked:

Revoke immediately - Don’t wait
Rotate - Generate new credentials
Audit - Check for unauthorized access
Remediate - Remove from git history if needed
Post-mortem - How did this happen?

1
2
3
4
5
6
7
# Remove secret from git history (nuclear option)
git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch path/to/secret" \
  --prune-empty --tag-name-filter cat -- --all

# Force push (coordinate with team!)
git push --force --all

Quick Reference

Solution	Best For	Rotation	Audit
Env vars	Simple apps	Manual	None
Vault	Enterprise	Automatic	Full
AWS Secrets Manager	AWS native	Automatic	CloudTrail
K8s Secrets	K8s workloads	Manual	API audit

Checklist

No secrets in source code
No secrets in Docker images
Pre-commit hooks detect secrets
Secrets stored in dedicated manager
Rotation policy defined
Access audited
Emergency rotation procedure documented

Treat every secret like it’s already been compromised. Store securely, rotate regularly, and always have a revocation plan.

The Basics#

What’s a Secret?#

Where Secrets Don’t Belong#

Secret Storage Options#

Environment Variables#

HashiCorp Vault#

AWS Secrets Manager#

AWS Parameter Store#

Kubernetes Secrets#

Secret Injection Patterns#

Sidecar Pattern#

Init Container Pattern#

External Secrets Operator#

Secret Rotation#

Why Rotate?#

AWS RDS Automatic Rotation#

Vault Dynamic Secrets#

Application Considerations#

CI/CD Secrets#

GitHub Actions#

GitLab CI#

Detection and Prevention#

Git Hooks (Pre-commit)#

Gitleaks#

TruffleHog#

Emergency Response#

Quick Reference#

Checklist#

📬 Get the Newsletter