Secrets Management: Stop Committing Your API Keys

We’ve all done it. Committed a database password. Pushed an API key. Then frantically force-pushed hoping nobody noticed. Here’s how to manage secrets properly so that never happens again.

The Problem

1
2
3
4
5
6
7
8
9
# Bad: Secrets in code
DATABASE_URL="postgres://admin:supersecret@db.example.com/prod"

# Bad: Secrets in .env checked into git
# .env
API_KEY=sk-live-abc123

# Bad: Secrets in CI/CD logs
echo "Deploying with $DATABASE_PASSWORD"

Secrets in code get leaked. Always. It’s just a matter of when.

Secret Types

Secret	Rotation Frequency	Storage
Database passwords	Quarterly	Vault/Secrets Manager
API keys	When compromised	Vault/Secrets Manager
SSL certificates	Before expiry	Cert Manager/Vault
OAuth tokens	Short-lived	Generated at runtime
Encryption keys	Rarely	HSM/KMS

Environment Variables (Minimum Viable)

1
2
3
4
5
6
# Don't commit .env files
echo ".env" >> .gitignore

# .env.example (committed, no real values)
DATABASE_URL=postgres://user:pass@host/db
API_KEY=your-api-key-here

1
2
3
4
5
6
# Load at runtime
import os
from dotenv import load_dotenv

load_dotenv()
db_url = os.getenv("DATABASE_URL")

Better than hardcoding, but secrets still exist as plaintext files on disk.

HashiCorp Vault

Setup

1
2
3
4
5
# Start dev server (not for production)
vault server -dev

# Production: Use HA backend
vault server -config=/etc/vault/config.hcl

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# /etc/vault/config.hcl
storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/etc/vault/cert.pem"
  tls_key_file  = "/etc/vault/key.pem"
}

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "alias/vault-unseal"
}

Store Secrets

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# KV secrets engine (version 2)
vault secrets enable -path=secret kv-v2

# Store a secret
vault kv put secret/myapp/database \
  username="dbuser" \
  password="supersecret"

# Read a secret
vault kv get secret/myapp/database

# JSON output
vault kv get -format=json secret/myapp/database | jq '.data.data'

Application Integration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
import hvac

client = hvac.Client(url='https://vault.example.com:8200')
client.token = os.getenv('VAULT_TOKEN')

# Read secret
secret = client.secrets.kv.v2.read_secret_version(
    path='myapp/database',
    mount_point='secret'
)

db_password = secret['data']['data']['password']

Dynamic Secrets

Vault generates credentials on-demand with automatic expiration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Enable database secrets engine
vault secrets enable database

# Configure PostgreSQL connection
vault write database/config/mydb \
  plugin_name=postgresql-database-plugin \
  allowed_roles="readonly,readwrite" \
  connection_url="postgresql://{{username}}:{{password}}@db.example.com:5432/mydb" \
  username="vault" \
  password="vault-password"

# Create a role
vault write database/roles/readonly \
  db_name=mydb \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Get dynamic credentials
vault read database/creds/readonly
# Returns: username=v-token-readonly-abc123, password=xyz789
# Expires in 1 hour, auto-revoked

AWS Secrets Manager

Store Secrets

1
2
3
4
5
6
7
8
9
# Create secret
aws secretsmanager create-secret \
  --name myapp/database \
  --secret-string '{"username":"admin","password":"supersecret"}'

# Update secret
aws secretsmanager put-secret-value \
  --secret-id myapp/database \
  --secret-string '{"username":"admin","password":"newsecret"}'

Retrieve in Application

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
import boto3
import json

def get_secret(secret_name):
    client = boto3.client('secretsmanager')
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response['SecretString'])

secrets = get_secret('myapp/database')
db_password = secrets['password']

Automatic Rotation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# Lambda rotation function
def lambda_handler(event, context):
    secret_id = event['SecretId']
    step = event['Step']
    
    if step == "createSecret":
        # Generate new password
        new_password = generate_password()
        client.put_secret_value(
            SecretId=secret_id,
            ClientRequestToken=event['ClientRequestToken'],
            SecretString=json.dumps({"password": new_password}),
            VersionStages=['AWSPENDING']
        )
    
    elif step == "setSecret":
        # Update the actual resource (database, etc.)
        pending = get_secret_version(secret_id, 'AWSPENDING')
        update_database_password(pending['password'])
    
    elif step == "testSecret":
        # Verify new credentials work
        pending = get_secret_version(secret_id, 'AWSPENDING')
        test_database_connection(pending['password'])
    
    elif step == "finishSecret":
        # Promote pending to current
        client.update_secret_version_stage(
            SecretId=secret_id,
            VersionStage='AWSCURRENT',
            MoveToVersionId=event['ClientRequestToken'],
            RemoveFromVersionId=get_current_version(secret_id)
        )

Kubernetes Secrets

Basic Secrets

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: myapp-secrets
type: Opaque
data:
  # Base64 encoded (NOT encrypted)
  database-password: c3VwZXJzZWNyZXQ=
  api-key: c2stbGl2ZS1hYmMxMjM=

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# Use in pod
apiVersion: v1
kind: Pod
spec:
  containers:
    - name: app
      env:
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: myapp-secrets
              key: database-password

Sealed Secrets (GitOps Safe)

1
2
3
4
5
6
7
8
# Install kubeseal
kubeseal --fetch-cert > pub-cert.pem

# Seal a secret (encrypted with cluster's public key)
kubectl create secret generic myapp-secrets \
  --from-literal=password=supersecret \
  --dry-run=client -o yaml | \
  kubeseal --cert pub-cert.pem -o yaml > sealed-secret.yaml

1
2
3
4
5
6
7
8
# sealed-secret.yaml (safe to commit)
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: myapp-secrets
spec:
  encryptedData:
    password: AgBy3i4O...encrypted...

External Secrets Operator

Pull secrets from Vault/AWS into Kubernetes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: myapp-secrets
  data:
    - secretKey: database-password
      remoteRef:
        key: secret/myapp/database
        property: password

SOPS (Secrets in Git)

Encrypt secrets in place, commit encrypted files.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
# Install sops
brew install sops

# Create .sops.yaml
creation_rules:
  - path_regex: \.enc\.yaml$
    kms: arn:aws:kms:us-east-1:123456789:key/abc-123
    # Or use age/pgp
    # age: age1...

# Encrypt
sops -e secrets.yaml > secrets.enc.yaml

# Edit encrypted file
sops secrets.enc.yaml

# Decrypt in CI
sops -d secrets.enc.yaml > secrets.yaml

1
2
3
4
5
6
7
8
# secrets.enc.yaml (safe to commit)
database:
    password: ENC[AES256_GCM,data:abc123...,type:str]
    username: ENC[AES256_GCM,data:def456...,type:str]
sops:
    kms:
        - arn: arn:aws:kms:us-east-1:123456789:key/abc-123
    version: 3.7.3

CI/CD Secrets

GitHub Actions

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# .github/workflows/deploy.yml
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Deploy
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
          API_KEY: ${{ secrets.API_KEY }}
        run: |
          # Secrets available as env vars
          ./deploy.sh

GitLab CI

1
2
3
4
5
6
# .gitlab-ci.yml
deploy:
  script:
    - echo "Deploying..."
  variables:
    DATABASE_URL: $DATABASE_URL  # From CI/CD settings

Avoid Printing Secrets

1
2
3
4
5
6
7
# Bad: Might leak in logs
- run: echo ${{ secrets.API_KEY }}

# Good: Mask automatically
- run: |
    # GitHub automatically masks secrets in logs
    curl -H "Authorization: Bearer $API_KEY" https://api.example.com

Secret Scanning

Pre-commit Hooks

1
2
3
4
5
6
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.0
    hooks:
      - id: gitleaks

CI Scanning

1
2
3
4
5
# GitHub Actions
- name: Scan for secrets
  uses: gitleaks/gitleaks-action@v2
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Detect Committed Secrets

1
2
3
4
5
# Scan history
gitleaks detect --source . --verbose

# Scan specific commits
gitleaks detect --source . --log-opts="HEAD~10..HEAD"

When Secrets Leak

Rotate immediately: Change the compromised secret
Revoke: Invalidate the old secret
Audit: Check for unauthorized access
Remediate: Remove from git history if needed

1
2
3
4
5
6
7
# Remove from git history (nuclear option)
git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch path/to/secret.env" \
  --prune-empty --tag-name-filter cat -- --all

# Force push (coordinate with team!)
git push origin --force --all

Better: Use BFG Repo-Cleaner (faster):

1
2
3
bfg --delete-files secret.env
git reflog expire --expire=now --all
git gc --prune=now --aggressive

The Checklist

No secrets in code or git
.env files in .gitignore
Secrets manager for production
Rotation policy defined
Pre-commit secret scanning
CI/CD secret scanning
Least-privilege access to secrets
Audit logging enabled

Start Here

Today: Add gitleaks pre-commit hook
This week: Move one hardcoded secret to env vars
This month: Set up Vault or Secrets Manager
This quarter: Implement automatic rotation

The goal: if your code leaks, your secrets don’t.

The best secret is one that doesn’t exist. The second best is one that rotates before anyone can use it.

The Problem#

Secret Types#

Environment Variables (Minimum Viable)#

HashiCorp Vault#

Setup#

Store Secrets#

Application Integration#

Dynamic Secrets#

AWS Secrets Manager#

Store Secrets#

Retrieve in Application#

Automatic Rotation#

Kubernetes Secrets#

Basic Secrets#

Sealed Secrets (GitOps Safe)#

External Secrets Operator#

SOPS (Secrets in Git)#

CI/CD Secrets#

GitHub Actions#

GitLab CI#

Avoid Printing Secrets#

Secret Scanning#

Pre-commit Hooks#

CI Scanning#

Detect Committed Secrets#

When Secrets Leak#

The Checklist#

Start Here#

📬 Get the Newsletter

The Problem

Secret Types

Environment Variables (Minimum Viable)

HashiCorp Vault

Setup

Store Secrets

Application Integration

Dynamic Secrets

AWS Secrets Manager

Store Secrets

Retrieve in Application

Automatic Rotation

Kubernetes Secrets

Basic Secrets

Sealed Secrets (GitOps Safe)

External Secrets Operator

SOPS (Secrets in Git)

CI/CD Secrets

GitHub Actions

GitLab CI

Avoid Printing Secrets

Secret Scanning

Pre-commit Hooks

CI Scanning

Detect Committed Secrets

When Secrets Leak

The Checklist

Start Here