SSL/TLS Certificate Management: Avoiding the 3 AM Expiry Crisis

Nothing ruins a morning like discovering your certificate expired overnight and customers are seeing security warnings. Let’s prevent that.

Certificate Basics

What You Actually Need

A certificate contains:

Your domain name(s)
Your public key
Certificate Authority’s signature
Expiration date

1
2
3
4
5
# View certificate details
openssl x509 -in cert.pem -text -noout

# Check what's actually served
openssl s_client -connect example.com:443 -servername example.com | openssl x509 -text -noout

Certificate Types

DV (Domain Validation): Proves you control the domain. Cheapest, fastest.

OV (Organization Validation): Proves organization exists. Shows company name.

EV (Extended Validation): Thorough vetting. Used to show green bar (browsers removed this).

For most purposes, DV is fine. Let’s Encrypt provides them free.

Let’s Encrypt + Certbot

The standard for automated certificates:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Install certbot
sudo apt install certbot python3-certbot-nginx

# Get certificate (with nginx plugin)
sudo certbot --nginx -d example.com -d www.example.com

# Get certificate (standalone)
sudo certbot certonly --standalone -d example.com

# Get certificate (DNS challenge - for wildcards)
sudo certbot certonly --manual --preferred-challenges dns -d "*.example.com"

Automatic Renewal

Certbot installs a timer/cron automatically:

1
2
3
4
5
# Check timer status
systemctl status certbot.timer

# Test renewal
sudo certbot renew --dry-run

The timer runs twice daily and renews certificates expiring within 30 days.

Renewal Hooks

Run scripts after renewal:

1
2
3
# /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
#!/bin/bash
systemctl reload nginx

Or in the renewal config:

1
2
3
# /etc/letsencrypt/renewal/example.com.conf
[renewalparams]
post_hook = systemctl reload nginx

ACME Clients for Different Platforms

acme.sh (Shell)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Install
curl https://get.acme.sh | sh

# Issue certificate
acme.sh --issue -d example.com -w /var/www/html

# With DNS API (Cloudflare example)
export CF_Token="your-api-token"
acme.sh --issue --dns dns_cf -d example.com -d "*.example.com"

# Install to nginx
acme.sh --install-cert -d example.com \
  --key-file /etc/nginx/ssl/key.pem \
  --fullchain-file /etc/nginx/ssl/cert.pem \
  --reloadcmd "systemctl reload nginx"

cert-manager (Kubernetes)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml

# Create ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

# Request certificate via Ingress annotation
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - example.com
    secretName: example-com-tls

AWS Certificate Manager

For AWS services (ALB, CloudFront, API Gateway):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Request certificate (validation via DNS)
aws acm request-certificate \
  --domain-name example.com \
  --subject-alternative-names "*.example.com" \
  --validation-method DNS

# List certificates
aws acm list-certificates

# ACM auto-renews - no action needed

Note: ACM certificates only work with AWS services, not EC2 directly.

Monitoring Expiration

Prometheus + Blackbox Exporter

1
2
3
4
5
6
7
# blackbox.yml
modules:
  https_cert:
    prober: http
    http:
      fail_if_ssl: false
      fail_if_not_ssl: true

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# prometheus.yml
scrape_configs:
  - job_name: 'ssl-certs'
    metrics_path: /probe
    params:
      module: [https_cert]
    static_configs:
      - targets:
        - https://example.com
        - https://api.example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

Alert on expiration:

1
2
3
4
5
6
7
- alert: SSLCertExpiringSoon
  expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 14
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "SSL certificate expires in {{ $value | humanizeDuration }}"

Simple Script Check

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
#!/bin/bash
# check-ssl-expiry.sh

DOMAINS="example.com api.example.com admin.example.com"
WARN_DAYS=30

for domain in $DOMAINS; do
  expiry=$(echo | openssl s_client -servername "$domain" -connect "$domain:443" 2>/dev/null | \
           openssl x509 -noout -enddate | cut -d= -f2)
  expiry_epoch=$(date -d "$expiry" +%s)
  now_epoch=$(date +%s)
  days_left=$(( (expiry_epoch - now_epoch) / 86400 ))
  
  if [ $days_left -lt $WARN_DAYS ]; then
    echo "WARNING: $domain expires in $days_left days"
  else
    echo "OK: $domain expires in $days_left days"
  fi
done

Run via cron:

1
0 9 * * * /usr/local/bin/check-ssl-expiry.sh | mail -s "SSL Expiry Report" admin@example.com

Common Pitfalls

Chain Issues

Serve the full certificate chain, not just your cert:

1
2
3
4
5
6
7
8
9
# Check chain
openssl s_client -connect example.com:443 -servername example.com

# Should show:
# Certificate chain
#  0 s:CN = example.com
#    i:C = US, O = Let's Encrypt, CN = R3
#  1 s:C = US, O = Let's Encrypt, CN = R3
#    i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

If intermediate is missing, some clients fail. Use the fullchain file:

1
2
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

Wrong Hostname

Certificate must match the requested hostname:

Use SAN (Subject Alternative Names) or wildcard:

1
2
3
4
5
# Multiple specific domains
certbot -d example.com -d www.example.com -d api.example.com

# Wildcard (requires DNS challenge)
certbot --manual --preferred-challenges dns -d "*.example.com" -d example.com

Renewal Fails Silently

Test your renewal actually works:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Force renewal test
sudo certbot renew --dry-run

# Check logs
sudo journalctl -u certbot -f

# Common issues:
# - Port 80 blocked (http-01 challenge)
# - Wrong DNS (dns-01 challenge)
# - Webroot path doesn't exist

Key Pinning Gone Wrong

Don’t pin certificates or public keys unless you really know what you’re doing. Certificate rotation becomes a nightmare.

Strong TLS Configuration

Nginx

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;

ssl_session_timeout 1d;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;

ssl_stapling on;
ssl_stapling_verify on;

add_header Strict-Transport-Security "max-age=63072000" always;

Test Your Configuration

1
2
3
4
5
6
7
8
# Mozilla SSL Test
# https://www.ssllabs.com/ssltest/

# Command line
nmap --script ssl-enum-ciphers -p 443 example.com

# testssl.sh
docker run --rm -ti drwetter/testssl.sh example.com

Certificate Inventory

Keep track of all your certificates:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
# certificates.yml
certificates:
  - domain: example.com
    provider: letsencrypt
    type: wildcard
    expires: 2026-06-01
    managed_by: certbot
    servers:
      - web1.internal
      - web2.internal
    
  - domain: api.example.com  
    provider: aws-acm
    type: single
    expires: auto-renewed
    managed_by: acm
    services:
      - api-alb

Review monthly. Automate discovery:

1
2
3
4
5
6
7
# Find all certs on a server
find /etc -name "*.pem" -o -name "*.crt" 2>/dev/null | while read cert; do
  if openssl x509 -in "$cert" -noout 2>/dev/null; then
    expiry=$(openssl x509 -in "$cert" -noout -enddate | cut -d= -f2)
    echo "$cert: $expiry"
  fi
done

Quick Checklist

All production certs from trusted CA (not self-signed)
Automatic renewal configured and tested
Monitoring alerts before expiration (14+ days)
Full chain served, not just leaf certificate
TLS 1.2+ only, no legacy protocols
HSTS header enabled
Certificate inventory maintained
Renewal tested with --dry-run

An expired certificate is the most preventable outage. Automate renewal, monitor expiration, and test your automation.

Certificate Basics#

What You Actually Need#

Certificate Types#

Let’s Encrypt + Certbot#

Automatic Renewal#

Renewal Hooks#

ACME Clients for Different Platforms#

acme.sh (Shell)#

cert-manager (Kubernetes)#

AWS Certificate Manager#

Monitoring Expiration#

Prometheus + Blackbox Exporter#

Simple Script Check#

Common Pitfalls#

Chain Issues#

Wrong Hostname#

Renewal Fails Silently#

Key Pinning Gone Wrong#

Strong TLS Configuration#

Nginx#

Test Your Configuration#

Certificate Inventory#

Quick Checklist#

📬 Get the Newsletter