Infrastructure

When you have one server, you SSH in and grep the logs. When you have fifty servers, that stops working. Log aggregation is how you make “what happened?” answerable at scale. The Pipeline Architecture Every log aggregation system follows the same basic pattern: ┌ │ └ ─ ─ ─ S ─ ─ o ─ ─ u ─ ─ r ─ │ │ └ ─ c ─ ─ ─ e ─ ─ ─ s ─ ─ ─ ─ ─ ┐ │ ┘ ─ ─ ─ ─ ─ ─ ─ ▶ ▶ ┌ │ └ ┌ │ └ ─ ─ ─ ─ ─ C ─ ─ ─ ─ o ─ ─ Q ─ ─ l ─ ─ u ─ ─ l ─ ─ e ─ ─ e ─ ─ r ─ ─ c ─ ─ y ─ ─ t ─ ─ ─ ─ ─ ─ ─ ┐ │ ┘ ┐ │ ┘ ─ ◀ ─ ─ ─ ─ ▶ ─ ┌ │ └ ─ ─ ─ ─ ─ P ─ ─ ─ r ─ ─ ─ o ─ ─ ─ c ─ ─ ─ e ─ ─ ─ s ─ ─ ─ s ─ ─ ─ ─ ─ ┐ │ ┘ ─ ─ ─ ─ ─ ─ ─ ▶ ─ ┌ │ └ ─ ─ ─ ─ ─ ─ ─ ─ S ─ ─ ─ t ─ ─ ─ o ─ │ ┘ ─ r ─ │ ─ e ─ ─ ─ ─ ─ ┐ │ ┘ Each stage has choices. Let’s walk through them. ...

Most production incidents I’ve debugged came down to configuration. A missing environment variable. A wrong database URL. A feature flag stuck in the wrong state. Code was fine; configuration was the problem. Configuration management is the unsexy work that prevents those 3 AM pages. The Core Principles 1. Separate Configuration from Code Configuration should never be baked into your application binary or container image. Wrong: 1 2 # Hardcoded in code DATABASE_URL = "postgres://prod:password@db.example.com/myapp" Also wrong: ...

The scariest moment in software delivery used to be clicking “deploy.” Will it work? Will it break? Will you be debugging at 2 AM? Blue-green deployments eliminate most of that fear. Instead of updating your production environment in place, you deploy to an identical standby environment and switch traffic over. If something’s wrong, you switch back. Done. The Core Concept You maintain two identical production environments: Blue: Currently serving live traffic Green: Idle, ready for the next release To deploy: ...

We’ve all done it. Committed an API key to git. Hardcoded a database password “just for testing.” Posted a screenshot with credentials visible in the corner. The security community has a name for this: Tuesday. But secrets management doesn’t have to be painful. Let’s walk through the progression from “please no” to “actually reasonable” in handling sensitive credentials. The Hierarchy of Secrets (From Worst to Best) Level 0: Hardcoded in Source 1 2 3 # Don't do this. Ever. db_password = "hunter2" api_key = "sk-live-definitely-real-key" This is how breaches happen. Credentials in source code get committed to git, pushed to GitHub, indexed by bots within minutes, and suddenly someone’s mining crypto on your AWS account. ...

DNS is invisible until it breaks. Then everything breaks. Understanding how DNS works helps you debug issues faster and configure services correctly. The Basics DNS translates human-readable names to IP addresses: e x a m p l e . c o m → 9 3 . 1 8 4 . 2 1 6 . 3 4 The resolution process: 1 2 3 4 5 . . . . . B O Q R A r S u e n o e s s w c r o w s h y l e e e v r r c g e k o r c c s e a h s q c e i u h c t t e e k s o r d s i c c e a l a o s t o c n c h f r e a e i o a l g o c u t h c r a e s l c d e e h r v e r v e e e l s r o s b l a v → s e e r T d L ( D o e n . s g e T . r T , v L e 8 r . s 8 . → 8 . a 8 u ) t h o r i t a t i v e s e r v e r s Record Types A Record (IPv4 Address) e x a m p l e . c o m . 3 0 0 I N A 9 3 . 1 8 4 . 2 1 6 . 3 4 Maps a name to an IPv4 address. ...

Redis gets introduced as a cache, but that undersells it. It’s an in-memory data structure server with atomic operations, pub/sub, streams, and more. These patterns show Redis’s real power. Basic Caching (The Familiar One) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 import redis import json r = redis.Redis(host='localhost', port=6379, decode_responses=True) def get_user(user_id): # Check cache first cached = r.get(f"user:{user_id}") if cached: return json.loads(cached) # Miss: fetch from database user = db.query("SELECT * FROM users WHERE id = %s", user_id) # Cache with TTL r.setex(f"user:{user_id}", 3600, json.dumps(user)) return user Rate Limiting Sliding window rate limiter with sorted sets: ...

Nginx is everywhere: reverse proxy, load balancer, static file server, SSL terminator. Its configuration syntax is powerful but has gotchas that catch everyone at least once. These patterns cover common use cases done right. Basic Reverse Proxy 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 upstream backend { server 127.0.0.1:3000; } server { listen 80; server_name example.com; location / { proxy_pass http://backend; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } } Always set those headers — your backend needs to know the real client IP and protocol. ...

Every infrastructure team eventually faces the same uncomfortable question: where do the secrets go? API keys, database passwords, TLS certificates, OAuth tokens — they all need to live somewhere. The wrong answer (“in the repo, it’s fine, it’s private”) creates technical debt that compounds silently until someone accidentally pushes to public or an ex-employee still has access to production credentials. The Anti-Patterns First Environment variables everywhere. Yes, the twelve-factor app says config comes from the environment. But “the environment” doesn’t mean a .env file committed to git. Environment variables are runtime config, not secret storage. ...

Systemd is how modern Linux manages services. It starts your applications at boot, restarts them when they crash, and handles dependencies between services. Understanding systemd transforms “it works when I run it manually” into “it runs reliably in production.” Basic Service Unit 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # /etc/systemd/system/myapp.service [Unit] Description=My Application After=network.target [Service] Type=simple User=appuser WorkingDirectory=/opt/myapp ExecStart=/opt/myapp/bin/server Restart=always RestartSec=5 [Install] WantedBy=multi-user.target 1 2 3 4 # Enable and start sudo systemctl daemon-reload sudo systemctl enable myapp sudo systemctl start myapp Service Types simple (default): Process started by ExecStart is the main process. ...

Everyone has backups. Few have tested restores. The backup that fails during a crisis is worse than no backup — it gave you false confidence. Backup strategy isn’t about the backup. It’s about the restore. The 3-2-1 Rule A minimum viable backup strategy: 3 copies of your data 2 different storage media/types 1 copy offsite P C C C r o o o i p p p m y y y a r 1 2 3 y : : : : P L R O r o e f o c m f d a o s u l t i c e t t s e i n r o a e a n p p r s l c d h i h a o c i t t a v a e b ( ( a s d ( s a i d e m f i e f f e f d r e a e r t n e a t n c t e r n e p t g r e i o r o v ) n i ) d e r ) What to Back Up Always: ...