Health-Checks

You wake up to angry messages. Your service has been down for hours. You check your monitoring dashboard — all green. What happened? The answer is almost always the same: your health check died with the thing it was checking. The Problem: Shared Failure Domains Here’s a common setup that looks correct but isn’t: ┌ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └ ─ ─ ─ ─ ─ ┌ │ │ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ( ─ ─ ─ ─ S p ─ ─ ─ ─ e o ─ ─ ─ ─ r r ─ ─ ─ ─ v t ─ │ └ ┌ │ │ │ └ ─ ─ ─ i ─ ─ ─ ─ ─ ─ ─ c 8 ─ ─ ─ ( ─ ─ ─ ─ e 0 ─ ─ ─ l c ─ ─ ─ ─ ) ─ ─ ─ T o l ─ ─ I ─ Y ─ ─ ─ ─ u c o ─ ─ n ─ o ─ ─ ─ ─ n a u ─ ─ t ─ u ┐ │ │ ┘ ─ ─ n l d ─ ─ e ─ r ┬ ▼ ─ e t f ─ │ │ ▼ r ─ ─ ─ l u l ─ ─ n ─ S ─ ─ n a ─ ─ e ─ e ─ ─ n r ─ ─ t ─ r ┌ │ │ └ ─ ─ e e ─ ─ ─ v ─ ─ ─ ─ l d ─ ─ ─ e ─ ─ ─ ┐ │ │ ) ┘ ─ ─ r ─ H ─ ─ │ ─ ─ ─ e ( ─ ─ ─ ─ ─ a c ─ ─ ─ ─ ─ l r ─ ─ ─ ─ ─ t o ─ ─ ─ ─ ─ h n ─ │ ┘ ─ ─ ─ ─ ─ ─ ─ C j ─ ─ ─ ─ h o ─ ─ ─ ─ e b ─ ─ ─ ─ c ) ─ ─ ─ ─ k ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ │ │ ┘ ─ ─ ─ ─ ─ ┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ┘ The health check runs on the same server, uses the same tunnel, and sends alerts through… the same tunnel. When the tunnel dies, both the service AND the alerting die together. ...

“Is everything working?” sounds like a simple question. It’s not. Here’s how to build health checks that give you real answers. What Health Checks Are For Health checks answer one question: Can this thing do its job right now? Not “is it running?” (that’s process monitoring). Not “did it work yesterday?” (that’s metrics). Not “will it work tomorrow?” (that’s capacity planning). Just: right now, can it serve traffic? The Levels of Health Level 1: Process Running Bare minimum — is the process alive? ...

Your load balancer routes traffic to a pod that’s crashed. Or kills a pod that’s just slow. Or restarts a pod that’s still initializing. Health checks prevent these failures — when configured correctly. Most teams get them wrong. Here’s how to get them right. The Three Probe Types Kubernetes offers three distinct probes, each with a different purpose: Probe Question Failure Action Liveness Is the process alive? Restart container Readiness Can it handle traffic? Remove from Service Startup Has it finished starting? Delay other probes Liveness: “Should I restart this?” Detects when your app is stuck — deadlocked, infinite loop, unrecoverable state. ...

Your application says it’s running. But is it actually working? Health checks answer that question. They’re the difference between “process exists” and “service is functional.” Get them wrong, and your orchestrator will either route traffic to broken instances or restart healthy ones. Three Types of Probes Liveness: “Is this process stuck?” Liveness probes detect deadlocks, infinite loops, and zombie processes. If liveness fails, the container gets killed and restarted. 1 2 3 4 5 6 7 livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 30 periodSeconds: 10 failureThreshold: 3 What to check: ...

A health check that always returns 200 OK is worse than no health check at all. It gives you false confidence while your application silently fails. Let’s build health checks that actually tell you when something’s wrong. The Three Types of Probes Kubernetes defines three probe types, each serving a distinct purpose: Liveness Probe: “Is this process stuck?” If it fails, Kubernetes kills and restarts the container. Readiness Probe: “Can this instance handle traffic?” If it fails, the instance is removed from load balancing but keeps running. ...