The Readiness Probe That Turned a Deploy Into an Outage

A routine deploy. The new pods come up, and for ninety seconds the entire service returns 503 — even though the previous version was perfectly healthy the whole time. The culprit wasn’t the new code. It was a readiness probe pointed at the wrong thing.

Readiness and liveness are opposites, not synonyms

A readiness probe decides whether a pod receives traffic. A liveness probe decides whether a pod gets killed and restarted. Teams wire them identically and treat them as interchangeable, but they want opposite temperaments: liveness should be lazy and forgiving; readiness should be quick and honest. Confuse them and a slow-starting app either gets killed mid-boot because liveness is too aggressive, or gets declared “ready” before it can actually serve.

The rollout death spiral

Here’s how a bad readiness probe takes down a healthy service. The new ReplicaSet’s pods fail readiness. The rolling update, following maxUnavailable, has already begun retiring old pods — but the new ones never become ready to replace them. Endpoints drain faster than they fill. For a window, almost nothing is behind the Service, and clients get 503s from the exact mechanism that’s supposed to prevent this.

# The trap: readiness hits a downstream dependency, with no warm-up
readinessProbe:
  httpGet:
    path: /healthz        # this handler also pings the database
    port: 8080
  initialDelaySeconds: 0  # checked before the app can even listen
  periodSeconds: 5
  failureThreshold: 1     # one blip = pulled from rotation

Two mistakes compound. The health endpoint checks a downstream dependency, so a brief database hiccup marks every replica unready at once — a self-inflicted full outage. And failureThreshold: 1 with no startup grace means the smallest transient becomes an eviction from the load balancer.

The fix: a shallow check plus a startupProbe

# Give slow starters room without loosening liveness
startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30
  periodSeconds: 5          # up to 150s to boot, then normal probes take over
readinessProbe:
  httpGet:
    path: /ready            # answers ONLY "can this process serve?" — no deep deps
    port: 8080
  periodSeconds: 5
  failureThreshold: 3

Split the endpoints. /ready answers “is this process able to serve a request,” nothing more — don’t fold the database into it, or one slow dependency yanks every pod out of rotation simultaneously. Use a startupProbe to grant slow-booting apps their warm-up time instead of weakening liveness. And set maxUnavailable: 0 on critical rollouts so Kubernetes won’t retire an old pod until a new one is genuinely ready.

The lesson

Readiness is a promise to your load balancer, not a general health dashboard. Make it answer the narrowest possible question — “send me traffic now?” — and nothing else. Every dependency you fold into that check is another way for one small failure to become a total one.

Probe configs and a reproducible rollout demo are on GitHub: github.com/waghmaredb/vexpose-labs. Got burned by a probe in a way I didn’t cover? Tell me on LinkedIn or X.