Kubernetes probes: liveness, readiness and startup

Kubernetes probes are three settings that look almost identical and behave completely differently. Getting them wrong is one of the most common causes of a deployment that either never becomes healthy or, worse, serves errors while reporting itself fine. Here is what each one does and how to configure them deliberately.

The three probes

Liveness — should this container be killed?

If the liveness probe fails, the kubelet restarts the container. That is the only thing it does. Use it for the narrow case of a process that is running but permanently stuck — a deadlock, an unrecoverable internal state — where a restart genuinely fixes things.

Readiness — should this pod receive traffic?

If the readiness probe fails, the pod is removed from service endpoints. It is not restarted. This is the probe you will use most: it covers startup warm-up, temporary dependency loss, and graceful shutdown.

Startup — has it finished booting yet?

While a startup probe is failing, the liveness and readiness probes are disabled entirely. Once it succeeds once, it never runs again and the other two take over. It exists for slow-starting applications so you do not have to inflate your liveness thresholds to accommodate boot time.

The mistake almost everyone makes

Pointing liveness and readiness at the same endpoint, and having that endpoint check dependencies.

# Don't do this
livenessProbe:
  httpGet: { path: /health, port: 8080 }
readinessProbe:
  httpGet: { path: /health, port: 8080 }

# ...where /health also pings the database

Now the database has a blip. Every pod’s liveness probe fails simultaneously. Kubernetes restarts your entire fleet, which does nothing for the database, and your application takes longer to come back than the outage itself would have lasted. You have converted a brief dependency hiccup into a full outage with a cold start on the other side.

The rule: liveness checks only the process itself. Readiness checks whether it can usefully serve.

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30
  periodSeconds: 5          # allows up to 150s to boot

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }   # process alive only
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

readinessProbe:
  httpGet: { path: /readyz, port: 8080 }    # dependencies checked here
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2

/healthz returns 200 if the HTTP server is responding at all. /readyz returns 503 when the database is unreachable. A database blip now drains traffic from pods without restarting any of them, and they return to service automatically when it recovers.

Choosing thresholds

The time to a restart is periodSeconds × failureThreshold, plus up to one timeoutSeconds. With the liveness values above, a wedged container is killed after roughly 30 seconds.

• Too aggressive and a garbage collection pause or a slow request restarts a healthy container, often cascading as load shifts to the survivors.
• Too lax and a genuinely stuck pod keeps its endpoint slot far longer than it should.

Set timeoutSeconds above your p99 latency for that endpoint — the default of 1 second is frequently too tight. Keep readiness more sensitive than liveness: dropping traffic is cheap and reversible, restarting is neither.

Graceful shutdown, the part people skip

When a pod is deleted, two things happen at once: it is removed from endpoints, and it receives SIGTERM. “At once” is the problem — endpoint removal propagates to every kube-proxy and ingress controller asynchronously, so traffic can still arrive for a second or two after your process started shutting down. That shows up as a handful of 502s on every deploy.

The fix is a preStop sleep, which delays SIGTERM while endpoint removal propagates:

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 10"]
terminationGracePeriodSeconds: 40

Make sure terminationGracePeriodSeconds comfortably exceeds the preStop sleep plus your longest in-flight request, or the kubelet will SIGKILL you mid-request and you are back where you started.

Debugging a probe that is failing

# The events tell you which probe failed and what it got back
kubectl describe pod <name> | tail -20

# Test the endpoint from inside the cluster's perspective
kubectl exec <name> -- curl -sv localhost:8080/readyz

# Restart count climbing = liveness is firing
kubectl get pod <name> -w

Two failures worth knowing on sight. A probe that fails with “connection refused” for the first few seconds and then succeeds means you need a startup probe, not a longer initialDelaySeconds. A probe that passes from kubectl exec but fails from the kubelet usually means the app is bound to 127.0.0.1 rather than 0.0.0.0.

The short version

• Liveness restarts. Readiness drains. Startup buys boot time.
• Never check dependencies in a liveness probe.
• Use separate endpoints for liveness and readiness.
• Prefer a startup probe over generous liveness thresholds.
• Add a preStop sleep if you want clean deploys.