The Shared /health Endpoint That Kills Healthy Kubernetes Pods

The database has a bad thirty seconds. A long-running migration, a lock, a noisy neighbour on the managed instance, it does not matter which. Thirty seconds later every single pod in your deployment is restarting at once, the service is fully down, and the application logs show nothing wrong at all because nothing was wrong with the application.

What happened is that all of them were pointing their liveness probe at the same /health endpoint that also checks the database. The database got slow, the probe timed out, kubelet concluded the container was dead, and killed it. Then the freshly started containers all tried to open new database connections simultaneously, which made the database slower, which failed the probes again.

A liveness probe that checks your dependencies is a distributed suicide pact. The two probes answer completely different questions and have completely different consequences, and the moment they share an endpoint you have wired a transient dependency blip directly to "kill every replica". This is the most common serious Kubernetes misconfiguration I run into, and it is entirely self-inflicted.

The exact error in kubectl describe

Events:
  Type     Reason     Age                 From     Message
  ----     ------     ----                ----     -------
  Warning  Unhealthy  2m (x9 over 5m)     kubelet  Readiness probe failed: HTTP probe failed with statuscode: 503
  Warning  Unhealthy  90s (x3 over 2m)    kubelet  Liveness probe failed: Get "http://10.244.1.7:8080/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   Killing    90s                 kubelet  Container api failed liveness probe, will be restarted

Two things to notice. The liveness failure is a client timeout, not a non-200 response: kubelet gave up waiting. And the readiness failure and the liveness failure are hitting the same path, which is the actual bug. When you see both messages against the same URL, you already know what to fix.

Why this happens

The two probes have wildly different blast radii

	Liveness	Readiness
Question it answers	Is this process wedged beyond recovery?	Should traffic go here right now?
On failure	kubelet kills and restarts the container	Pod IP removed from the Service EndpointSlice
Pod keeps running	No	Yes
Recovers on its own	Only by restarting	Yes, as soon as the check passes again
May check dependencies	No	Yes, that is what it is for
Cost of a false positive	Lost capacity, dropped in-flight requests, restart storm	Briefly less capacity, requests routed elsewhere

A false readiness failure is cheap and self-correcting. A false liveness failure destroys running state and, because every replica shares the same dependencies, tends to happen to all of them within the same probe period.

Shared dependencies make the failure correlated

Independent random failures are fine; Kubernetes handles those. What breaks a cluster is failures that arrive together. Ten pods all checking the same Postgres in their liveness probe are not ten independent health signals, they are one signal copied ten times. When it goes bad, it goes bad everywhere at once, and the restart is the worst possible response because the reconnection storm makes the original problem harder to recover from.

The Kubernetes documentation is blunt about this: "Incorrect implementation of liveness probes can lead to cascading failures. This results in restarting of container under high load; failed client requests as your application became less scalable; and increased workload on remaining pods due to some failed pods."

The default timeout is one second

The probe defaults are easy to miss because they are usually inherited rather than written:

Field	Default	Note
initialDelaySeconds	0	Does not start until the startup probe succeeds, if one is defined
periodSeconds	10	How often the probe runs
timeoutSeconds	1	The one that bites. One second.
successThreshold	1	Must be 1 for liveness and startup probes
failureThreshold	3	Consecutive failures before kubelet acts

A one second timeout means a JVM stop-the-world pause, a busy Node.js event loop or a Python process mid-GC can fail a liveness probe while being completely healthy. Under load. Which is when your remaining pods can least afford to lose a sibling.

The fix, step by step

Step 1: Write two endpoints that do different things

// livez: can this process answer at all? No dependencies. Ever.
app.get('/livez', (req, res) => res.status(200).send('ok'))

// readyz: should this pod receive traffic right now?
app.get('/readyz', async (req, res) => {
  if (shuttingDown) return res.status(503).send('draining')
  try {
    await db.query('SELECT 1')
    res.status(200).send('ready')
  } catch (err) {
    res.status(503).send('db unavailable')
  }
})

/livez is deliberately trivial. If you are tempted to add a check to it, the test is: would restarting the container fix this condition? A wedged event loop, yes. A database outage, no, so it does not belong there.

Step 2: Wire them up separately, with a startup probe

containers:
  - name: api
    image: registry.example.com/api:1.4.2
    ports:
      - containerPort: 8080

    startupProbe:
      httpGet:
        path: /livez
        port: 8080
      periodSeconds: 5
      failureThreshold: 30        # up to 150s to boot, then give up

    livenessProbe:
      httpGet:
        path: /livez
        port: 8080
      periodSeconds: 10
      timeoutSeconds: 3
      failureThreshold: 3

    readinessProbe:
      httpGet:
        path: /readyz
        port: 8080
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 2
      successThreshold: 1

The startupProbe is what lets you keep the liveness settings tight. While it is running, liveness and readiness do not fire at all, so a container that takes two minutes to warm a cache is not killed at ninety seconds. Once it succeeds, the strict liveness probe takes over. This is strictly better than the older pattern of setting a large initialDelaySeconds, which delays detection of genuine hangs for the entire life of the pod.

Step 3: Make readiness fail during shutdown

Set shuttingDown = true on SIGTERM before you stop accepting connections, and add a preStop hook that sleeps a few seconds:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 5"]

Endpoint removal and container termination are not synchronised. Without that pause, the kube-proxy rules on some nodes still point at a pod that has already closed its listener, and you get a handful of connection resets on every deploy. Five seconds of doing nothing removes them.

Step 4: Give the probes their own path to the process

If your probe endpoint is served by the same worker pool that is currently saturated with slow requests, the probe queues behind them and times out, and you restart a pod whose only crime was being busy. In a threaded server, reserve a worker. In Go, this is free. In a single-threaded runtime, keep the handler synchronous and allocation-free so it cannot be starved by application work.

Verification

Break the dependency on purpose and watch what does not happen:

# 1. Baseline
kubectl get pods -l app=api
# expect: READY 1/1, STATUS Running, RESTARTS 0

# 2. Make the database unreachable (scale it down, or drop a NetworkPolicy in)

# 3. Watch
kubectl get pods -l app=api -w
# expect: READY 0/1, STATUS Running, RESTARTS stays at 0

kubectl get endpointslices -l kubernetes.io/service-name=api
# expect: no addresses, or addresses marked not ready

# 4. Restore the database
kubectl get pods -l app=api
# expect: READY 1/1 again, RESTARTS still 0

RESTARTS staying at 0 through a full dependency outage is the whole test. If that number moves, your liveness probe is still checking something it should not.

What people get wrong

"Raise failureThreshold until the restarts stop." This is the most popular fix and it is treating a design error with a delay. You have not stopped the probe from killing healthy pods, you have made it take longer, and you have simultaneously made the probe useless for detecting the genuine deadlock it exists to catch. Separate the concerns first; tune thresholds afterwards, if at all.

"One /health endpoint is simpler." It is simpler right up until it takes production down. The two probes are asking questions with opposite correct answers during a dependency outage: readiness should say no, liveness should say yes. A single endpoint cannot express both, so whichever answer it gives is wrong for one of them.

"Remove the liveness probe entirely." Tempting after an incident, and better than a dependency-checking one, but you give up recovery from genuine hangs: a deadlocked thread pool, an event loop blocked forever, a wedged connection pool. Keep it, and keep it trivial.

"Readiness should not check the database either, it causes flapping." Readiness checking dependencies is the correct design; flapping is a tuning problem. Cache the dependency check result for a second or two so a probe every five seconds does not become five queries per second per pod, and set failureThreshold to 2 or 3 so one slow response does not pull the pod out.

When it is still broken

• Restarts with no probe events. Look at kubectl describe pod for OOMKilled and check lastState.terminated.reason. Memory limits kill containers silently as far as probes are concerned, and it looks identical from the outside.
• Exec probes on a busy node. An exec probe forks a process on every period, on every pod. At scale that is real CPU on the kubelet. Prefer httpGet where you can.
• The probe passes locally and fails in the cluster. Check the app is bound to 0.0.0.0 and not 127.0.0.1, and that the containerPort matches. This is nearly always the answer when the probe never succeeds even once.
• Everything is correct and traffic still drops during deploys. Look at terminationGracePeriodSeconds against how long your longest request takes, and at the rolling update maxUnavailable. Probes are only half of a graceful deploy.

If you take one rule from this: a liveness probe may only fail for a condition that restarting the container would fix. Everything else, every dependency, every downstream service, every slow query, belongs in readiness. That one sentence prevents more Kubernetes outages than any amount of threshold tuning.