Kubernetes Rolling Updates Still Return Connection Refused: The preStop Fix

You run a rolling deploy at 11pm because that is the quiet window. The dashboard says the new pods are healthy, the old ones drained, zero replicas unavailable the whole time. Then you check the error tracker and there is a little spike of failed requests, all in the ninety seconds the rollout was running, all with the same message: connection refused. Not a 500 from your app. Something lower down, a connection that never completed a handshake.

Everyone tells you Kubernetes rolling updates are zero-downtime. They are zero capacity loss, which is a different promise. The default rollout keeps enough pods running to serve traffic. What it does not do is guarantee that every pod receiving traffic is still listening. There is a race between the moment a pod is told to shut down and the moment the load balancer stops sending it work, and by default nothing closes that gap.

The fix that gets copied around, terminationGracePeriodSeconds: 60, does nothing for this. It just gives a dying pod more time to die. The pod stopped listening in the first second regardless. Here is the actual mechanism and the two-line change that fixes it.

The symptom: connection refused during an otherwise clean rollout

The requests that fail look like this from the client side, whether that client is another service, an ingress controller, or an external load balancer doing its own health checks:

dial tcp 10.244.3.17:8080: connect: connection refused

The tell is the IP. 10.244.3.17 is a specific pod IP, and if you look it up you will find it belonged to a pod that was terminating at that exact second. The connection was refused because the kernel on that pod had already closed the listening socket. The traffic arrived a beat too late. Nothing in kubectl get events flags this, because from Kubernetes' point of view everything went perfectly.

Why this happens: SIGTERM and endpoint removal race each other

When you trigger a rollout, the Deployment controller creates a new pod and, once it is ready, starts terminating an old one. The termination of that old pod kicks off two independent things at the same instant:

1. The kubelet sends SIGTERM to your container. Your process gets the signal and, unless you wrote code to handle it, the runtime default is to stop and exit. The listening socket closes.
2. The pod is marked terminating and removed from the Endpoints (or EndpointSlice) object for its Service. This is the signal that eventually tells every router "stop sending traffic here".

Step 2 is not instant. The updated EndpointSlice has to reach the API server, then propagate to every kube-proxy on every node, which rewrites iptables or IPVS rules. If you have an external cloud load balancer or an ingress that keeps its own pool of backends, it has to notice the change through its own health-check or watch interval, which can be seconds. During that propagation window the routers still believe the pod is a valid target. But step 1 already finished. The socket is closed. Every request routed during the gap gets connection refused.

The fix, in steps

Step 1: Add a preStop hook that sleeps before the app sees SIGTERM

Kubernetes runs the preStop hook before it sends SIGTERM to your container. That is the whole trick: if the hook blocks for a few seconds, the container keeps serving traffic on its open socket while endpoint removal propagates in the background. Only after the hook returns does the SIGTERM go out.

spec:
  containers:
    - name: web
      image: registry.example.com/web:2026.07
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 10"]

Ten seconds is a reasonable default for in-cluster Service routing. If you sit behind a cloud load balancer with a slow deregistration interval, size the sleep to that interval. The sleep is not doing anything clever; it is buying the deregistration time to finish while the pod is still able to answer.

If your image has no shell (distroless, scratch), sleep will not exist. Either ship a tiny static sleep binary, or use the app's own signal handling to implement the delay, which leads to step 2 anyway.

Step 2: Handle SIGTERM in the app so in-flight requests finish

The preStop sleep keeps new traffic flowing during deregistration, but the moment SIGTERM finally arrives your app must not drop the requests it is already processing. Most HTTP servers exit immediately on SIGTERM unless you tell them to drain. Graceful shutdown means: stop accepting new connections, let in-flight ones complete, then exit.

const server = app.listen(8080)

process.on('SIGTERM', () => {
  // stop accepting new connections, finish the ones in flight
  server.close(() => process.exit(0))
})

The equivalent exists in every stack: Go's http.Server.Shutdown(ctx), Python's uvicorn already does graceful shutdown on SIGTERM if your handlers return, Spring Boot's server.shutdown=graceful. Without this, a preStop sleep alone still drops any request that was mid-flight when the grace period ends.

Step 3: Make terminationGracePeriodSeconds bigger than sleep plus the slowest request

Kubernetes gives the pod terminationGracePeriodSeconds to finish everything: the preStop hook and the graceful shutdown after SIGTERM. If the grace period expires, the pod gets SIGKILL and any request still open is severed. The grace period clock starts when termination begins, and the preStop hook runs inside it.

spec:
  terminationGracePeriodSeconds: 45   # > preStop sleep (10) + slowest request
  containers:
    - name: web
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 10"]

The arithmetic: if your slowest legitimate request takes 20 seconds and your preStop sleep is 10, you need at least 30, and you want headroom. The default of 30 is often fine, but if you raised the sleep or you have long requests, raise this too or SIGKILL will undo your graceful shutdown.

Step 4: Keep readiness honest, and pair it with the shutdown

Some teams make the preStop hook flip the app into failing its readiness probe, so kube-proxy deregisters it faster. That is a valid pattern, but readiness probes have their own interval and failure threshold, so it is not instant either. The plain sleep is simpler and does not depend on probe timing. If you do go the readiness route, be careful not to share one endpoint between liveness and readiness, which causes its own pod-killing bug. I wrote about that in the shared /health endpoint that kills healthy pods.

Verification: prove the gap is closed

Do not trust a quiet dashboard. Generate steady traffic and roll the deployment under it. A tiny loop against the Service, from inside the cluster, is enough:

kubectl run loadtest --rm -it --image=busybox --restart=Never -- \
  sh -c 'while true; do wget -q -O- -T 2 http://web.default.svc/healthz \
    || echo "FAIL $(date +%T)"; done'

In a second terminal, trigger the rollout:

kubectl set image deployment/web web=registry.example.com/web:2026.07
kubectl rollout status deployment/web

Before the fix you will see a cluster of FAIL lines timestamped to the rollout. After the fix the loop should print nothing but successful responses through the entire rollout. Run it a few times, because the race is probabilistic; one clean rollout is luck, five clean rollouts is a fix.

What people get wrong

"Just raise terminationGracePeriodSeconds." This is the single most repeated non-fix. The grace period governs how long the pod is allowed to take to die. The connection-refused race happens in the first second, long before any grace period matters. Raising it from 30 to 300 changes nothing about the race and just makes rollouts slower.

"Set maxUnavailable: 0." Useful, but it solves a different problem. maxUnavailable: 0 guarantees you never lose serving capacity during the rollout. It says nothing about whether a pod that is still in the routing table is still listening. You can have full capacity and still refuse connections. Keep it, but do not expect it to fix this.

"Our app exits cleanly, so we are fine." A clean, instant exit is exactly the problem. The faster your app closes its socket on SIGTERM, the wider the gap between socket-closed and routing-updated. Graceful shutdown that lingers is what you want here, not a fast clean exit.

When it is still broken

If you added the preStop sleep and still see refusals, work down this list:

• External load balancer deregistration is slower than your sleep. Cloud LBs (ALB/NLB target groups, GCP backend services) deregister on their own health-check cadence. Check the target group's deregistration delay and make the preStop sleep meet or exceed the interval that actually removes the target from rotation.
• Long-lived connections, not new ones. If clients hold keep-alive connections, closing the listener does not close those. Your graceful shutdown must actively drain them, and clients should respect Connection: close on the way out.
• Ingress controller with stale upstreams. Some ingress controllers reload their upstream list on a timer. Check whether yours watches EndpointSlices in real time or polls.
• The failures are not at termination at all. If the refusals happen when new pods come up, that is a readiness problem, not a shutdown one, and points the other direction. And if pods are dying rather than being replaced cleanly, rule out crashes first with a systematic CrashLoopBackOff diagnosis or an OOMKill.

The mental model to keep: a rolling update is two clocks that start together and finish apart. Your job is to make the pod outlive its own removal from the routing table by a few seconds. That is all the preStop sleep is doing, and it is the piece the defaults leave out.