Why Kubernetes Rolling Updates Still Give You 503s

"We use rolling updates, so deploys are zero-downtime." Then you deploy, and for 2–3 seconds users get 503s. This exact complaint shows up constantly on Stack Overflow and Reddit, and it has a precise, fixable cause — actually four of them, stacked.

The core misunderstanding: rolling updates guarantee pod replacement order, not traffic correctness. Kubernetes replaces pods gracefully; whether requests survive depends on details it leaves to you.

Cause 1: traffic arrives before the app is ready

A pod becomes a load-balancing target when its readiness probe passes. If you have no readiness probe, that's the moment the container starts — before your app has loaded config, connected to the database, or warmed anything. First requests: connection refused.

Fix: a readiness probe that checks what "ready to serve" actually means for your app:

readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 2
  failureThreshold: 2

And /ready should verify dependencies (DB connection up), not just return 200 unconditionally.

Cause 2: traffic keeps arriving after the pod is told to die

This is the one that causes the classic "2–3 seconds of 503s," and it's the least intuitive. When a pod terminates, two things happen in parallel, not in sequence:

1. The pod gets SIGTERM.
2. Endpoint controllers start removing the pod from load balancers.

Step 2 takes time to propagate — kube-proxy rules on every node, cloud LB target deregistration. For seconds after SIGTERM, traffic is still routed to a pod that may already be shutting down. If your app exits immediately on SIGTERM, every one of those requests fails.

Fix — the famous sleep:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "8"]

The preStop hook runs before SIGTERM is sent. Sleeping a few seconds keeps the app fully serving while endpoints propagate. It looks like a hack; it's the standard, load-bearing pattern (Kubernetes docs and every ingress vendor recommend some form of it).

Cause 3: the app doesn't drain on SIGTERM

After preStop, SIGTERM arrives. The app must: stop accepting new connections, finish in-flight requests, then exit. Many frameworks do this only if you ask — and some servers keep-alive connections open forever unless told to close them.

Fix: enable your framework's graceful shutdown (Spring: server.shutdown=graceful; Go: http.Server.Shutdown(); Node: close the server and track sockets), and make sure terminationGracePeriodSeconds (default 30) exceeds preStop + your longest request. If it doesn't, SIGKILL wins and kills in-flight work.

Cause 4: the rollout replaces capacity too aggressively

Defaults (maxUnavailable: 25%) can briefly leave you under-provisioned under load: old pods dying while new pods are technically Ready but cold (empty caches, JIT not warmed, connection pools empty). Latency spikes read as downtime.

Fix:

strategy:
  rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }

New capacity comes up before old capacity goes away. Slower rollout, no dip. Add a PodDisruptionBudget so node drains during upgrades obey the same rule.

The complete checklist

For every user-facing service:

• ☐ Readiness probe that checks real dependencies
• ☐ preStop sleep (5–10s)
• ☐ Graceful shutdown on SIGTERM, verified (kill a pod, watch error rates)
• ☐ terminationGracePeriodSeconds > preStop + longest request
• ☐ maxUnavailable: 0, maxSurge: 1 for critical services
• ☐ PodDisruptionBudget
• ☐ Actually test it: kubectl rollout restart under synthetic load and watch the error rate

That last item is the real gap. Teams configure all of this and never verify it. A deploy under load in staging with a load generator running takes fifteen minutes and tells you the truth.

Why this matters beyond deploys

The same mechanics run during node upgrades, autoscaler scale-downs, and spot instance reclaims. If deploys cause 503s, so does every cluster maintenance event — your "deployment problem" is actually an "any pod ever moves" problem. Fix it once and cluster upgrades, autoscaling, and spot instances all become safe as a side effect. It's one of the highest ROI afternoons available in Kubernetes.

Deployment safety is one of seven categories in the free production readiness scorecard at kestrion.dev.