Zero-Downtime Deployments with Kubernetes
Understand readiness, surge capacity, graceful shutdown, and telemetry without treating rolling updates as an availability guarantee.
When hosts and probes look fine but users still fail — separating ALB health checks, Kubernetes readiness, and dependency health from real traffic paths.
On-call charts often show normal CPU, memory, and process uptime while support tickets describe failed checkouts, blank screens, or API errors. Infrastructure signals and user-visible behavior are related, but they are not the same measurement.
Before scheduling a host reboot, name whether you are debugging platform reachability or application correctness. That distinction saves time when every dashboard tile is green.
An Application Load Balancer sends health checks to registered targets using the protocol, port, path, and success codes you set. AWS documents this in target group health checks. Checks run on an interval; each request is independent, and the target must pass one successful check after registration before traffic is routed to it.
A target can be healthy in the console while failing paying users. Common mismatches include a probe path that returns 200 without exercising checkout logic, a check on an admin port while the public listener fails, or matcher rules that accept the wrong status codes. When users fail but targets look healthy, line up the probe URL, port, and matcher against the path and headers a real client uses.
AWS also notes that if every registered target is unhealthy, the load balancer may route to all targets anyway (fail open). That behavior is easy to miss when you only watch the healthy-host count.
Kubernetes readiness controls whether a Pod is included in Service endpoints; liveness can restart a container. The upstream probe task guide states that failed readiness removes the Pod from Service traffic, while liveness addresses deadlocks that need a restart.
Neither probe proves every user journey works — only the check you configured. Sharing one expensive dependency check across readiness and liveness can amplify an outage: a database stall may drain endpoints and trigger restarts at the same time.
If incident language is “the nodes are fine” while EndpointSlices show few ready backends, you are in probe or routing semantics, not hypervisor failure. For how Pods become ready after a deploy, see /blog/kubernetes-rolling-deployments as a separate lens from this article.
The primary process can stay alive while outbound calls fail. Pools to a database, cache, identity broker, or payment API may be saturated. Certificate expiry, trust store drift, and DNS changes surface as application errors without crashing the container.
“Restart the server” is not a default playbook here. Follow one failing request through the edge, the app, and each dependency, and note which timeout or status code appears first. For correlating signals once you know the hop, see /blog/observability-stack.
When hardware metrics look calm and customers do not:
This order avoids treating a routing or dependency fault as a hardware incident.
Understand readiness, surge capacity, graceful shutdown, and telemetry without treating rolling updates as an availability guarantee.
Fair on-call means a named owner, clear severity rules, escalation, and real recovery time—not a whole-team “always available” chat. Practical guidance for employees, managers, and HR.
Diagnose Kubernetes CrashLoopBackOff by reading Waiting state, exit codes, kubectl logs --previous, and env config—then fix the cause instead of restarting the Pod.