Platform Engineering

Why Production Applications Go Down Even When Servers Are Healthy

When hosts and probes look fine but users still fail — separating ALB health checks, Kubernetes readiness, and dependency health from real traffic paths.

Two different questions: is the machine up, is the journey working?

On-call charts often show normal CPU, memory, and process uptime while support tickets describe failed checkouts, blank screens, or API errors. Infrastructure signals and user-visible behavior are related, but they are not the same measurement.

Before scheduling a host reboot, name whether you are debugging platform reachability or application correctness. That distinction saves time when every dashboard tile is green.

Application Load Balancer health is configured, not assumed

An Application Load Balancer sends health checks to registered targets using the protocol, port, path, and success codes you set. AWS documents this in target group health checks. Checks run on an interval; each request is independent, and the target must pass one successful check after registration before traffic is routed to it.

A target can be healthy in the console while failing paying users. Common mismatches include a probe path that returns 200 without exercising checkout logic, a check on an admin port while the public listener fails, or matcher rules that accept the wrong status codes. When users fail but targets look healthy, line up the probe URL, port, and matcher against the path and headers a real client uses.

AWS also notes that if every registered target is unhealthy, the load balancer may route to all targets anyway (fail open). That behavior is easy to miss when you only watch the healthy-host count.

Readiness and liveness answer narrower questions

Kubernetes readiness controls whether a Pod is included in Service endpoints; liveness can restart a container. The upstream probe task guide states that failed readiness removes the Pod from Service traffic, while liveness addresses deadlocks that need a restart.

Neither probe proves every user journey works — only the check you configured. Sharing one expensive dependency check across readiness and liveness can amplify an outage: a database stall may drain endpoints and trigger restarts at the same time.

If incident language is “the nodes are fine” while EndpointSlices show few ready backends, you are in probe or routing semantics, not hypervisor failure. For how Pods become ready after a deploy, see /blog/kubernetes-rolling-deployments as a separate lens from this article.

Dependencies break while the main process keeps running

The primary process can stay alive while outbound calls fail. Pools to a database, cache, identity broker, or payment API may be saturated. Certificate expiry, trust store drift, and DNS changes surface as application errors without crashing the container.

“Restart the server” is not a default playbook here. Follow one failing request through the edge, the app, and each dependency, and note which timeout or status code appears first. For correlating signals once you know the hop, see /blog/observability-stack.

Inspection order when charts disagree with users

When hardware metrics look calm and customers do not:

  1. Edge path — latency and status at the load balancer or ingress, not only inside the VPC.
  2. Health-check contract — does the probe use the same dependency chain as revenue traffic?
  3. Ready backends — healthy targets in the target group or ready endpoints behind the Service.
  4. Dependency signals — connection errors, queue depth, or external API failures tied to the broken route or flag.
  5. Recent change — certificates, feature flags, schema, or routing config that would not move CPU graphs.

This order avoids treating a routing or dependency fault as a hardware incident.

Related technical guides

LET’S BUILD WHAT’S NEXT

Good engineering starts with a conversation.

Let’s connect