Why Production Applications Go Down Even When Servers Are Healthy
When hosts and probes look fine but users still fail — separating ALB health checks, Kubernetes readiness, and dependency health from real traffic paths.
Fair on-call means a named owner, clear severity rules, escalation, and real recovery time—not a whole-team “always available” chat. Practical guidance for employees, managers, and HR.
Cloud9 Solutions — "On call vs. always available" (narrated video transcript) It's Friday evening. You're finally off work. Then your phone lights up. Urgent. Can someone take a look? Six people get the message. Nobody knows who's responsible. That's not an on-call rotation. That's an always-available culture. Real on-call means clear ownership, meaningful alerts, proper escalation and time to recover. Because reliable systems need sustainable teams. Cloud9 Solutions. Cloud, DevOps and Platform Engineering.
A common frustration sounds like this: “My salary covers about 40 hours. Why am I expected to be available all night?”
Contracts differ. Some roles explicitly include evenings, weekends, or paid on-call. Some countries and employers structure hours differently. This article does not claim everyone has a 40-hour contract, and it does not claim all after-hours work is unreasonable.
The useful distinction is different:
Reliable systems need people who can think. Exhausted teams ship brittle ones. For a well-known industry discussion of healthy on-call design, see Google’s SRE book chapter Being On-Call.
This guide is for employees, managers, HR partners, and founders first—with practical detail for engineering managers and DevOps / platform teams later. The animation above is an illustrative scenario, not a client story.
Related reading: Zero-Downtime Deployments with Kubernetes, CrashLoopBackOff troubleshooting, technical interviews that test real skills, and AI, automation, and accountability.
| Term | Plain meaning |
|---|---|
| On-call | A scheduled person (or small set) expected to respond to important failures outside normal work |
| Incident | Something broken or degraded that needs coordinated response |
| Escalation | Bringing in backup, specialists, or managers when the primary cannot resolve it alone |
| Alert fatigue | So many noisy alerts that people stop trusting—or answering—them |
Fair on-call usually includes:
An always-available culture looks different: one alert to the whole team, and everyone feeling they have to answer.
Teams should write their own definitions. This table is an educational starting point:
| Level | Example | Interrupts an evening? |
|---|---|---|
| SEV-1 | Checkout or login fully down for customers | Yes — page primary immediately |
| SEV-2 | Major feature degraded; workaround exists | Often yes — primary triages; may page backup |
| SEV-3 | Internal tool slow; no customer impact | Usually no — next working day |
| SEV-4 | Cosmetic or informational | No — ticket / backlog |
Educational template—adapt to your org:
Write when to escalate, not only who. Ambiguity is how whole-team chats form.
If phones buzz for noise, people will eventually ignore a real SEV-1.
Alert-quality checklist (illustrative)
Tuning alerts is reliability work, not a side hobby.
Common forms include:
This varies by country, employer, and contract. Local labour law may set floors. This article is not legal advice and invents no figures. HR and managers should align written policy with how the rota actually runs.
At the end of a shift / start of the next working day:
After meaningful incidents, run a blameless review focused on: what happened, what signals were missing, what will reduce the next page. Assign follow-ups with owners and dates. Reviews that only assign blame teach people to hide problems.
There is no single perfect size. Consider:
Each shift has one primary on-call person published in advance—not a whole-team group chat hoping someone replies.
The team agrees which failures interrupt an evening and which can wait until the next working day.
Primary → backup → subject-matter help → management is written down, with when to page each step.
Alerts are actionable. Noise is reviewed. People are not woken for dashboards that need no human action.
On-call time is recognized somehow that fits local contracts—stipend, paid response, and/or time off in lieu—and recovery time after hard nights is normal.
Shifts end with a short handover. Incidents get blameless follow-ups that reduce the next page, not blame theatre.
Focus: Named owner. Each shift has one primary on-call person published in advance—not a whole-team group chat hoping someone replies.
Illustrative self-check for teams. Use it in a retro: green means the practice exists and is used; amber means partial; red means missing. Not a legal or HR assessment.
Reliable systems need sustainable teams.
Clear ownership. Fair rotation. Real time off. That is how on-call supports the product instead of burning the people who keep it running.
Need help shaping incident response, platform reliability, or sustainable on-call for a DevOps team? Explore Cloud9 Solutions.
When hosts and probes look fine but users still fail — separating ALB health checks, Kubernetes readiness, and dependency health from real traffic paths.
Fundamentals matter in DevOps hiring—but isolated trivia rarely shows how someone troubleshoots, communicates, or weighs risk. A fairer loop tests knowledge in context.
AI can draft infrastructure code, pipelines, and incident notes in seconds. Production still needs humans who weigh risk, approve changes, and own outcomes when automation is wrong.