Creating observable systems with actionable telemetry, meaningful alerts, and operational practices that support resilient services.
Saturation and replica counts can look fine while a dependency timeout breaks checkout. SLIs should follow the user path.
Request IDs, emails, and unbounded path labels will exhaust a metrics backend. High-cardinality identifiers belong in logs or traces.
Paging on every CPU spike destroys response quality. Alert on symptoms with runbooks, not on every gauge that moves.
Instrument for user impact first. Service-level indicators, structured logs with request context, and traces across dependencies explain failures that CPU graphs hide. Alerts should page a human only when a user-facing symptom needs a response.
Metrics answer 'is it broken and how much.' Logs answer 'what happened on this request.' Traces answer 'where time went.' A hypothetical stack uses Prometheus-compatible metrics, Loki for logs, Tempo or another OpenTelemetry-compatible trace backend, and Grafana as the correlation UI. Amazon CloudWatch is the AWS-native default when a separate stack is not justified. The same ideas apply to other backends.
Metrics. Is it broken, and by how much? Bound label sets. Request IDs do not belong here.
Illustrative telemetry paths. Cardinality, retention, and alert ownership matter as much as the vendor names on the diagram.
Availability, latency, and correctness on the user-facing path come first. Error budgets make reliability negotiations concrete.
A trace id in logs and traces lets an incident responder jump from an alert to the slow dependency without grep folklore.
A rollout verification window is part of deploy, not a later nice-to-have. Watch errors, latency, and saturation against the previous revision.
If the runbook is 'look at a dashboard,' the alert is unfinished. Include impact, likely causes, and a safe mitigation.
Start from the user-facing symptom, open the trace, then the logs for that trace id, then the dependency that consumed the time. Dashboards without that join are scenery.
More labels and longer retention improve forensics and raise ingest cost. Sample traces, bound label sets, and expire debug logs.
Rate, errors, and duration cover most request services. Utilization, saturation, and errors help for infrastructure. Checkout completion is still a better product signal than CPU.
A platform can provide collectors and defaults. Teams still own the SLIs for their user paths.
After a rolling update, responders watch error ratio and p95 latency for the public API against a 15-minute baseline. A trace shows time spent in a database call that grew after the release. Logs for the trace id show a new query plan. This is an educational investigation path, not a reconstructed incident from a client system.
CPU is a saturation clue, not a user-experience signal. A saturated thread pool or a dependency timeout can page first.
user_id or raw URLs as Prometheus labels will degrade scrape and query performance for everyone.
A wall of graphs is not an operating model. Decide what 'good' is before the incident.
Correlate hypothetical metrics, logs, and traces, then choose a mitigation. These are not production incidents.
Investigate a teaching incident →Build a usable telemetry model around user-facing SLIs, bounded cardinality, and alerts that have a next step.
Understand readiness, surge capacity, graceful shutdown, and telemetry without treating rolling updates as an availability guarantee.
How HPA, resource requests, and cluster capacity interact—and why a target utilization is not a capacity plan.
Architecture, delivery, identity, and recovery—not a tool list. No fabricated case studies, no implied retainer.