Observability with Metrics, Logs, and Traces

Telemetry is for questions, not dashboards

A useful observability practice answers three questions during an incident: is the user path broken, where is time going, and what changed. Metrics, logs, and traces are tools for those questions. A wall of graphs is not an operating model.

This guide describes a hypothetical request-driven API with Prometheus-compatible metrics, structured logs, and OpenTelemetry traces. Vendor names are interchangeable. The constraints are not.

Start with service-level indicators

Pick signals that follow the user:

• Availability: the ratio of successful requests on the public path.
• Latency: a percentile that matches user patience, not average CPU.
• Correctness: if you can measure it (failed checkouts, empty search), do.

An SLO is a target for that SLI over a window. An error budget is the remainder. These are negotiation tools: they tell you whether to ship features or stop and repair. They are not marketing numbers. Do not publish an SLO you do not measure.

Infrastructure gauges (CPU, memory, disk) explain saturation. They should not be the first page-worthy alert unless the host is the product.

Metrics: bound the label set

Prometheus-style metrics fail when labels are unbounded. user_id, raw URLs, and emails explode series count, scrape time, and query cost.

# Educational Prometheus scrape fragment. Not a production scrape config.
scrape_configs:
  - job_name: example-api
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: example-api
        action: keep

Prefer a small set: service, route (normalized), status_class, method. Put the request id in logs and traces. Recording rules can pre-aggregate SLI numerators and denominators so dashboards do not rescan raw series every thirty seconds.

RED (rate, errors, duration) fits request services. USE (utilization, saturation, errors) fits nodes, queues, and thread pools. Use both; do not pretend one panel explains checkout.

Logs: structure and a join key

A log line should be parseable. Timestamp, severity, service, trace_id, and a short event name beat a paragraph of prose.

# Educational structured log. Do not log tokens or personal data.
ts=2026-10-12T09:14:02Z level=info service=example-api
trace_id=6b4c2a91 route=/checkout status=503 duration_ms=1842 error=payments_timeout

Retention is a cost and a privacy decision. Debug logs on every replica at information volume will dominate the bill and the leak surface. Sample or gate verbose logs behind an explicit flag.

Traces: where time went

OpenTelemetry traces connect the incoming request to downstream calls. A span that says the handler waited 1.8s in payments is worth more than a CPU graph that says the pod was fine.

# Educational collector sketch. Exporters and sampling are environment-specific.
receivers:
  otlp:
    protocols:
      http: {}
processors:
  batch: {}
  tail_sampling:
    policies:
      - name: errors-and-slow
        type: status_code
        status_code: {status_codes: [ERROR]}
exporters:
  otlp/tempo:
    endpoint: tempo.example.svc:4317
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/tempo]

insecure: true is for a local example, not a production collector. Sample in production. Exporting 100% of spans from a busy edge will hurt the application and the backend. Tail sampling on errors and slow traces is a common pattern; it is a recommendation, not a law.

Never copy trace ids into Prometheus labels.

Alerting and investigation

Page a human when a user-facing SLI is burning budget and someone can act. A runbook should include impact, the first two checks, and a safe mitigation (rollback, shed load, disable a feature). “Look at the dashboard” is not a runbook.

A practical investigation path:

1. Alert fires on error ratio or latency SLO.
2. Jump to the service’s RED panel and the current release annotation.
3. Open an example trace from the bad window.
4. Read logs for that trace_id.
5. Decide: application regression, dependency, or capacity.

Compare the verification window after a deploy to the previous revision. That is how you catch a bad rollout before the error budget is gone.

Security and reliability of the stack

Logs and traces contain tokens, emails, and internal URLs if you let them. Redact at the source. Restrict who can query raw telemetry. Collector identities should not be able to change production workloads.

The observability stack is itself a dependency. A missing metrics backend should not take down the API, but it will blind responders. Document a backup page path if the chat or paging vendor is down.

Common failure modes

• Paging on CPU alone.
• Unbounded metric labels.
• Traces without a join into logs.
• Dashboards built before SLIs.
• Alerts with no owner and no expiry.

Continue with related material

Zero-downtime Kubernetes deployments uses the same verification window. Autoscaling explains why CPU utilization is a poor standalone SLI. See Observability and the observability lab.