Most observability stacks fail the same way: a dashboard with forty panels, an alerting channel everyone has muted, and an on-call engineer who still finds out about outages from a customer email first. The tooling isn't the problem. The problem is that the metrics being watched were chosen by what was easy to instrument, not by what a user would actually notice.
We inherited an environment once with a genuinely impressive observability stack by every technical measure — distributed tracing, structured logs, a metrics pipeline with sub-second resolution — and a mean time to detection for customer-facing incidents of over forty minutes. The instrumentation wasn't the gap. Nobody had connected any of it back to a definition of 'the user is having a bad time.'
Start from the SLI a user would recognize
CPU utilization, memory pressure, and queue depth are useful for diagnosis, but they are not Service Level Indicators — a user has never once cared what your CPU is doing. The indicators that matter are the ones a user could describe without technical language.
- Checkout completes in under 2 seconds, 99% of the time.
- The dashboard loads with data, not a spinner, on first paint.
- A submitted form either saves or shows a clear error — never a silent failure.
- Search results return in under 500ms at the 95th percentile.
Everything else — the CPU graphs, the connection-pool charts — exists to explain a regression in one of these, not to stand on its own as a thing worth paging someone over.
Alert fatigue kills the signal
A threshold alert that fires every time a metric crosses an arbitrary line trains the team to ignore the channel. We moved every production system to error-budget burn-rate alerting instead: define an SLO — say, 99.9% of requests succeed in a rolling 30-day window — and alert only when the current error rate would exhaust that month's entire budget within a few hours. A blip gets absorbed silently. A real incident pages someone in minutes, with the trajectory already visible in the alert.
What this looks like in practice
Pick 3–5 SLOs per service
Availability, latency, and correctness for the primary user journey — not forty metrics nobody will review in a postmortem.
Set the error budget and the burn-rate thresholds
A fast-burn alert, where the budget would be exhausted in hours, pages immediately. A slow-burn alert, exhausted in days, opens a ticket instead.
Review the budget monthly, not just during incidents
If a service never comes close to spending its error budget, the SLO is too loose and hiding real degradation. If it's always nearly spent, it's too strict for the architecture actually in place.
Traces and logs are for 'why', not 'is something wrong'
Once an SLO-based alert fires, the team needs to move fast from 'something is wrong' to 'here is the specific request path that's failing and why.' That's what traces and structured logs are for — but only if they're queryable by the dimensions an engineer will actually reach for during an incident: customer ID, request ID, deployment version, region. A trace that exists but can't be filtered by the one field the on-call engineer needs at 3am is functionally the same as not having it.
- Propagate a single request ID across every service a request touches — the first five minutes of most incidents are lost stitching logs together by timestamp instead.
- Tag every span and log line with the deploy version — 'did this start right after the last release' is the single most common root-cause question.
- Keep high-cardinality fields (customer ID, request ID) out of metrics and in traces/logs — metrics systems degrade badly under high cardinality, which is exactly when you need them to still work.
In closing
Observability earns its keep when it answers "is the user having a bad time right now" faster than a support ticket does. Everything below that — logs, traces, infrastructure metrics — exists to answer the follow-up question of why, not to be the first thing anyone looks at.
