Production Observability — Instrumentation That Answers Questions

The test for observability is not how many dashboards you have. It is whether, during an incident, you can answer a question you did not anticipate.

Most systems fail that test while having a great many dashboards.

Instrument the symptom, not the implementation

Start from what a user experiences. For a request-serving system that is four numbers, and they cover most of what you need:

• Rate — requests per second
• Errors — failures per second, and as a proportion
• Duration — latency distribution, not an average
• Saturation — how full the constrained resource is

Averages hide the problem. A mean latency of 200ms is compatible with 5% of users waiting eight seconds. Record histograms and alert on quantiles.

# p99 latency by service
histogram_quantile(0.99,
  sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))
)

# Error ratio, which is what users notice -- not the raw count
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
  / sum by (service) (rate(http_requests_total[5m]))

The error ratio matters more than the count. A hundred errors per second is fine at a million requests per second and an outage at a hundred and one.

Cardinality is what your bill is made of

A time series is created for every unique combination of label values. Add a label with a thousand possible values to a metric with ten existing series and you now have ten thousand.

The rule: labels for things you group by, traces and logs for things you look up. You group by service, endpoint, status class, and region. You look up a specific request by ID — and that is a trace, not a metric.

Normalise paths before they become labels:

/users/12345/orders/98765  →  /users/{id}/orders/{id}

The three signals do different jobs

Metrics answer “is something wrong, and when did it start”. Cheap, aggregated, retained a long time. Bad at explaining why.

Logs answer “what happened to this specific thing”. Expensive at volume. Only useful if structured — a log line you cannot query by field is a log line you will read manually at 3am.

Traces answer “where did the time go”. Indispensable in distributed systems, where latency is a sum across services and the slow one is rarely the one being blamed.

Link them. A trace ID in every log line and as an exemplar on metrics turns three disconnected tools into one investigation. This is the single highest-value integration in observability work and it is frequently skipped because it requires touching every service once.

Structured logs, or none

{"ts":"2026-08-24T22:41:03Z","level":"error","service":"api","trace_id":"4bf92f...",
 "event":"db_query_failed","query":"select_user","duration_ms":5031,"error":"timeout"}

Queryable by field. You can ask “every db_query_failed over five seconds in the last hour, grouped by query” and get an answer in seconds.

2026-08-24 22:41:03 ERROR Failed to query database after 5031ms: timeout

Not queryable. You can grep it, which is not the same thing.

Log levels, used consistently:

• ERROR — a human needs to know. If nobody would act on it, it is not an error.
• WARN — degraded but handled. A retry succeeded, a fallback was used.
• INFO — significant state changes. Startup, shutdown, configuration loaded.
• DEBUG — off in production, enabled deliberately when investigating.

The most common failure is logging at ERROR for conditions nobody acts on, which trains everyone to ignore errors.

Alerting: page for symptoms, ticket for causes

An alert should mean: a human needs to do something now. Anything else is a ticket, a dashboard, or nothing.

groups:
  - name: api-slo
    rules:
      - alert: ApiErrorRateHigh
        expr: |
          sum(rate(http_requests_total{job="api",status=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="api"}[5m])) > 0.02
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "API error rate above 2%"
          description: "Error ratio is {{ $value | humanizePercentage }} over 10 minutes."
          runbook_url: "https://insidetheaistack.com/runbooks/..."

The for clause is the single most important field. Without it, one bad scrape pages someone. With ten minutes, a transient spike resolves itself and nobody wakes up.

Symptom alerts (“users are getting errors”) page. Cause alerts (“disk is 80% full”) create tickets. The difference is whether a user is affected right now.

The alert review

Every alert should survive three questions:

1. Is a user affected right now? If not, it is not a page.
2. Can a human do something about it in the next few minutes? If not, it is not a page.
3. Has it fired more often than it found a real problem? If so, fix it or delete it.

Run this review on your alert set quarterly. Alerts accumulate, nobody removes them, and the cumulative effect is that people stop reading them — which is worse than having no alerts, because you believe you are covered.

What to instrument first

In order, for a service that has nothing:

1. Request rate, error rate, latency histogram at the edge — this alone answers most “is it broken” questions
2. Dependency calls — the same three numbers for every outbound call, which is how you find out whose problem it is
3. Saturation of the constrained resource — connection pool, queue depth, worker utilisation
4. Business events — the small number of things that indicate the system is doing its job, not merely responding

Point 4 catches the failures nothing else does. A payment system with perfect latency and zero errors that has processed no payments in twenty minutes is broken, and only a business metric knows that.

Retention

Not everything needs the same retention, and undifferentiated retention is where the budget goes:

• High-resolution metrics — days. You need the detail during and just after an incident.
• Downsampled metrics — months to years. Capacity planning and trends.
• Logs — days to weeks for most, longer only where compliance requires it.
• Traces — sampled. Keeping every trace is rarely worth it; keeping every error trace and a small percentage of successful ones almost always is.

Tail-based sampling — decide after the trace completes, keeping the slow and failed ones — gives you the traces that matter at a fraction of the volume.

Checklist

• Latency recorded as a histogram, alerted on quantiles
• Error ratio tracked, not just error count
• No unbounded label values anywhere
• Paths normalised before becoming labels
• Every log line structured and carrying a trace ID
• ERROR level reserved for things a human acts on
• Every page-level alert has a for duration and a runbook link
• Alerts reviewed against the three questions on a schedule
• At least one business-level metric per service
• Trace sampling keeps errors and slow requests