Incident Diagnosis — Working From Evidence Instead of Intuition

Most incident time is not spent fixing. It is spent finding out what is wrong, and most of that time is wasted on hypotheses the available evidence had already ruled out.

This is the method. It is not clever; it is just applied consistently, which is the hard part at 3am.

Mitigate first, diagnose second

If users are affected and you have a mitigation — roll back, fail over, shed load, scale up — do it before you investigate.

Diagnosis is much easier when nobody is waiting for it, and a rollback is itself diagnostic information: if it fixes things, the cause is in what you rolled back.

Establish the timeline before the cause

Two questions, always, before any hypothesis:

When did it start? Precisely. Not “this morning” — the minute. Metrics give you this and it is the single most valuable fact you will get, because it lets you ask the second question.

What changed near that time? Deploys, configuration changes, feature flags, certificate expiries, scheduled jobs, traffic shifts, someone else’s deploy.

# Deploys and rollouts
kubectl rollout history deployment/<name>
kubectl get events -A --sort-by=.lastTimestamp | tail -50

# Config changes
git log --since="4 hours ago" --oneline --all

# The overlooked one: certificates
echo | openssl s_client -connect <host>:443 2>/dev/null \
  | openssl x509 -noout -dates

Correlation is not causation, but when the graph turns at 14:32 and something was deployed at 14:31, start there. You can be wrong quickly and cheaply.

If nothing changed, the causes worth considering first are the ones that happen without a change: something filled up, something expired, something crossed a threshold, or load changed.

Form hypotheses the evidence supports

The discipline that separates fast investigations from slow ones: before checking anything, write down what could produce exactly these symptoms.

For each candidate, three things:

1. The mechanism — how would this cause produce this specific symptom? A candidate you cannot explain mechanically is a guess.
2. What supports it in the evidence you already have.
3. What contradicts it in the evidence you already have.

Point 3 eliminates most candidates for free, before you run a single command. The evidence you already collected usually rules out half of what you were about to check.

Order checks by what they eliminate, not by likelihood

The instinct is to check the most likely cause first. The faster approach is to check whatever splits the remaining possibilities most evenly — a binary search over the hypothesis space.

If you have eight candidates and one check eliminates four of them, that check is worth more than one which confirms the single most likely candidate, because if the likely one is wrong you have learned almost nothing.

Practically, this means preferring checks that answer structural questions:

• Is it all users or some users? — splits global from partial
• All requests or one endpoint? — splits infrastructure from application
• Every instance or one instance? — splits systemic from local
• Started with a change or without one? — splits deployment from environment

Four questions, sixteen outcomes, and most incidents are located by the third one.

Read the evidence in the right order

1. What is the user-visible symptom, precisely?
     "Checkout fails" is not precise. "Checkout returns 502 for
      about 30% of requests, starting 14:32" is.

2. What is the blast radius?
     One endpoint / all endpoints
     One region / all regions
     One customer / all customers
     One instance / all instances

3. What changed near the start time?

4. What does the dependency chain say?
     Work outward from the failing component. Each dependency is
     either healthy, degraded, or unknown -- and "unknown" is a
     check you have not run yet, not a healthy result.

5. What does the evidence NOT explain?

Step 4’s distinction matters. Treating an unchecked dependency as fine because nothing is complaining about it is how investigations go past the actual cause without stopping.

Common traps

Fixating on the first anomaly. The first strange thing you find is often unrelated. Systems under stress produce many anomalies. Ask whether it explains the timeline — if it started three hours before the symptom, it is probably not the cause.

Confusing correlated with causal. Two metrics moving together may share a cause rather than one causing the other. High CPU and high latency during an incident are both usually symptoms of a third thing.

Restarting before capturing. A restart that fixes it and destroys the evidence guarantees the same incident again, with the same duration.

Reasoning about what the code should do. Read what it did. Logs, traces, and metrics beat inference from source every time — the running system is not always the code you are looking at.

Stopping at the first sufficient explanation. A cause that explains the symptom is not necessarily the cause. Check that it also explains the timing, the blast radius, and everything else you observed.

Knowing when to stop

Investigation has a cost, and if the impact is ongoing, that cost is measured in user harm.

Set a time box. If you have not identified the cause within it, switch to mitigation and containment, and continue diagnosis with the pressure off. An incident that is mitigated but not understood is a much better position than one that is understood in three hours.

The exception is a cause that could recur or worsen — data corruption, a security incident, a cascading failure. Those justify continuing under pressure, because mitigating without understanding might make them worse.

After

Write it down while you still remember it. Not the polished version — the actual sequence, including the wrong turns, because those are the parts that produce useful action items.

The questions worth answering:

• What would have made this shorter? Usually a check that was not obvious, a missing piece of instrumentation, or a runbook that did not exist.
• What would have made this not happen? A different default, a guard, a limit.
• What did we believe that was wrong? These are the most valuable findings and the most frequently omitted.

Then turn the answers into something durable: a runbook if the failure can recur, an alert if it was noticed by a customer first, or a change if it should not have been possible.