Incident investigation worksheet

Before deciding what is wrong, decide who is affected.

• Are running workloads affected, or only the control plane?
• What did you check to establish that? (Name the command or observation.)
• If workloads are fine: you have time to diagnose properly. Say so to whoever is watching.

The shape of the failure narrows the cause more than any log line.

• One service everywhere, all services one host, everything at once, one tenant, or intermittent?
• If everything at once: which shared dependency? (bus, database, identity, load balancer)
• If intermittent: suspect a load balancer backend, a cluster partition, or multi-writer database.

Each of these takes seconds and removes a whole category.

• Identity: does a token issue?
• Database: do reads work, do writes work, is quorum intact?
• Message bus: any partition, any alarm, which queue is deepest and does it have consumers?
• Service state: read the pattern across the listings, not individual rows.

What you ran, what it returned, and what that eliminated. The third column is the one people skip and the one that stops you going in circles.

• Command / observation → Result → What this rules out
• (Repeat. Keep the ones that returned nothing useful: they are eliminations too.)

State it before testing it, so you notice when you were wrong.

• Current hypothesis:
• If it is right, what else must be true?
• What would disprove it fastest?

The interventions that make an incident worse are all bookkeeping fixes that look harmless.

• Is this reversible? If not, what is the position you are giving up?
• Have you captured the evidence a postmortem will need?
• Have you said what you are about to do, before doing it?
• Could this affect a workload that is currently fine?

The useful questions are not "what broke".

• What did we check that told us nothing?
• Which step would have found it, and when did we actually take it?
• What would have caught this earlier that does not exist yet?
• Did any intervention make it worse?