Incident Response — A Method for Working an OpenStack Failure

The first question is not what broke

It is: what is the blast radius? Everything about how you proceed depends on the answer, and it takes under a minute to establish.

  Are running workloads affected?
    │
    ├── No  ──► control plane only. Users can use what exists;
    │           they cannot change anything. Urgent, not critical.
    │           You have time to diagnose properly.
    │
    └── Yes ──► data plane. Something users depend on right now is
                down. Mitigate first, diagnose second.

This distinction governs the whole incident. A control plane failure — every API down, every service reporting down — is alarming and usually not an outage for anyone running a workload. Treating it as a five-alarm emergency produces rushed interventions that cause the real outage.

Then: is it one thing or everything?

The scope of a failure narrows the cause enormously, and OpenStack’s dependency structure makes the mapping fairly reliable.

Scope	Look at
Everything, at once	Shared dependencies: message bus, database, identity, load balancer
One service, everywhere	That service — its own processes and its own backend
All services, one host	That host — network, clock, disk, its agents
One tenant	Quota, roles, that tenant’s networks and security groups
Intermittent	Load balancer with one bad backend, a partitioned cluster, or multi-writer database

The first and last rows are the ones people get wrong. Everything failing simultaneously is almost never a coincidence of separate failures — it is one shared thing. And intermittent failure has a short, specific list of causes that is worth checking directly rather than searching for.

The elimination sequence

Run in order. Each step is cheap and removes a category.

1. Is identity working?

openstack token issue

If not, nothing downstream tells you anything. Everything will fail and none of it is informative.

2. Is the message bus healthy?

Cluster status, alarms, and queue depths sorted descending. The deepest queue with no consumer usually names the failed component outright.

3. Is the database healthy and writable?

Cluster size on every node, and whether reads work while writes fail — that pattern is quorum.

4. What do the service listings say?

openstack compute service list --long
openstack network agent list
openstack volume service list

Read the pattern, not the individual rows. All down means a shared dependency and you have already checked those. One host down across several services means that host.

5. Now look at the failing operation.

Only now. With identity, bus, database, and service state established, the error you started with is finally interpretable — and quite often already explained.

Mitigate before diagnosing, when workloads are affected

When something users depend on is down, restoring service comes first and the cause can wait. That is a real trade — mitigating often destroys evidence — so it should be a deliberate choice rather than a reflex.

Mitigations that are usually safe:

• Disabling a bad compute service so nothing new is scheduled to it. Reversible, and stops the problem spreading.
• Removing an unhealthy backend from the load balancer. Reversible, and immediately stops the fraction of requests that were failing.
• Migrating workloads off a failing host, when it is healthy enough to migrate.

Before destroying evidence, capture it: the service listings, the relevant logs, the queue depths. Ten seconds of docker logs > /tmp/incident-*.log preserves what the postmortem will otherwise lack.

The interventions that make it worse

Three more worth naming explicitly, each covered in an earlier lesson:

• virsh destroy to resolve a Nova/libvirt disagreement. If the instance is genuinely running, that is an unannounced outage for a real workload.
• Resetting a volume’s state while the backend still has it exported. Two hosts writing to one device corrupts the filesystem quickly.
• Removing a placement allocation without confirming the instance is gone. Causes under-counting, over-scheduling, and memory exhaustion later.

Each is a bookkeeping fix that looks harmless and is not.

Communicating during the incident

Two things are worth saying early, and both are frequently omitted:

Whether workloads are affected. This is the only fact most people actually need, and it is usually determined in the first minute. Saying it early stops a great deal of parallel panic.

What you are about to do, before doing it. Particularly anything that restarts a service. The cost is one line; the benefit is that nobody attributes the resulting blip to a second, imaginary fault.

After: the questions that produce useful findings

The valuable postmortem questions are not “what broke” but:

• What did we check that told us nothing? Those checks are candidates for deletion or replacement.
• What would have caught this earlier? Usually a functional check that does not exist yet.
• Which step in the elimination sequence would have found it, and when did we actually take it? The gap between those is the time the method would have saved.
• Did any intervention make it worse? Recording this honestly is what stops it recurring.

Exercise

Take a real incident you have worked and re-run it on paper using the elimination sequence. Write down what each of the five steps would have returned, in order.

Identify the step that would have located the cause. Then compare against what actually happened: how far into the incident was that step taken, and what was done before it. The difference is the value of the method, measured in your own environment rather than in the abstract.