Nova Architecture — Which Component Owns Which Failure

Nova is not one service

nova in conversation means a handful of separate processes that hand work to each other over the message bus. When an instance fails, exactly one of them is usually at fault, and naming it early is most of the diagnosis.

Component	Owns	Runs on
nova-api	Accepting requests, validating them, writing the initial database record	Controllers
nova-scheduler	Choosing which host will run the instance	Controllers
nova-conductor	Orchestrating the build, talking to the database on behalf of compute	Controllers
nova-compute	Actually creating the instance on a hypervisor	Every compute host
placement	Tracking what resources exist and what is allocated	Controllers

Two design decisions explain most of Nova’s behaviour:

nova-compute does not talk to the database. It goes through nova-conductor. This was done so a compromised compute host cannot reach the database directly, and it means a conductor problem presents as compute hosts being unable to do anything while looking perfectly healthy.

Nothing is synchronous. The API returns as soon as the record is written. Everything after that happens in the background, which is why openstack server create succeeding tells you almost nothing about whether the instance will exist.

The path of a boot request

  client
    │  POST /servers
    ▼
  nova-api ──────────► database (instance record, state: BUILDING)
    │  message bus
    ▼
  nova-scheduler ────► placement (which hosts can fit this?)
    │  message bus     ◄──── candidate list
    ▼
  nova-conductor ────► database (target host recorded)
    │  message bus
    ▼
  nova-compute ──────► hypervisor (create the domain)
       │             ► neutron   (plug the ports)
       │             ► cinder    (attach the volumes)
       ▼
     ACTIVE

Each arrow is a place the request can stop. The instance’s state tells you which arrow you did not get past.

Task state is the diagnostic field

openstack server list shows Status. That is the coarse view. The useful field is task_state, which names what is being attempted right now.

| OS-EXT-STS:vm_state | building || OS-EXT-STS:task_state | spawning || OS-EXT-SRV-ATTR:host | compute-03 || status | BUILD |

An instance stuck in a task state points directly at a component:

Stuck in	Waiting on	Open this log first
scheduling	The scheduler finding a host	nova-scheduler
block_device_mapping	Cinder creating or attaching a volume	cinder-volume, then the backend
networking	Neutron creating and binding ports	neutron-server, then the L2 agent on the host
spawning	The hypervisor on the target host	nova-compute on that host, then libvirt
deleting	Cleanup that is not completing	nova-compute on the recorded host
null with ERROR	Nothing — it has already failed	openstack server show fault field

Where the two most confusing failures live

Stuck in spawning forever. The scheduler picked a host, conductor recorded it, and nova-compute on that host is not finishing. The instance record looks healthy; the API is fine; the scheduler did its job. Only the compute host has the answer, and you need to know which host — which is why the host field matters more than it looks.

ERROR with no obvious cause. Nova recorded a failure that came from somewhere else. Volume attachment failures, port binding failures, and image download failures all surface as a Nova error, because Nova is the service that was orchestrating. The fault field usually names the real origin.

Placement is separate, and its state can be wrong

Placement tracks inventory (what a host has) and allocations (what is spoken for). The scheduler asks placement which hosts can fit a request; placement answers from its own records, not by asking the hosts.

This means placement can be wrong. Allocations are created during a build and removed during a delete, and a failure between those two points can leave an allocation for an instance that no longer exists. The result is a host that appears full while running very little.

$ openstack resource provider list –name compute-03$ openstack resource provider inventory list <uuid>$ openstack resource provider allocation list <uuid># compare against what is actually running there$ openstack server list –host compute-03 –all-projects

If placement claims more consumers than the host has instances, the difference is stale. This is the most common cause of a cloud that reports being full at 40% utilisation.

Which component to suspect first

Work from the evidence you already have rather than from where the error appeared:

1. Did the instance get a host? No host recorded means the scheduler never succeeded — a scheduling problem, not a compute problem.
2. What is the task state? It names the component currently responsible.
3. What does the fault field say? It often names a different service entirely.
4. Is the recorded host healthy? An enabled-but-down compute service will accept work it cannot do.
5. Do placement’s allocations match reality? Only relevant when scheduling fails despite apparent free capacity.

The instinct to start with nova-compute logs on a controller is a common and expensive mistake: nova-compute does not run there.

Exercise

Boot an instance and poll openstack server show every couple of seconds, recording the sequence of task states and which component owns each. You should see scheduling, then networking, then spawning, then a transition to ACTIVE with a null task state.

Then create a failure deliberately. Stop nova-compute on a host, force an instance onto it, and observe: which state it stops in, how long the cloud takes to notice the service is down, and what the fault field says once it gives up. The lag between the failure and the cloud noticing is the number worth remembering.