Compute Troubleshooting — From Instance Symptom to Hypervisor Cause

Three layers, and the answer is in exactly one

A compute failure lives at one of three levels, and each has its own vocabulary:

1. Nova — the instance record, the task state, the fault field. Speaks in UUIDs and states.
2. libvirt — the domain definition and the hypervisor. Speaks in domain names and XML.
3. The host — kernel, storage, memory, devices. Speaks in dmesg and syscall errors.

Diagnosis is a descent. You start at Nova, and each layer either explains the failure or hands you a more specific question for the layer below. The mistake that costs the most time is jumping straight to the bottom, because the host almost always looks fine — it is running other instances perfectly well.

Establish the host first

Everything below this depends on knowing which host to log into.

$ openstack server show <id> -f value -c OS-EXT-SRV-ATTR:host -c OS-EXT-SRV-ATTR:instance_namecompute-03instance-000004a7

The second field is the libvirt domain name. Nova speaks UUIDs, libvirt speaks instance-XXXXXXXX, and the translation between them is this field. Not knowing it is why people grep libvirt logs for a UUID and find nothing.

The Nova layer: what the record already tells you

Only when the fault is generic, empty, or clearly hypervisor-shaped do you go further.

The relevant log is on the compute host, not the controller:

# On the compute host, scoped to the instance
docker logs nova_compute 2>&1 | grep <instance-uuid>

Two patterns are worth recognising immediately:

• A traceback ending in a libvirt error. Nova asked libvirt to do something and libvirt refused. The message is usually specific and is the real answer.
• A timeout with no libvirt error. Nova asked and never got a response. This points at the host being under pressure rather than at anything wrong with the request.

The libvirt layer: does the domain exist, and what does it think?

$ virsh list –all | grep instance-000004a7 - instance-000004a7 shut off$ virsh dominfo instance-000004a7$ virsh dumpxml instance-000004a7 | head -40

Three outcomes, three different problems:

No domain at all. Nova never got as far as defining it. The failure is above libvirt — go back to the nova-compute log; the answer is there.

Domain defined but shut off. libvirt was asked to create it and either failed to start it or it exited. The libvirt log for that domain is the next stop.

Domain running, Nova says ERROR. The two layers disagree. This is the state that needs care and is covered below.

The per-domain log is where hypervisor-level failures actually surface:

tail -100 /var/log/libvirt/qemu/instance-000004a7.log

The four failures that account for most of it

Memory the host does not have. The instance is scheduled by placement’s accounting, but libvirt allocates from the host’s actual memory. Overcommit ratios that placement honours and the kernel does not produce exactly this. The QEMU log says allocation failed; dmesg may show the OOM killer.

A missing or unreadable image file. The backing file for the instance disk is not where the domain XML says it is, usually after a failed migration or a cleanup that removed too much.

CPU model not available. The flavor or image requests CPU features the host does not offer. Common after adding hosts of a different generation to an existing cloud — the instance boots everywhere except on the new hardware, or only on it.

Device or PCI passthrough conflicts. The requested device is already claimed, or the host does not have it. Distinctive because it fails immediately and identically every time.

When Nova and libvirt disagree

This is the case that produces the most dangerous instincts.

An instance that Nova reports as ERROR or SHUTOFF while libvirt is running the domain means the two records have diverged — usually because a nova-compute restart interrupted a state transition, or because an operation was performed directly through virsh.

The supported reconciliation is to have Nova re-read the hypervisor rather than to edit either side by hand. openstack server reset-state changes what Nova believes and does not touch the domain — which makes it the right tool for a stuck record and completely the wrong tool for an instance that is actually broken, because it will report ACTIVE while remaining unusable.

The descent, in order

1. openstack server show — status, task state, host, instance name, fault
2. If the fault names another service, stop and investigate that service
3. nova-compute log on the recorded host, filtered by instance UUID
4. virsh list --all — does the domain exist, and what state is it in?
5. /var/log/libvirt/qemu/<domain>.log — the hypervisor’s own account
6. Host level: df -h on the instance path, free -m, dmesg -T | tail

Each step either answers the question or produces a sharper one. If a step tells you nothing, that is itself information: it eliminates that layer.

Exercise

Fill the filesystem holding /var/lib/nova/instances on a test compute host — a large sparse file is sufficient — and then try to boot an instance onto it.

Work the failure from the top without skipping. Note what the fault field says, what nova-compute logs, whether a domain is defined, and what the QEMU log reports. Write down which of those first named the actual cause, and how far you would have gone if you had started at the bottom.