Lab: Diagnose a Nova Compute Failure Returning NoValidHost

The situation

You are on call. At 09:12 the platform team reports that new instances are failing to build. Existing instances are unaffected. The Horizon dashboard shows the cloud at roughly 40% capacity.

The error returned to users:

No valid host was found. There are not enough hosts available.

Nothing was deployed today. The last change to the cloud was a compute node added on Friday.

What you have

Before reading further, decide what you would run first. Then look at the output below and see whether it says what you expected.

Compute services

+––––––––+———––+–––––+———+—––+––––––––––––––+| Binary | Host | Zone | Status | State | Updated At |+––––––––+———––+–––––+———+—––+––––––––––––––+| nova-conductor | ctl1 | internal | enabled | up | 2026-08-24T09:48:11.000000 || nova-scheduler | ctl1 | internal | enabled | up | 2026-08-24T09:48:09.000000 || nova-compute | compute-01 | nova | enabled | up | 2026-08-24T09:48:07.000000 || nova-compute | compute-02 | nova | enabled | up | 2026-08-24T09:48:10.000000 || nova-compute | compute-03 | nova | enabled | up | 2026-08-24T09:48:06.000000 || … | … | nova | enabled | up | … || nova-compute | compute-12 | nova | enabled | up | 2026-08-24T09:48:08.000000 |+––––––––+———––+–––––+———+—––+––––––––––––––+

All twelve compute services are enabled and up. Every timestamp is recent.

Hypervisors

+––+———————+—––+———–+—––+————+| ID | Hypervisor Hostname | State | vCPUs Used| vCPUs | Memory MB |+––+———————+—––+———–+—––+————+| 1 | compute-01 | up | 24 | 96 | 393216 || 2 | compute-02 | up | 32 | 96 | 393216 || 3 | compute-03 | up | 28 | 96 | 393216 || .. | … | up | .. | 96 | 393216 || 12 | compute-12 | up | 20 | 96 | 393216 |+––+———————+—––+———–+—––+————+

Roughly a quarter to a third of vCPUs in use per host. Consistent with the dashboard.

The scheduler log

2026-08-24 09:47:52.118 INFO nova.scheduler.manager Starting to schedule instance 6c1e…2026-08-24 09:47:52.402 INFO nova.filters Filter ComputeFilter returned 12 host(s)2026-08-24 09:47:52.410 INFO nova.filters Filter AvailabilityZoneFilter returned 12 host(s)2026-08-24 09:47:52.418 INFO nova.filters Filter ComputeCapabilitiesFilter returned 0 host(s)2026-08-24 09:47:52.419 INFO nova.filters Filtering removed all hosts for the request with instance ID ‘6c1e…’. Filter results: [‘ComputeFilter: (start: 12, end: 12)’, ‘AvailabilityZoneFilter: (start: 12, end: 12)’, ‘ComputeCapabilitiesFilter: (start: 12, end: 0)’]

The failing request

$ openstack server show 6c1e… -c flavor -c status -c fault| flavor | gpu.large (8 vCPU, 65536 MB RAM) || status | ERROR |$ openstack flavor show gpu.large -c properties| properties | capabilities:gpu_model=‘a100’, || | hw:numa_nodes=‘2’ |

Stop and work it

Before reading the analysis, write down:

1. Which hypotheses does the evidence you have already eliminate?
2. What is the single most informative next command?
3. What would you expect that command to return if your leading hypothesis is right?

---

Working the evidence

What is already eliminated

Service liveness. All twelve compute services are enabled and up with recent timestamps. The control plane is participating. This is not the common “enabled but down” case.

Genuine capacity exhaustion. Hypervisors report a quarter to a third utilisation, and — more conclusively — ComputeFilter returned twelve hosts. ComputeFilter is the one that removes hosts which cannot host the instance for basic resource reasons. Twelve hosts passing it means capacity is not the constraint.

A recent change. Nothing was deployed today. The last change was Friday.

That is three hypotheses eliminated from evidence you already had, before running anything new.

What the log actually says

The filter chain is explicit:

ComputeFilter:              12 → 12    (capacity is fine)
AvailabilityZoneFilter:     12 → 12    (zone is fine)
ComputeCapabilitiesFilter:  12 →  0    ← everything died here

ComputeCapabilitiesFilter matches flavor capabilities: extra specs against what each compute node reports about itself. The flavor requires capabilities:gpu_model='a100'.

So the question is no longer “why is there no capacity”. It is: why does no host report the capability this flavor requires?

The most informative next command

# What do the hosts actually report?
openstack hypervisor show compute-01 -c cpu_info -c service_details

# And what does placement think exists?
openstack resource provider list
openstack resource provider trait list <compute-01-uuid>

If the leading hypothesis — hosts no longer report the GPU capability — is right, the trait or capability will be absent on hosts that previously had it.

The root cause

The compute node added on Friday was deployed from an updated configuration that omitted the custom capability declaration. That alone would only affect the new node.

The damaging part is what happened next: the deployment tool ran a full reconfigure rather than a host-limited one, regenerating configuration on every compute node from the same updated template. All twelve nodes lost the gpu_model capability declaration at the same time.

Nothing failed. No service went down. Every host is healthy and reports honestly that it has no gpu_model capability — which is now true, because the declaration is gone.

Instances using flavors without capability requirements continued to build normally throughout, which is why the failure looked partial and confusing rather than total.

Remediation

# Restore the capability declaration in the source of truth, not the generated file
# /etc/kolla/config/nova/nova-compute.conf

[compute]
# Host capabilities matched by flavor extra specs

Then apply, scoped to compute nodes:

kolla-ansible reconfigure -i /etc/kolla/inventory --tags nova --limit compute

Validation

# 1. Hosts report the capability again
openstack hypervisor show compute-01 -c service_details

# 2. A test instance with the affected flavor builds
openstack server create --flavor gpu.large --image <image> --network <net> lab-validation
openstack server show lab-validation -c status    # expect ACTIVE, not ERROR

# 3. The filter no longer eliminates everything
docker logs --since 5m nova_scheduler 2>&1 | grep -i 'ComputeCapabilitiesFilter'
# Expect a non-zero end count

# 4. Clean up
openstack server delete lab-validation

Validate with the flavor that was failing. A test with a plain flavor would have succeeded throughout the incident and tells you nothing.

What to take away

Read the filter chain, not the error. NoValidHost is the same message for a dozen different causes. The filter results line names which one, and it is in the log every time.

“Enabled and up” is not “configured correctly”. Service liveness is one property. A perfectly healthy service can report accurate information about a configuration that is wrong.

Scope your configuration management. A reconfigure that touched only the new host would have made this a one-node problem discovered during its own validation, rather than a cloud-wide outage discovered by users.

Validate a new node with the workload it is meant to run. The Friday deployment was validated by checking that the service registered — which it did.