Nova Scheduling — Placement, Filters, and Reading a Rejection

Two stages, not one

Scheduling an instance happens in two distinct stages, and knowing which one rejected your request is most of the diagnosis.

request
   │
   ▼
┌────────────────────────────────────────────┐
│ 1. PLACEMENT                               │
│    "which hosts have the raw resources?"   │
│    vCPU, memory, disk, and any custom      │
│    resource classes. Returns candidates.   │
└────────────────────────────────────────────┘
   │  candidate list
   ▼
┌────────────────────────────────────────────┐
│ 2. FILTERS                                 │
│    "which candidates are actually          │
│     suitable?"                             │
│    Each filter removes hosts. Survivors    │
│    are weighed and one is chosen.          │
└────────────────────────────────────────────┘
   │
   ▼
 chosen host  (or NoValidHost, if zero survive)

Placement answers a quantitative question: is there room? Filters answer qualitative ones: is this host in the right zone, does it have the required capability, does it satisfy the affinity rule?

NoValidHost means zero hosts survived. It does not tell you which stage ended it — but the log does.

Reading the filter chain

This is the single most useful skill in this lesson.

Filter ComputeFilter returned 12 host(s)Filter AvailabilityZoneFilter returned 12 host(s)Filter ComputeCapabilitiesFilter returned 0 host(s)Filtering removed all hosts for the request with instance ID ‘6c1e…’.Filter results: [‘ComputeFilter: (start: 12, end: 12)’, ‘AvailabilityZoneFilter: (start: 12, end: 12)’, ‘ComputeCapabilitiesFilter: (start: 12, end: 0)’]

Read the start and end pairs. The filter where end becomes zero is your answer, and it is printed on every failed scheduling attempt.

docker logs nova_scheduler 2>&1 | grep -iE 'returned 0 host|Filter results' | tail -20

Two seconds of grep replaces an hour of hypothesising.

What the common filters actually check

Filter	Rejects a host when	Typical cause of a zero
ComputeFilter	The compute service is down or cannot serve the request	Services enabled but down
AvailabilityZoneFilter	The host is not in the requested zone	Zone named in the request has no live hosts
ComputeCapabilitiesFilter	Host capabilities do not match flavor extra specs	A capability declaration was removed or misspelled
ImagePropertiesFilter	Host cannot satisfy image properties (architecture, hypervisor)	Image property set that no host provides
AggregateInstanceExtraSpecsFilter	Host aggregate metadata does not match	Flavor scoped to an aggregate with no live hosts
ServerGroupAntiAffinityFilter	Host already runs a member of the group	Anti-affinity group larger than the host count
NUMATopologyFilter	Host cannot satisfy the requested NUMA layout	hw:numa_nodes exceeding what hosts have

Anti-affinity is worth knowing specifically: a group with a strict anti-affinity policy and more members than you have hosts is permanently unschedulable, and it produces a NoValidHost that no amount of added capacity will fix.

When placement is the problem

If the filter chain starts with fewer hosts than you have, placement is not offering them.

openstack hypervisor list
openstack resource provider list
openstack resource provider inventory list <provider-uuid>
openstack resource provider allocation list <provider-uuid>

Three things to check, in order:

Is the host a resource provider at all? A compute node that deployed but did not register with placement will appear in hypervisor list and not in resource provider list. It will never receive an instance and nothing will tell you.

Does its inventory look right? Inventory is what the host says it has. Compare it against the hardware.

Do the allocations correspond to real instances? This is the common failure. Orphaned allocations — left behind when an instance was deleted during a compute service crash — consume capacity that nothing is using.

nova-manage placement audit --verbose

Extra specs and traits

Flavor extra specs constrain placement. They are powerful and they are the most common way to create a flavor that cannot be scheduled anywhere.

openstack flavor set gpu.large --property capabilities:gpu_model=a100
openstack flavor set gpu.large --property hw:numa_nodes=2
openstack flavor set gpu.large --property trait:CUSTOM_GPU_A100=required

Each property is a constraint that some host must satisfy. If none does, every boot of that flavor fails — and only that flavor, which is why the failure looks partial and confusing.

Before creating a flavor with a constraint, verify something satisfies it:

# Which providers have the trait?
openstack resource provider list --required CUSTOM_GPU_A100

# What traits does a given host actually have?
openstack resource provider trait list <provider-uuid>

The diagnostic path

NoValidHost.

1. Read the filter results line.
   ├─ A filter went from N to 0 → that filter's criteria. Go to 3.
   └─ The first filter started low → placement. Go to 2.

2. Placement
   ├─ Host missing from resource provider list  → registration
   ├─ Inventory wrong                            → host reporting
   └─ Allocations exceed real instances          → orphans; audit in report mode

3. Filters
   ├─ ComputeFilter              → check service state first
   ├─ ComputeCapabilitiesFilter  → flavor extra specs vs host capabilities
   ├─ AvailabilityZoneFilter     → live hosts in the requested zone
   ├─ ImagePropertiesFilter      → image properties vs host support
   ├─ Aggregate*                 → aggregate metadata and membership
   └─ ServerGroup*               → group size versus host count

Exercise

Make it fail deliberately, then find it in the logs.

# 1. Create a flavor requiring something no host has
openstack flavor create --vcpus 2 --ram 2048 --disk 10 lesson-unschedulable
openstack flavor set lesson-unschedulable --property capabilities:nonexistent_thing=true

# 2. Attempt to boot it
openstack server create --flavor lesson-unschedulable --image <image> \
  --network <net> lesson-test

# 3. Find the filter that rejected every host
docker logs --since 5m nova_scheduler 2>&1 | grep -i 'Filter results'

Then remove the property, boot again, and confirm the filter chain completes with a non-zero count.

openstack flavor unset lesson-unschedulable --property capabilities:nonexistent_thing
openstack server delete lesson-test
openstack flavor delete lesson-unschedulable

The value of doing this deliberately is that you will have seen the log line before you need to find it under pressure. The Nova NoValidHost lab takes this further with a scenario where the cause is not the flavor.