Cinder Volume Creation Fails With "No Valid Backend Was Found"

Purpose

Restore volume provisioning when the Cinder scheduler reports that no backend is available. This covers the common case where cinder-volume has stopped reporting capacity while the API layer continues to accept requests.

Impact

Service-affecting. No new volumes can be created, and any instance build that requires a new volume will fail. Existing volumes remain attached and usable — this is a provisioning outage, not a data-path outage.

Preconditions

• Administrative credentials for the cloud (admin project, or a role that can list services)
• Shell access to the control node running cinder-scheduler and cinder-volume
• If deployed with Kolla-Ansible, access to run docker commands on that node

Safety considerations

Every step in the diagnosis section is read-only. The remediation section restarts services, which is safe for cinder-volume and cinder-scheduler — both are stateless and will re-register.

Initial checks

Run all three. They take seconds and they separate the three distinct causes.

$ openstack volume service list+——————+———––+——+———+—––+| Binary | Host | Zone | Status | State |+——————+———––+——+———+—––+| cinder-scheduler | ctl1 | nova | enabled | up || cinder-volume | ctl1@ceph | nova | enabled | down |+——————+———––+——+———+—––+

# 2. What is the scheduler actually complaining about?
docker logs --tail 100 cinder_scheduler 2>&1 | grep -iE 'no valid backend|capabilities|filter'

# 3. Is the volume service reaching the message bus?
docker exec rabbitmq rabbitmqctl list_queues name messages consumers | grep -i cinder

Decision tree

Volume creation failing.

├─ Is cinder-volume enabled and down?
│   ├─ yes → the service is not reporting. Go to Diagnosis A.
│   └─ no  → continue
│
├─ Is cinder-volume up but reporting no capabilities?
│   ├─ yes → the backend is unreachable from the volume host. Diagnosis B.
│   └─ no  → continue
│
├─ Are capabilities reported but every backend filtered out?
│   ├─ yes → capacity or filter configuration. Diagnosis C.
│   └─ no  → continue
│
└─ Is the volume type requesting a backend that does not exist?
    └─ Diagnosis D.

Diagnosis A — service enabled but down

The scheduler has no candidate backends because cinder-volume has stopped sending its periodic capability report.

# Is the container running at all?
docker ps -a --filter name=cinder_volume --format '{{.Names}}\t{{.Status}}'

# What did it say before it stopped reporting?
docker logs --tail 200 --timestamps cinder_volume 2>&1 | tail -60

Three sub-cases, distinguished by what the logs show:

Container is restarting. Read the logs for the startup failure. Usually a configuration error or an unreachable backend at initialisation.

Container is running, logs stop abruptly. The service is alive but not communicating. This is a message bus problem — check the queue consumer count from the initial checks. A cinder queue with messages and zero consumers confirms it.

Container is running and logging normally. The reports are being generated but not arriving. Check clock skew between the volume host and the control plane — the service state is derived from report timestamps, and a host whose clock is far ahead or behind will be marked down while working perfectly.

timedatectl status | grep -E 'System clock|synchronized'

Diagnosis B — up but reporting no capabilities

The service is running and reachable, but cannot talk to its storage backend, so it has nothing to report.

# The driver logs the backend connection attempt at startup
docker logs cinder_volume 2>&1 | grep -iE 'driver|backend|initialization|ceph|rbd' | tail -40

For a Ceph backend, verify connectivity from the volume host, not from a control node that happens to have the tooling installed:

docker exec cinder_volume ceph -s --conf /etc/ceph/ceph.conf
docker exec cinder_volume rbd ls -p volumes | head

Common causes: a keyring that was rotated, a monitor address that changed, a network path that was firewalled, or a Ceph cluster in a state that refuses IO.

Diagnosis C — capabilities reported but everything filtered out

The scheduler has candidates and rejects them all. The log names the filter.

docker logs cinder_scheduler 2>&1 | grep -iE 'filter|returned 0 host' | tail -30

Then check the actual reported capacity:

openstack volume backend pool list --long 2>/dev/null || \
  docker logs cinder_scheduler 2>&1 | grep -i 'free_capacity' | tail -5

If free_capacity_gb is genuinely near zero, this is a capacity problem and the remediation is capacity, not this runbook.

If capacity is available and the capacity filter still rejects it, check max_over_subscription_ratio and reserved_percentage for the backend — a reserved percentage set high enough will filter out a pool with plenty of space.

Diagnosis D — volume type points at a missing backend

openstack volume type list
openstack volume type show <type> -c properties

If the type has volume_backend_name=fast-ssd and no running backend reports that name, every request for that type fails scheduling while the default type works fine. The tell is that some volume creates succeed and others do not.

# What backend names are actually being reported?
docker logs cinder_scheduler 2>&1 | grep -oE 'volume_backend_name[^,}]*' | sort -u

Remediation

For Diagnosis A — service not reporting

# Restart the volume service. Stateless; safe.
docker restart cinder_volume

# Watch it register
sleep 30 && openstack volume service list

If it comes back up and then goes down again within a few minutes, the restart is not the fix — return to Diagnosis A and read the logs from the new startup.

For clock skew:

sudo timedatectl set-ntp true
sudo systemctl restart systemd-timesyncd    # or chronyd, per your deployment

For Diagnosis B — backend unreachable

Fix the underlying connectivity — keyring, monitor addresses, network path, or the storage cluster itself — then restart the volume service so it re-initialises the driver:

docker restart cinder_volume

For Diagnosis C — filter configuration

Correct the value in your Kolla configuration override rather than in the generated file:

# /etc/kolla/config/cinder/cinder-volume.conf
[ceph]
reserved_percentage = 5
max_over_subscription_ratio = 2.0

kolla-ansible reconfigure -i /etc/kolla/inventory --tags cinder

For Diagnosis D — volume type

Either correct the volume type’s volume_backend_name to match a backend that exists, or bring up the backend it expects.

openstack volume type set --property volume_backend_name=<actual-name> <type>

Validation

All four checks should pass before you consider this resolved.

# 1. Service is enabled and up
openstack volume service list

# 2. Create a small test volume
openstack volume create --size 1 runbook-validation

# 3. It reaches available, not error
openstack volume show runbook-validation -c status -c size

# 4. Clean up
openstack volume delete runbook-validation

# 5. No new scheduler errors since the fix
docker logs --since 5m cinder_scheduler 2>&1 | grep -ic 'no valid backend'
# Expected: 0

Rollback

Nothing in this runbook destroys data or changes state that cannot be reverted.

• Service restarts — no rollback needed; the service either registers or it does not.
• Configuration changes — revert the file in /etc/kolla/config/ and run the same reconfigure command with the same --tags.
• Volume type property changes — record the original value before changing it, and set it back with the same command.

# Capture the original before changing anything
openstack volume type show <type> -c properties > /tmp/voltype-before.txt

Escalation

Escalate to storage engineering if:

• The Ceph cluster itself is unhealthy (ceph -s shows anything other than HEALTH_OK, and the reason is not a known transient rebalance)
• The backend is reachable and initialising but reports zero capacity with the storage cluster reporting free space
• The problem recurs within hours of remediation, which suggests an underlying instability rather than a discrete failure

Escalate to network engineering if the volume host cannot reach the storage network and the cause is not a configuration change you can identify.

Include in the escalation: the output of openstack volume service list, the last 100 lines of both cinder_volume and cinder_scheduler, the queue consumer counts, and what you have already ruled out.