Storage Troubleshooting — No Valid Backend and the Attachment Path

One message, five causes

No valid backend was found is Cinder’s NoValidHost. It means the scheduler evaluated every backend and none survived, and it does not say which test they failed.

The causes, in rough order of frequency:

1. The volume type requests something no backend advertises — an extra spec mismatch
2. Every backend genuinely lacks the space
3. Backends have space but report themselves reserved — the reserved percentage
4. The only qualifying backend is down — cinder-volume for that backend is not reporting
5. Availability zone mismatch — the request specifies a zone with no backend in it

Cause 3 is the one that produces the most confusion, because the array has room and Cinder is correct to refuse.

Start by comparing the request against what exists

Nearly every instance of this failure is resolved by putting two outputs side by side.

$ openstack volume type show fast-ssd| name | fast-ssd || properties | volume_backend_name=‘ssd-tier’ |# what actually exists$ openstack volume service list -c Binary -c Host -c State| cinder-volume | storage-01@ceph-hdd | up |

A volume_backend_name that no running backend advertises fails 100% of the time, immediately, regardless of capacity — and it is the single most common cause. Volume types outlive the backends they were written for.

The scheduler log states the outcome plainly:

docker logs cinder_scheduler 2>&1 | grep -i "no valid backend" -A5

Distinguishing “no capacity” from “no match”

These need different responses and are easy to confuse.

No match fails instantly and identically every time, for every size, including a 1 GB volume. That is the signature: if the smallest possible volume also fails, capacity is not the issue.

No capacity is size-dependent. A small volume succeeds and a large one fails, and the threshold moves as other volumes are created and deleted.

Over-subscription and reserved capacity

Two configuration values sit between the array’s free space and what Cinder will allocate, and both are routinely forgotten.

reserved_percentage withholds a fraction of each backend from scheduling. With this at 10, a backend reporting 10 TB will refuse allocations past 9 TB — correctly, deliberately, and to the surprise of whoever is watching the array report 1 TB free.

max_over_subscription_ratio applies to thin-provisioned backends and allows Cinder to allocate more logical space than physically exists, on the assumption that volumes are not all full. A ratio of 20 permits 200 TB of volumes on 10 TB of storage.

Attachment failures are a different investigation

Attachment involves the compute host and the data path, so the checks move there.

  instance on compute-03
        │  needs a volume
        ▼
  nova-compute  ──► cinder-api      "give me a connection"
        │           cinder-volume   "here is how to reach it"
        ▼
  data path: iSCSI session / Ceph connection / FC fabric
        │
        ▼
  storage backend

The failure is reported by Nova, at block_device_mapping, and it is generally not Nova’s fault.

Work it in this order:

1. Is the volume available? A volume already in-use elsewhere cannot attach here unless it is multi-attach.
2. Can the compute host reach the storage network? From that host, not from a controller.
3. Is the initiator configured? For iSCSI, the host needs an initiator name and the backend needs to accept it. A new compute host missing this attaches nothing while looking healthy.
4. What does cinder-volume say? It logs the connection information it handed back.
5. What does the compute host say? nova-compute logs the attempt to use it, and the kernel logs iSCSI or multipath failures in dmesg.

Volumes stuck in a transitional state

A volume in creating, attaching, or detaching for an extended period is waiting on something that will not arrive — usually because the service handling it restarted mid-operation.

The safe sequence:

1. Check the backend directly — is this volume mapped or exported to a host?
2. Check whether any instance believes it has the volume attached
3. Only when both say no, reset the state
4. Verify by attaching it somewhere deliberately and confirming it behaves

The diagnostic path

For a provisioning failure:

1. Try a 1 GB volume of the same type — capacity or matching?
2. Compare the type’s extra specs against openstack volume service list and the backends’ reported capabilities
3. Check the availability zone in the request against the backends’ zones
4. Read the scheduler log for the rejection
5. If capacity: compare Cinder’s figure against the array’s real free space, and check reserved_percentage and the over-subscription ratio

For an attachment failure:

1. Volume state — is it actually available?
2. Data path from the compute host to storage
3. cinder-volume log, then nova-compute log, then dmesg on the host

Exercise

Produce the same error message from two different causes. First, create a volume type whose volume_backend_name matches no running backend, and attempt a 1 GB volume. Second, fill a test backend and attempt a volume larger than the remaining space.

Both report no valid backend. Find the line in the scheduler log that differs between them, and confirm that the one-gigabyte test separates them without needing the log at all.

Then attach a volume to an instance and trace the path: identify the connection information cinder-volume returned, find the corresponding session or connection on the compute host, and confirm the device appears in the kernel. Knowing what that looks like when it works is what makes it recognisable when it does not.