Cinder Architecture — Volumes, Backends, and Capability Reporting

The service that talks to storage is not the one you check

Cinder has the same shape as Nova, and the same consequence follows from it.

Component	Owns	Talks to storage?
cinder-api	Accepting requests, writing the volume record	No
cinder-scheduler	Choosing a backend for each volume	No
cinder-volume	Driving the backend driver	Yes
cinder-backup	Backup operations	Only for backups

Only cinder-volume touches the array, the Ceph cluster, or the LVM group. Everything else works from the database and the message bus.

That is why cinder-api answering promptly means nothing about whether storage works. The API is healthy in almost every storage incident, because its health does not depend on storage.

Backends describe themselves, and the scheduler believes them

cinder-volume runs one driver per configured backend, and each periodically reports what it can do and how much room it has. The scheduler makes decisions purely from those reports.

+——————+———————+———+—––+| Binary | Host | Status | State |+——————+———————+———+—––+| cinder-scheduler | controller-01 | enabled | up || cinder-volume | storage-01@ceph | enabled | up || cinder-volume | storage-01@lvm | enabled | down |+——————+———————+———+—––+

Note the host@backend form. One host runs several backends and they fail independently — a fact that is easy to miss and explains why “storage is down” is usually too coarse a statement to act on.

The reported capabilities are what volume types are matched against:

openstack volume backend capability show storage-01@ceph

Volume states name the stage that failed

  creating   ──► available ──► reserved ──► attaching ──► in-use
      │                                          │
      ▼                                          ▼
    error                                  error_attaching

Each terminal error state points somewhere specific:

State	What failed	Look at
error after creating	Backend could not create the volume	cinder-volume on the backend host
error_attaching	The volume exists; connecting it to the instance failed	nova-compute on the instance’s host, then cinder-volume
error_deleting	Backend refused or failed to remove it	cinder-volume, then the backend itself
Stuck in creating	No backend accepted it, or the driver is not responding	cinder-scheduler first, then cinder-volume
Stuck in detaching	Cleanup not completing on the compute side	nova-compute on the recorded host

The distinction between error and error_attaching is the most useful one in practice: the first is a storage problem, the second is very often a compute or connectivity problem with storage as an innocent bystander.

Why volume attachment is the fragile part

Creating a volume involves Cinder and the backend. Attaching one involves Cinder, Nova, the compute host, and the data path between that host and the storage — iSCSI sessions, Ceph connectivity, or fibre channel.

That is a much longer list of things that can be individually broken, and the failure is reported by Nova because Nova was orchestrating. An instance stuck in block_device_mapping is a Cinder problem being reported by Nova, and looking only at Nova will not resolve it.

The failure that keeps the API green

The most confusing Cinder incident is the one where every check passes and volumes do not work:

• cinder-api responds immediately
• openstack volume list returns instantly
• The dashboard shows storage as healthy
• New volumes sit in creating forever

This happens when cinder-volume is running but its driver cannot reach the backend. The service still heartbeats — it is alive — so State shows up. It simply cannot do any work.

The check that reveals it is not a health check but a functional one: create a small volume and see whether it reaches available. Anything less than that is testing the parts of Cinder that do not touch storage.

The diagnostic path

1. Is the backend service up? openstack volume service list — and read the host@backend carefully, because one backend can be down while others are fine.
2. What state is the volume in? It names the stage that failed.
3. Creation failure? cinder-volume log on that backend’s host.
4. Attachment failure? nova-compute on the instance’s host first, then the data path from that host to storage.
5. Scheduling failure? cinder-scheduler log, and compare the volume type’s extra specs against what the backends actually report.
6. Everything green but nothing works? Create a test volume. A functional check is the only one that exercises the driver.

Exercise

Read the capabilities your backend reports with openstack volume backend capability show, and note the free space it claims. Compare that against what the storage system itself reports — they are frequently not the same number, and understanding why for your backend is worth the time.

Then create a volume type with an extra spec that no backend advertises and try to create a volume with it. Record the error and where it appears. Compare it against the error from a genuinely full backend: both prevent volume creation, both surface at the scheduler, and the fix for one is not the fix for the other.