OpenStack Production Operations — Reading a Cloud That Is Lying to You

OpenStack rarely fails by returning an error. It fails by continuing to accept requests while the thing that fulfils them has stopped participating. The API is up, the dashboard is green, and nothing works.

Learning to read that state is most of what production OpenStack operations consists of.

Status and State are different columns

Every OpenStack service listing reports two things that people read as one:

• Status — administrative intent. enabled or disabled. Someone set this.
• State — reality. up or down. Derived from whether the service reported in recently.

$ openstack compute service list+––––––––+————+———+—––+| Binary | Host | Status | State |+––––––––+————+———+—––+| nova-scheduler | ctl1 | enabled | up || nova-compute | compute-01 | enabled | up || nova-compute | compute-02 | enabled | down || nova-compute | compute-03 | disabled| down |+––––––––+————+———+—––+

compute-03 is disabled and down: somebody took it out deliberately. Fine.

compute-02 is enabled and down. The control plane still counts it as part of the cloud. It contributes to capacity arithmetic, it is a scheduling candidate, and nothing is answering there. That is the row that ruins your evening.

The first five minutes

Three commands, in this order, before anything else:

openstack compute service list
openstack network agent list
openstack volume service list

Neutron reports differently and it trips people up: the Alive column is liveness (:-) or XXX) and State is administrative. The opposite way round from compute and volume.

If all three are clean, service liveness is not your problem and you move on to placement, quota, and the message bus. If any service is enabled and down, stop and fix that first — everything downstream of it will produce misleading symptoms.

The message bus is the real dependency

Every OpenStack service talks to every other through RabbitMQ. Services do not connect to each other directly, which means a message bus problem presents as an unrelated service failing.

The characteristic symptom is a service that is running — the process is alive, it responds to a signal, its logs show it starting — but is reported as down because its periodic report is not arriving.

# Cluster health and partitions
docker exec rabbitmq rabbitmqctl cluster_status

# Queues that are growing are the ones nobody is consuming
docker exec rabbitmq rabbitmqctl list_queues name messages consumers \
  | awk '$2 > 100'

A queue with many messages and zero consumers identifies precisely which service stopped listening. That is often faster than reading any log.

Placement disagreements

Placement tracks resource inventory and allocations, and Nova schedules against it. When the two disagree, capacity appears consumed that is not, and you get NoValidHost on a cloud that is half empty.

openstack hypervisor list
openstack resource provider list
openstack resource provider inventory list <provider-uuid>
openstack resource provider allocation list <provider-uuid>

The usual cause is orphaned allocations: instances that were deleted in a way that did not clean up their allocation, typically after a compute service crash mid-operation.

# Nova ships a consistency check for exactly this class of problem
nova-manage placement audit --verbose

Diagnosing by symptom

Instances will not build

NoValidHost / "No valid host was found"

├─ Any compute service enabled but down?     → fix that first
├─ Hypervisors listed but not in placement?  → resource provider registration
├─ Placement shows capacity consumed?
│   ├─ genuinely consumed → capacity problem
│   └─ orphaned           → nova-manage placement audit
├─ Flavor requests something no host offers? → extra specs, traits, aggregates
└─ Scheduler filters excluding everything?   → read the scheduler log; the
                                               filter that returned zero is named

The scheduler log states which filter eliminated all hosts. That single line usually replaces an hour of speculation.

docker logs nova_scheduler 2>&1 | grep -iE 'filter returned 0|no valid host' | tail -20

Instances build but have no network

openstack network agent list                  # any agent not alive?
openstack port list --server <instance>       # does the port exist?
openstack port show <port-id>                 # is it ACTIVE and bound?

A port stuck in DOWN or BUILD means the L2 agent on that host did not wire it up. That is usually the agent being dead or the agent’s connection to the bus being broken — back to the message bus.

Volumes fail to create

openstack volume service list
docker logs cinder_scheduler 2>&1 | tail -50

No valid backend was found is the volume equivalent of NoValidHost, and the usual cause is identical in shape: cinder-volume stopped reporting capacity, so capacity filtering returns zero candidates. The Cinder scheduler runbook covers this procedure in full.

Everything is slow

Check the database before the services:

# Galera: is the cluster whole?
docker exec mariadb mysql -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
docker exec mariadb mysql -e "SHOW STATUS LIKE 'wsrep_local_state_comment';"

A Galera cluster that has lost quorum accepts reads and rejects writes, which produces timeouts scattered across every service and no obvious single cause.

What to check daily

# Service liveness across all three
openstack compute service list -f value -c Binary -c Host -c State | grep -v ' up$'
openstack network agent list -f value -c 'Agent Type' -c Host -c Alive | grep -v ':-)'
openstack volume service list -f value -c Binary -c Host -c State | grep -v ' up$'

# Instances that ended up in error
openstack server list --all-projects --status ERROR

# Message bus queues without consumers
docker exec rabbitmq rabbitmqctl list_queues name messages consumers | awk '$3 == 0 && $2 > 0'

Any output from these is worth investigating. Silence is the healthy result.

The pattern to internalise

Almost every confusing OpenStack incident resolves to one of three things:

1. A service is enabled but not reporting. The control plane thinks it exists; it does not answer.
2. The message bus is not delivering. Services are alive and not talking.
3. Placement and reality disagree. Capacity accounting is wrong in one direction or the other.

Check those three before investigating the service that appears to be failing. The service that appears to be failing is usually a victim.