The OpenStack Control Plane — Who Talks to Whom

The single most useful fact

OpenStack services do not talk to each other directly.

Nova’s API does not call Nova’s scheduler. It puts a message on a queue, and the scheduler picks it up. The same is true across every service boundary in the system.

Everything about operating OpenStack follows from this. It is why a healthy service can be reported as down, why a message bus problem presents as an unrelated service failing, and why “restart the service that is failing” so often does nothing.

The request path

Creating an instance touches most of the control plane:

  client
    │  POST /servers
    ▼
┌─────────────┐
│ keystone    │  validate token, return service catalog
└─────────────┘
    │
    ▼
┌─────────────┐
│ nova-api    │  validate request, check quota, write DB record
└─────────────┘
    │  ──── message bus ────▶
    ▼
┌──────────────┐
│nova-scheduler│  ask placement for candidates, run filters, pick a host
└──────────────┘
    │                      │
    │                      ▼
    │              ┌─────────────┐
    │              │ placement   │  inventory and allocations
    │              └─────────────┘
    │  ──── message bus ────▶
    ▼
┌──────────────┐
│nova-conductor│  orchestrate, talk to the database on compute's behalf
└──────────────┘
    │  ──── message bus ────▶
    ▼
┌──────────────┐      ┌──────────┐   image
│ nova-compute │◀────▶│ glance   │
└──────────────┘      └──────────┘
    │        │
    │        ├────▶ neutron    port, network wiring
    │        └────▶ cinder     volume, if requested
    ▼
  libvirt → the instance runs

Every arrow marked “message bus” is asynchronous. The caller does not wait for a reply on an open connection; it publishes and moves on.

What each service owns

Service	Owns	Fails as
Keystone	Identity, tokens, service catalog	Everything fails authentication at once
Nova	Instance lifecycle, scheduling	NoValidHost, instances stuck in BUILD
Placement	Resource inventory and allocations	Phantom capacity exhaustion
Neutron	Networks, ports, routers, security groups	Instances build with no connectivity
Cinder	Volumes and volume scheduling	Volume creation fails, instances needing volumes fail
Glance	Images and metadata	Slow or failing instance builds
RabbitMQ	Message delivery between all of the above	Services alive but reported down
MariaDB	Persistent state for every service	Timeouts scattered everywhere with no single cause

The bottom two rows are the ones to internalise. They are not OpenStack projects, and they cause more confusing incidents than all the projects above them.

The message bus

Every service maintains a connection to RabbitMQ and consumes from queues addressed to it.

Services also publish periodic reports — “I am alive, here is my capacity” — and those reports are what the control plane uses to decide whether a service is up.

That produces the characteristic OpenStack failure:

Process running          ✓
Logs showing activity    ✓
Reported as "up"         ✗

The service is fine. Its reports are not arriving. From the control plane’s perspective it is indistinguishable from a dead service, and it will be excluded from scheduling.

docker exec rabbitmq rabbitmqctl cluster_status
docker exec rabbitmq rabbitmqctl list_queues name messages consumers | awk '$2 > 100'

A queue with messages and zero consumers names precisely which service stopped listening. It is frequently faster than reading logs.

The database

Every service stores its state in MariaDB, usually as a Galera cluster.

Galera requires quorum to accept writes. A cluster that has lost quorum will serve reads and reject writes, which produces a very specific symptom: you can list things, but you cannot create anything. Dashboards populate normally. Every action fails.

docker exec mariadb mysql -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
docker exec mariadb mysql -e "SHOW STATUS LIKE 'wsrep_local_state_comment';"

wsrep_cluster_size should equal your control node count, and wsrep_local_state_comment should read Synced.

Predicting failures

Work through these before reading the answers. Given the dependency graph above, what happens when:

Keystone is down? Every request fails authentication. Existing instances keep running — they do not need Keystone — but no API operation succeeds anywhere. It looks like total cloud failure and is one service.

RabbitMQ is partitioned? Services stay alive but stop coordinating. Some are reported down. Operations start and never complete. Two halves of the cluster may both look healthy from inside.

Placement is down? Scheduling fails, because the scheduler cannot get candidates. Running instances are unaffected. Presents as NoValidHost on a cloud with free capacity.

One nova-compute is down? Instances on that host keep running (libvirt does not need nova-compute), but you cannot manage them, and the host will not receive new ones once it is marked down.

Glance is down? New instance builds fail while downloading the image. Everything else works. Presents as builds timing out rather than as an image error.

That last one is a good illustration of the general pattern: the symptom appears in Nova, and the cause is in Glance. Diagnosing OpenStack is largely the practice of following the dependency graph rather than investigating whichever service reported the error.

Checking the whole control plane

# The three service listings
openstack compute service list
openstack network agent list
openstack volume service list

# The two dependencies underneath them
docker exec rabbitmq rabbitmqctl cluster_status
docker exec mariadb mysql -e "SHOW STATUS LIKE 'wsrep_cluster_size';"

# Does the catalog resolve?
openstack catalog list

Five commands. Run them in this order at the start of any OpenStack incident and you will have eliminated most of the possibilities before you have opened a log file.

Exercise

Trace a single instance creation end to end on a cloud you have access to.

openstack server create --flavor <flavor> --image <image> --network <net> trace-test

Then follow it through the logs using the request ID:

docker logs nova_api 2>&1 | grep <request-id>
docker logs nova_scheduler 2>&1 | grep <request-id>
docker logs nova_conductor 2>&1 | grep <request-id>
docker logs nova_compute 2>&1 | grep <instance-id>

Write down the order the services appeared in, and how long each stage took. Then find the service in the diagram above that you could not account for in the logs, and work out what it was doing.

The point of the exercise is not the answer. It is that you will need this exact skill — following one request across services — during the first real incident you work.