RabbitMQ — The Dependency Every OpenStack Service Shares

Every service conversation goes through the broker

OpenStack services do not call each other over HTTP. Except for the public REST APIs, they talk over RPC on a message bus, and that bus is RabbitMQ in essentially every production deployment.

  nova-api ──┐
             │
  nova-scheduler ──┤
             │      RabbitMQ  ── the only thing every service shares
  nova-compute ────┤
             │
  neutron-* ──┘

Two consequences follow, and they explain most confusing OpenStack incidents:

A broker problem looks like every service failing. Not because the services are broken, but because none of them can talk. Each logs its own timeouts in its own vocabulary and none of them mentions RabbitMQ prominently.

Liveness runs over the same bus. State: up is derived from heartbeats published to the broker. A broker problem therefore makes healthy services appear down — the report is not arriving, and nothing distinguishes that from the service being gone.

Queues, and what their depth means

Each service consumes from named queues. Two numbers describe a queue’s health, and the pair matters more than either alone.

$ rabbitmqctl list_queues name messages consumers | sort -k2 -rn | headq-agent-notifier-port-update_fanout_9f2e 184203 0scheduler.controller-01 3 1compute.compute-03 0 1

Messages	Consumers	Meaning
Low	≥1	Normal
High	0	Nobody is reading. The consumer died and left the queue behind.
High	≥1	A consumer exists and cannot keep up, or is stuck
Low	0	An idle queue with no consumer — often harmless, sometimes a stale fanout

The high-with-zero-consumers case is the one that identifies a failed service by name: the queue is named after the service and the host, so the queue that is growing tells you exactly which process to investigate.

Growth is the leading indicator

A queue that is growing steadily is a service that is failing now and has not yet been noticed. Depth is more useful sampled twice than read once — the absolute number matters far less than whether it is rising.

Unbounded growth eventually becomes a broker problem in its own right. RabbitMQ holds messages in memory until a threshold, then pages to disk, and past a further threshold it applies back pressure and blocks publishers. When that happens, the original fault — one stalled service — becomes an outage of everything, because nobody can publish.

This is why one stuck agent can take down a cloud several hours later, with no apparent connection between cause and effect.

The failure modes worth recognising

A partition. Cluster nodes lose contact and each believes it is authoritative. Clients get different answers depending on which node they reached, producing intermittent, inconsistent behaviour that is maddening to diagnose. rabbitmqctl cluster_status reports partitions explicitly, and it is worth checking early precisely because the symptoms do not suggest it.

Memory or disk alarm. The broker blocks publishers to protect itself. Services hang on publish rather than erroring, so they appear stuck rather than failed.

Stale queues after a restart. A service that restarts may create new queues while the old ones remain, unconsumed, accumulating messages until they expire.

Connection churn. Services reconnecting in a loop — visible as a connection count far higher than the number of services — usually means credentials or TLS problems rather than a broker fault.

Restart order matters

The instinct during a broker incident is to restart RabbitMQ. When that is genuinely necessary, the order that follows is not optional.

The sequence that behaves best:

1. Identify the actual fault — usually one service, named by its queue
2. Fix or restart that service, and watch whether its queue drains
3. Only if the broker itself is faulted, restart the broker
4. After a broker restart, restart the services that are still not reconnecting, in dependency order: Keystone, then the API services, then agents

Step 2 resolves the majority. Skipping it and restarting everything is what turns a ten-minute incident into an hour.

The diagnostic path

1. Are workloads affected, or only the control plane? Running instances unaffected with everything reporting down points at the bus.
2. rabbitmqctl cluster_status — partitions and node health.
3. Any alarms? rabbitmqctl status shows memory and disk alarms; an alarm explains hanging publishers.
4. Queue depths sorted descending. The growing queue names the faulty service.
5. Consumers on that queue. Zero means the service is gone; non-zero means it is stuck.
6. Only then the service’s own log, now with a specific service to look at.

Exercise

Find the queues belonging to one service — the compute agent on a specific host is a good choice — and record their depth and consumer count while everything is healthy. This baseline is the thing most operators lack when an incident starts.

Stop that service and watch. Note which queue begins growing, how quickly, when the service is reported down, and what the dependent services log while it is away. Then start it again and watch the queue drain.

The value of the exercise is the timing: knowing how long the cloud takes to notice a failure, and how fast a backlog builds, converts queue depth from a number into a clock.