Service State and Liveness — Reading What the Cloud Is Telling You

Two columns, two different questions

Every OpenStack service listing reports two things that look like one:

• Status — what an operator wants. enabled or disabled. Someone set this deliberately.
• State — what is actually happening. up or down. Derived, not set.

+––––––––+————+———+—––+| Binary | Host | Status | State |+––––––––+————+———+—––+| nova-compute | compute-01 | enabled | up | ← normal| nova-compute | compute-02 | enabled | down | ← the problem| nova-compute | compute-03 | disabled| down | ← deliberate| nova-compute | compute-04 | disabled| up | ← wasting capacity+––––––––+————+———+—––+

Four combinations, four different meanings:

Status	State	Meaning	Action
enabled	up	Working and in service	None
enabled	down	The cloud thinks it is available. It is not.	Investigate now
disabled	down	Taken out deliberately and stopped	Confirm it is on someone’s list
disabled	up	Running but excluded from scheduling	Usually a forgotten maintenance leftover

Why enabled-and-down is the dangerous one

The scheduler treats an enabled service as part of the cloud. It counts toward capacity arithmetic and it is a scheduling candidate.

When that service is not actually answering, the results are indirect and confusing:

• Capacity reporting overstates what you have
• Scheduling attempts route work to a host that will not respond
• Failures appear as timeouts and NoValidHost rather than as “that host is down”

Nothing in the error message points at the host. You get a symptom three layers away from the cause.

The disabled-and-down case is the opposite: the cloud already knows not to use it, so nothing misbehaves. Same process state, completely different operational meaning.

How liveness is actually determined

State is not a health check. Nothing probes the service.

Each service periodically publishes a report over the message bus. The control plane records the timestamp of the last one. If the most recent report is older than a threshold, the service is marked down.

Which means down can be caused by:

1. The process is dead. The obvious case.
2. The process is alive but not reaching the message bus. Network, credentials, or a broker problem.
3. The message bus is not delivering. A partition, or a queue with no consumer.
4. Clock skew. The report arrives, but its timestamp is far enough off that it is treated as stale.

This is why docker ps showing the container up does not contradict State: down. They are measuring different things — one is process existence, the other is successful communication.

Neutron is inverted

Neutron reports the same information with different column names, in the opposite arrangement.

+––––––––––+————+—––+—––+| Agent Type | Host | Alive | State |+––––––––––+————+—––+—––+| Open vSwitch agent | compute-01 | :-) | UP || Open vSwitch agent | compute-02 | XXX | UP || DHCP agent | ctl1 | :-) | UP |+––––––––––+————+—––+—––+

• Alive is liveness: :-) means reporting, XXX means not.
• State is administrative: UP or DOWN.

So in Neutron, State means what Status means in Nova. Reading the State column the way you read it in a compute listing gives you exactly the wrong answer.

compute-02 above is the dangerous row: administratively UP, not alive. Instances scheduled to that host will build and have no network.

Correlation across services

A single service down on a host is a service problem. Several services down on the same host is a host problem, and investigating them individually wastes time.

nova-compute compute-07 enabled downOpen vSwitch agent compute-07 XXX UPcinder-volume compute-07 enabled down

Three unrelated services on one host all stopped reporting. They do not share code, they share the host, its network path, and its connection to the message bus.

Investigate the host. Do not investigate Nova.

The same logic applies to zones: if every failing service sits in one availability zone, look for what that zone shares — a rack, a switch, a power feed.

Building a health check that is worth running

The useful health check reports only things that need attention. A check that lists every service is a report nobody reads.

#!/usr/bin/env bash
# Report only services that need a human.
set -euo pipefail

echo "=== Enabled but down ==="
openstack compute service list -f value -c Binary -c Host -c Status -c State \
  | awk '$3 == "enabled" && $4 == "down" { print "  compute: " $1 " on " $2 }'

openstack volume service list -f value -c Binary -c Host -c Status -c State \
  | awk '$3 == "enabled" && $4 == "down" { print "  volume:  " $1 " on " $2 }'

openstack network agent list -f value -c 'Agent Type' -c Host -c Alive -c State \
  | awk '$NF == "UP" && $(NF-1) != ":-)" { print "  network: " $0 }'

echo "=== Running but disabled (lost capacity) ==="
openstack compute service list -f value -c Binary -c Host -c Status -c State \
  | awk '$3 == "disabled" && $4 == "up" { print "  " $1 " on " $2 }'

echo "=== Instances in error ==="
openstack server list --all-projects --status ERROR -f value -c ID -c Name | sed 's/^/  /'

No output means healthy. That is the property that makes a check worth running daily — you notice output because output is unusual.

Exercise

Extend the script above so it also reports any host with more than one failing service, since that indicates a host-level problem rather than a service one.

You will need to collect failures from all three listings into a single list of hosts, then count occurrences.

Then run it against a cloud you have access to and confirm it produces no output when everything is healthy. A check that always prints something will be ignored within a week.

Validation

You understand this lesson if you can answer, without looking back:

1. A service shows enabled / down. Name three causes other than the process being dead.
2. A Neutron agent shows Alive: XXX and State: UP. Is it working? Is it supposed to be?
3. docker ps shows nova_compute running, and the service list shows it down. Do these contradict each other?
4. Four services across two hosts are down. What do you investigate first?