Health Checks — Designing Monitoring That Catches Quiet Failures

Three kinds of check, detecting three different things

Most OpenStack monitoring consists entirely of the first two, and the failures that cause incidents are visible only to the third.

Check	Asks	Misses
Liveness	Is the process running?	Everything about whether it works
Readiness	Does the API respond?	Everything behind the API
Functional	Can a user actually do the thing?	Very little

Every earlier lesson in this path contains an example of a failure that passes the first two:

• cinder-volume running with a driver that cannot reach the backend — process up, API up, volumes never create
• An L2 agent stopped on one compute host — every controller check green, instances there get no network
• A backend kept in HAProxy rotation by a TCP health check — API responds, a third of requests fail
• Galera in multi-writer — everything healthy, intermittent errors everywhere

None of these is detectable by asking whether a process is running.

What a functional check looks like

A functional check performs a real operation and cleans up after itself.

# 1. createopenstack server create –flavor m1.tiny –image cirros –network monitoring-net hc-1# 2. wait for ACTIVE, with a timeout that means something# 3. verify it got an address# 4. delete it# 5. verify the delete completed

Five steps, and step 5 is the one usually omitted. A cloud that creates instances and cannot delete them is broken in a way that fills up quietly and is invisible to a check that stops at step 4.

What to check functionally

One check per user-visible capability, not one per service:

Capability	The check
Compute	Boot an instance, confirm ACTIVE, delete it
Networking	Confirm the instance received an address and can route out
Storage	Create a volume, attach it, write, detach, delete
Identity	Issue a token and read the catalog
Images	Upload a small image, confirm it becomes active, delete it

Five checks cover what users actually do. They also implicitly test everything underneath — the message bus, the database, the load balancer — because a real operation cannot succeed without all of it working.

That is the argument for functional checks in one sentence: they test the dependencies you forgot to monitor.

Per-host checks, because fleet-level checks hide single-host failures

A functional check that boots an instance somewhere passes while one compute host is completely broken — the scheduler simply avoids it, or does not, and the check happened to land elsewhere.

Two additions close that gap:

Scheduled per-host checks. Periodically boot an instance on each host by name, in rotation. On a large fleet this need not be frequent; even daily catches a host that has been silently unusable.

Alert on the count of healthy hosts, not on individual host state. A fleet of 50 with 3 down is not an emergency; the same fleet with 30 down is. Alerting per host produces noise that gets muted, and muted alerts are worse than absent ones.

The quiet failures worth explicit checks

Some conditions never announce themselves and are worth checking directly:

Capacity that will run out. Not “is it full” but “when will it be” — projected from the trend. Full clouds are avoidable and their incidents are self-inflicted.

Queue depth on the message bus. A growing queue is a failing service, before that service reports anything.

Certificate expiry. Predictable, entirely preventable, and a recurring cause of total outages.

Placement allocations against reality. Drift here causes scheduling failures that look like capacity problems and are not.

Galera cluster size and node state. A cluster running degraded works fine until it does not.

Backup restorability. Not whether the backup ran — whether it restores. An untested backup is an assumption.

Thresholds that reflect impact

Alert on what users experience, not on component state.

Poor alert	Better alert
nova-compute down on compute-07	Healthy compute capacity below 80% of normal
Disk 85% full	Disk projected full within 7 days
API latency above 2s	Instance creation P95 above its normal range
Queue depth above 1000	Queue depth growing for 15 minutes

The right-hand column has a property the left lacks: each one, if it fires at 3am, is worth waking someone for. That is the actual test of an alerting threshold, and applying it honestly usually removes most of an existing alert set.

The diagnostic value of good checks

Well-designed checks do not only detect failure — they localise it. When the compute functional check fails and the networking one passes, the search space has already halved before anyone opens a log.

That is worth designing for deliberately: checks that fail in patterns tell you where to look, and checks that all fail together tell you to look at what they share — the bus, the database, or identity.

Exercise

Write one functional check that creates a resource, verifies it, and removes it, with a timeout and a duration measurement. Run it on a schedule for a day and record the normal duration range.

Then break something it depends on — stop an agent, fill a backend — and confirm the check fails. A check that has never been observed failing has not been tested, and a surprising number of monitoring systems contain checks that cannot fail at all.