Production Operations — Running the Cloud Between Incidents

You cannot recognise abnormal without knowing normal

The single most useful artefact an operations team can have is a written baseline — and it is almost always missing, because it is only obviously valuable during an incident, which is exactly when there is no time to establish it.

Worth recording, and re-recording quarterly:

Measure	Why it matters during an incident
Instance creation time, typical and P95	Tells you whether slow is actually slow
Service and agent counts, per type	Tells you whether one being missing is new
Queue depths at rest	Distinguishes a backlog from normal
Capacity headroom per resource	Tells you whether full is plausible
API latency per service	Distinguishes degradation from perception

“Instance creation takes 40 seconds” is meaningless alone. Against a baseline of 18 it is a finding, and against a baseline of 45 it is noise. Most incident time is spent establishing facts that could have been written down in advance.

Change discipline is what makes faults attributable

Most OpenStack incidents follow a change. When several changes land together, attribution becomes guesswork and the rollback is correspondingly blunt.

Three rules do most of the work:

One change at a time, with a gap. Two changes deployed together mean a subsequent failure has two candidate causes and rolling back either is a coin flip.

Every change has a written rollback. Not “revert the commit” — the actual sequence, including what to do if the change has already had effects. A change whose rollback cannot be written down is a change that needs more thought before it is made, not after.

Configuration changes go through the deployment tool. A change made directly to a generated file works and then vanishes at the next reconfigure. The Kolla lesson covers the mechanism; the operational point is that hand edits create incidents weeks later with no apparent cause.

Host maintenance without disturbing workloads

Taking a compute host out of service has an order, and each step exists for a reason.

# 1. Stop new work arriving — reversible, do this first$ openstack compute service set –disable –disable-reason “ticket 4821” compute-03 nova-compute# 2. See what is actually on it$ openstack server list –host compute-03 –all-projects# 3. Move the workloads off$ openstack server migrate –live-migration <instance># 4. Confirm it is empty before touching anything$ openstack server list –host compute-03 –all-projects

Step 1 before step 3, always. Migrating first leaves a window in which the scheduler places new instances on the host you are about to work on.

The --disable-reason is not decoration. A disabled service with no reason is indistinguishable from one someone forgot to re-enable, and forgotten disabled hosts are a routine source of mysteriously reduced capacity.

Returning the host is the reverse, with a check that has to be real: enable the service, then boot an instance on that host specifically and confirm it works. A host returned to service without that check can accept scheduling and fail everything sent to it.

Operational debt predicts incidents

Some conditions are not failures and reliably become them. They are worth tracking as debt with owners, because none of them will ever be urgent until they are an outage.

• Disabled services with no reason recorded, or with a reason from months ago
• Instances in ERROR that nobody has triaged — each one is unexplained and some share a cause
• Placement allocations without instances — capacity that silently disappears
• Volumes stuck in transitional states
• Certificates approaching expiry
• Backups that have never been restored
• A deployment several releases behind, where each deferred upgrade makes the next larger

The last is the compounding one. Upgrades are sized by how many releases you are behind, and deferring makes the eventual upgrade both riskier and harder to roll back.

Routine that pays for itself

Weekly: service listings compared against the baseline; instances in ERROR triaged; queue depths checked; capacity headroom reviewed.

Monthly: placement allocations reconciled against reality; disabled services reviewed; database archived and purged; a backup actually restored somewhere.

Quarterly: baseline re-recorded; upgrade position reviewed; the alert set audited for anything that has fired and been ignored.

That last one matters more than it sounds. An alert that fires regularly and is routinely ignored is worse than no alert: it trains everyone that alerts are noise, and it is the one that will be ignored when it finally means something.

What separates teams that run OpenStack well

It is not diagnostic skill under pressure. It is that fewer things break, and the things that do break are recognised faster — both of which are products of routine rather than talent.

Concretely: a written baseline, one change at a time with a rollback, functional checks that have been observed failing, maintenance with a real validation step, and a debt list that someone owns.

None of it is difficult. All of it is the first thing dropped when the team is busy, which is precisely when the incidents it prevents become most expensive.

Exercise

Record your cloud’s baseline — creation time, service counts, queue depths, capacity headroom — and write it somewhere the on-call person can find at 3am. That artefact alone will shorten your next incident.

Then perform a full maintenance cycle on one host: disable with a reason, inventory, migrate, confirm empty, do the work, return, and validate by booting an instance on that host specifically.

Afterwards, compare against the baseline. If anything moved, you have found something the maintenance did that you did not intend — which is exactly what the baseline is for.