Upgrade Strategy — Sequencing, Validation, and Rollback Positions

An upgrade is a sequence of stages, each with a way out

The failure mode that produces long outages is not a broken upgrade. It is a broken upgrade with no defined position to return to, discovered halfway through.

Every stage needs three things decided before it runs:

1. What changes
2. What you check afterwards, specifically
3. What you do if that check fails

The third is the one that gets skipped, and it is the only one that matters at 2am.

OpenStack supports mixed versions, deliberately

OpenStack services are designed to run one release apart during an upgrade. That is what makes a staged upgrade possible: the control plane can be newer than the compute hosts while both keep working.

This is a bounded guarantee, not a general one:

• One release, not several. Skipping releases is not supported by this mechanism.
• Control plane ahead of compute, not behind. Upgrading a compute host past the controllers is outside the tested combinations.
• For a limited period. Running mixed for months is not a supported operating mode.

The stages, and where rollback changes character

  1. Pre-flight checks           reversible: nothing has changed
  2. Pull images                 reversible: nothing is running them
  3. Database backup             the rollback position for stage 4
  4. Database migrations         ◄── the character of rollback changes here
  5. Control plane services      reversible by re-deploying old images
  6. Compute hosts, in batches   reversible per host
  7. Post-upgrade validation     the decision point for the whole change

Pre-flight checks that prevent the common failures

Most upgrade failures are predictable from the state of the cloud beforehand.

Every service is healthy now. An upgrade will not fix an already-down service, and starting with known failures makes it impossible to tell what the upgrade broke.

openstack compute service list --long
openstack network agent list
openstack volume service list

No instances in a transitional state. An instance in BUILD or deleting when its compute service is upgraded may never leave that state, because the process tracking the operation is replaced mid-flight.

Disk space for images and the database backup. Container images are large and the old ones are retained. Running out mid-upgrade fails in the least recoverable place.

The cluster is fully healthy. A Galera cluster running degraded is not a cluster you want to run schema migrations against.

You have read this release’s notes. Configuration deprecations and removed options are the most common cause of services that fail to start after an otherwise clean upgrade.

Validation that would actually catch a failure

The temptation is to check that services are up. Services being up is necessary and proves very little — every service can be up while nothing works, which is the theme of several earlier lessons.

Validation has to exercise the path end to end:

Stage	Weak check	Check that would catch a failure
Control plane	API returns 200	Create and delete an instance
Compute batch	Service reports up	Boot an instance on that host and reach it
Networking	Agent alive	New instance gets an address and can route out
Storage	Service up	Create a volume, attach it, write to it

The compute check matters most and is skipped most: a host whose services report up but cannot successfully boot an instance will accept scheduling and fail every request sent to it. Testing one instance per upgraded host, before moving to the next batch, is what makes a batched upgrade safe.

Batching compute hosts

The unit of risk is a batch. Small first, then larger once the shape of the upgrade is known.

Batch	Scope	Question it answers
1	One host, no workloads	Does the procedure work at all?
2	One host with workloads	Does migration behave?
3	10% of the fleet	Does it scale?
n	25% at a time	Routine

Per host: disable the service so nothing new is scheduled to it, migrate the workloads off, upgrade, validate with a real instance, then re-enable.

The decision point

Stage 7 exists to answer one question: is this upgrade staying?

That decision needs a deadline agreed in advance. An upgrade that is neither completed nor rolled back is the worst available state — a cloud running mixed versions indefinitely, outside the supported window, with everyone assuming someone else is finishing it.

Decide beforehand: how long you observe, what would constitute a failure, and who decides. Then hold to it.

Exercise

Write the upgrade plan for one release step in your own environment. For each of the seven stages, write down what changes, the specific command or observation you would use to validate it, and exactly what you would do if that validation failed.

Two of those answers will be uncomfortable to write: the rollback for stage 5, and the rollback for stage 4. If you cannot write the stage 4 answer concretely — including where the backup is and when it was last restored successfully — that is the finding, and it is worth more than the plan.