OpenStack Production Engineer

OpenStack is documented as a set of projects. It is operated as a single system, and almost nothing explains the difference.

This path is built for the operator’s view: not “what does Nova do”, but “when an instance fails to build, what is the fastest sequence of checks that finds out why”. Every module teaches the mental model first, then the commands, then the failure modes, then makes you use them.

How this path is sequenced

The order is deliberate. Architecture comes first because every later diagnosis depends on knowing which service talks to which and through what. Nova comes before Neutron and Cinder because the scheduling model it introduces — capability reporting, filters, placement — recurs in both. Deployment comes after the services, because a configuration change you do not understand is not made safer by knowing which file to put it in.

What this path assumes

You are comfortable on a Linux command line, you understand roughly what a hypervisor and a virtual network are, and you are prepared to read logs rather than search for error strings.

You do not need prior OpenStack experience. If you have some, the first module will still be worth reading — the service-state material in particular is the thing most experienced operators learn by getting caught out.

Why OpenStack is hard to learn from documentation

The upstream documentation is genuinely good, and it is organised by project. Nova’s documentation explains Nova. Neutron’s explains Neutron. Neither explains what happens to a Nova instance build when a Neutron agent stops reporting, because that question belongs to neither project.

Almost every production OpenStack problem lives in that gap. The service showing the error is usually a victim of something two dependencies away, and the error message names the symptom rather than the cause. NoValidHost is the canonical example: it is returned for capacity exhaustion, for a dead service, for stale placement allocations, for a flavor requesting something no host provides, and for an anti-affinity group larger than the cluster. Five different incidents, one message.

Learning OpenStack from project documentation therefore produces a specific kind of engineer: one who knows what each service does and cannot diagnose a cloud. This path is built to produce the other kind.

The model this path builds

By the end of the available modules you should be able to answer, from memory:

• Which service owns a given piece of state, and therefore where to look when it is wrong
• What each service assumes about the ones beneath it, and what happens when that assumption breaks
• How liveness is actually determined, and the four distinct causes of a service being reported down
• Where a scheduling decision is made, and how to find which stage rejected a request
• Where configuration comes from, and how to change it so the change survives

That last point sounds like an operations detail and is not. A configuration change you do not understand the provenance of is a change that will silently revert, and the incident it was preventing will come back weeks later with no obvious connection to anything.

How to work through it

Read the model before the commands. Each lesson opens with how the system works and only then gives you commands. It is tempting to skip to the commands. The commands are worth very little without the model — they will tell you a service is down and you will not know what that implies.

Do the exercises on a real cloud. Every lesson ends with an exercise designed to be run against a cloud you have access to, including a small single-node deployment. Reading that a service can be enabled and down is not the same as seeing it, and the second one is what you will recognise at 3am.

Predict output before running a command. This is the single habit that separates engineers who diagnose quickly from those who do not. If the output surprises you, your model of the system is wrong — and finding that out during a lesson is considerably cheaper than finding it out during an incident.

Break things deliberately. Two of the lessons ask you to create a failure on purpose and then find it in the logs. This is the point of having a lab environment. The first time you see a filter chain return zero hosts should be a time you caused it.

What you need

A cloud you can break. An all-in-one deployment on a single machine with reasonable memory is sufficient for everything in the available modules — you do not need a multi-node cluster to understand service state, scheduling, or configuration provenance.

If you do not have one, the lessons are still followable, but the exercises are where the understanding actually forms, and reading them is not a substitute.

Where this path leads

The modules that are available cover the foundation: architecture, service state, scheduling, and deployment. Those four are what most OpenStack incidents come down to, which is why they are written first rather than in project order.

The remaining modules go into per-service depth — Keystone, Neutron, Cinder — and then into the operations material that assumes all of it: health check design, capacity management, and upgrade sequencing with rollback positions. They will be published when they are finished.

In the meantime, the OpenStack production operations guide covers the cross-service diagnostic material at reference depth, and the Nova NoValidHost lab puts the scheduling module into practice against a scenario where the obvious answer is wrong.

What this path does not cover

• Installing OpenStack from source. We use Kolla-Ansible, because that is how production OpenStack is actually deployed.
• Every project. Keystone, Nova, Neutron, Cinder, Glance, and Placement are the ones you operate daily. The rest are covered where they intersect.
• Certification preparation. This path is aimed at the job, not at an exam.