OpenStack Production Engineer

OpenStack is documented as a set of projects. It is operated as a single system, and almost nothing explains the difference.

This path is built for the operator’s view: not “what does Nova do”, but “when an instance fails to build, what is the fastest sequence of checks that finds out why”. Every module teaches the mental model first, then the commands, then the failure modes, then makes you use them.

How this path is sequenced

The order is deliberate. Architecture comes first because every later diagnosis depends on knowing which service talks to which and through what. Nova comes before Neutron and Cinder because the scheduling model it introduces — capability reporting, filters, placement — recurs in both. Deployment comes after the services, because a configuration change you do not understand is not made safer by knowing which file to put it in.

What this path assumes

You are comfortable on a Linux command line, you understand roughly what a hypervisor and a virtual network are, and you are prepared to read logs rather than search for error strings.

You do not need prior OpenStack experience. If you have some, the first module will still be worth reading — the service-state material in particular is the thing most experienced operators learn by getting caught out.

Why OpenStack is hard to learn from documentation

The upstream documentation is genuinely good, and it is organised by project. Nova’s documentation explains Nova. Neutron’s explains Neutron. Neither explains what happens to a Nova instance build when a Neutron agent stops reporting, because that question belongs to neither project.

Almost every production OpenStack problem lives in that gap. The service showing the error is usually a victim of something two dependencies away, and the error message names the symptom rather than the cause. NoValidHost is the canonical example: it is returned for capacity exhaustion, for a dead service, for stale placement allocations, for a flavor requesting something no host provides, and for an anti-affinity group larger than the cluster. Five different incidents, one message.

Learning OpenStack from project documentation therefore produces a specific kind of engineer: one who knows what each service does and cannot diagnose a cloud. This path is built to produce the other kind.

The model this path builds

By the end of the path you should be able to answer, from memory:

• Which service owns a given piece of state, and therefore where to look when it is wrong
• What each service assumes about the ones beneath it, and what happens when that assumption breaks
• How liveness is actually determined, and the four distinct causes of a service being reported down
• Where a scheduling decision is made, and how to find which stage rejected a request
• Where configuration comes from, and how to change it so the change survives

That last point sounds like an operations detail and is not. A configuration change you do not understand the provenance of is a change that will silently revert, and the incident it was preventing will come back weeks later with no obvious connection to anything.

How to work through it

Read the model before the commands. Each lesson opens with how the system works and only then gives you commands. It is tempting to skip to the commands. The commands are worth very little without the model — they will tell you a service is down and you will not know what that implies.

Do the exercises on a real cloud. Every lesson ends with an exercise designed to be run against a cloud you have access to, including a small single-node deployment. The exercises are roughly half the time in the estimate for this path, and they are where the understanding forms. Reading that a service can be enabled and down is not the same as seeing it, and the second one is what you will recognise at 3am.

Predict output before running a command. This is the single habit that separates engineers who diagnose quickly from those who do not. If the output surprises you, your model of the system is wrong — and finding that out during a lesson is considerably cheaper than finding it out during an incident.

Break things deliberately. Two of the lessons ask you to create a failure on purpose and then find it in the logs. This is the point of having a lab environment. The first time you see a filter chain return zero hosts should be a time you caused it.

What you need

A cloud you can break. An all-in-one deployment on a single machine with reasonable memory is sufficient for everything in the available modules — you do not need a multi-node cluster to understand service state, scheduling, or configuration provenance.

If you do not have one, the lessons are still followable, but the exercises are where the understanding actually forms, and reading them is not a substitute.

How the path is shaped

The first five modules go service by service, and each pairs a model with the troubleshooting that depends on it: you cannot diagnose a Neutron failure without the port and binding model, and the model is worth little until you have used it on a broken instance.

Module six is the one most curricula omit. RabbitMQ, MariaDB, and HAProxy sit on the path of every request in the cloud and belong to no OpenStack project, so no project documents them — which is exactly why their failures get attributed to whichever service happened to report the error.

Modules seven and eight assume all of it. Deployment and upgrades come after the services, because a configuration change you do not understand is not made safer by knowing which file to put it in. Production operations comes last because health checks, capacity, and incident method are all built out of the dependency structure the earlier modules teach.

The OpenStack production operations guide covers the same cross-service material at reference depth, and the Nova NoValidHost lab puts the scheduling module into practice against a scenario where the obvious answer is wrong.

What this path does not cover

• Installing OpenStack from source. We use Kolla-Ansible, because that is how production OpenStack is actually deployed.
• Every project. Keystone, Nova, Neutron, Cinder, Glance, and Placement are the ones you operate daily. The rest are covered where they intersect.
• Certification preparation. This path is aimed at the job, not at an exam.