Kolla-Ansible Production Architecture — Deploying OpenStack in Containers

Kolla-Ansible runs every OpenStack service as a container and generates every configuration file from templates. Both facts change how you operate the cloud, and the second one catches everyone at least once.

The shape of a deployment

Deployment host
  └─ kolla-ansible ──▶ generates config ──▶ deploys containers
                              │
      ┌───────────────────────┴───────────────────────┐
      │                                               │
  Control nodes                                  Compute nodes
   ├─ keystone, glance, placement                 ├─ nova_compute
   ├─ nova_api, nova_scheduler, nova_conductor    ├─ neutron agents
   ├─ neutron_server                              └─ (libvirt on the host)
   ├─ cinder_api, cinder_scheduler
   ├─ horizon
   ├─ mariadb (Galera), rabbitmq, memcached
   └─ haproxy + keepalived  ◀── the VIP lives here

Each service is a container named after itself: nova_api, neutron_server, cinder_scheduler. That naming makes docker logs <service> the fastest diagnostic path in the entire system.

Configuration is generated, so do not edit it

The rule that catches everybody:

Configuration comes from three places, in increasing specificity:

1. /etc/kolla/globals.yml — deployment-wide settings: enabled services, network interfaces, the VIP, TLS, backends
2. /etc/kolla/passwords.yml — every generated credential. Back this up; without it you cannot redeploy the cloud
3. /etc/kolla/config/ — per-service overrides merged into the generated files

Overrides follow a path convention:

/etc/kolla/config/nova.conf                      # all nova services
/etc/kolla/config/nova/nova-scheduler.conf       # just the scheduler
/etc/kolla/config/nova/compute-01/nova.conf      # just that host

# /etc/kolla/config/nova/nova-scheduler.conf
[filter_scheduler]
enabled_filters = AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter

Kolla merges the override into its template rather than replacing the file, so you only specify what differs.

Applying changes

# Regenerate configuration and restart only what changed
kolla-ansible reconfigure -i /etc/kolla/inventory

# Scope it -- reconfiguring the whole cloud to change one service is
# unnecessary risk during business hours
kolla-ansible reconfigure -i /etc/kolla/inventory --tags nova
kolla-ansible reconfigure -i /etc/kolla/inventory --limit compute-02

--tags and --limit together are what makes this safe to run during the day. A full reconfigure touches every service on every node, and while it is designed to be safe, a change to Nova has no business restarting Neutron.

Operating containerised services

$ docker ps –format ‘table {{.Names}}\t{{.Status}}’ | grep -v Up# Anything not “Up” — restarting containers are the interesting ones$ docker logs –tail 100 –timestamps nova_scheduler# Kolla services log to the container and to /var/log/kolla on the host$ docker exec -it nova_api bash# For running nova-manage and other admin tooling in the right environment$ docker restart nova_scheduler# Safe for stateless services. Not for mariadb or rabbitmq — those are# clustered and need the cluster-aware procedure.

A container in a restart loop is the clearest signal available. docker ps showing Restarting (1) 5 seconds ago tells you where to look before you have read a single log line.

HAProxy and the VIP

Every API endpoint is fronted by HAProxy on a virtual IP managed by keepalived. Two consequences worth knowing:

The VIP moves. If the node holding it fails, keepalived moves it. During that window, connections are reset rather than gracefully drained.

HAProxy has its own health checks. A backend can be marked down by HAProxy while the service itself is running, usually because the health check is stricter than “the process exists”.

# HAProxy's own view of backend state
echo "show stat" | docker exec -i haproxy socat stdio /var/lib/kolla/haproxy/haproxy.sock \
  | awk -F, '$18 != "UP" && $2 != "BACKEND" {print $1, $2, $18}'

# Where is the VIP right now?
ip addr show | grep -A2 "<vip-address>"

A service that responds when queried directly on its port but fails through the VIP is an HAProxy problem, not a service problem. Testing both is how you tell them apart in one step.

The stateful services

MariaDB (Galera) and RabbitMQ are clustered and do not tolerate the treatment the stateless services do.

# Galera cluster size should equal your control node count
docker exec mariadb mysql -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
docker exec mariadb mysql -e "SHOW STATUS LIKE 'wsrep_local_state_comment';"

# RabbitMQ partitions
docker exec rabbitmq rabbitmqctl cluster_status

Adding a compute node

# 1. Add the host to the [compute] group in the inventory
# 2. Bootstrap the host (packages, docker, users)
kolla-ansible bootstrap-servers -i /etc/kolla/inventory --limit compute-04

# 3. Pull images first, so the deploy window is short and predictable
kolla-ansible pull -i /etc/kolla/inventory --limit compute-04

# 4. Deploy just that host
kolla-ansible deploy -i /etc/kolla/inventory --limit compute-04

# 5. Verify it joined
openstack compute service list --host compute-04
openstack network agent list --host compute-04
openstack hypervisor list | grep compute-04

Step 5 is not optional. A compute node that deployed successfully but did not register with placement is a node that will never receive an instance, and nothing will tell you.

Upgrades

kolla-ansible pull -i /etc/kolla/inventory              # images first, always
kolla-ansible prechecks -i /etc/kolla/inventory         # catches config problems early
kolla-ansible upgrade -i /etc/kolla/inventory --tags keystone

Order matters and follows the dependency graph: Keystone first, then the services that depend on it. prechecks is genuinely useful and skipping it is how upgrades fail in the middle.

The critical point: database migrations are not reversible. Once a service’s schema has been upgraded, rolling that service’s container image back will not work. The rollback plan for an OpenStack upgrade is a database restore, and it must be tested before the window, not designed during it.

Backup list

The things you cannot rebuild:

• /etc/kolla/passwords.yml — without it, a redeployed cloud cannot talk to its own database
• /etc/kolla/globals.yml and /etc/kolla/config/ — the deployment definition
• The inventory
• MariaDB dumps — every service’s state lives here
• Fernet keys — losing them invalidates every issued token at once

Keep the first four in version control. Keep the database dumps somewhere you have restored from at least once.