MariaDB and Galera — The State Underneath Every Service

The database holds everything the cloud believes

Instance records, network definitions, volume state, tokens, quotas, and placement allocations all live in MariaDB. Running workloads do not depend on it — a hypervisor keeps running regardless — but nothing about the cloud can be changed or accurately reported without it.

That split is the key to recognising a database incident: instances keep serving traffic while every API is slow or failing, and nothing can be created.

What Galera actually guarantees

Galera is synchronous multi-master replication. A transaction is certified across all nodes before it commits, so every node has the same data.

What it does not provide is conflict-free concurrent writing. If two nodes commit conflicting transactions at the same time, one is certified and the other is rolled back — and the rollback surfaces at the application as a deadlock error.

Cluster health in three values

MariaDB> SHOW STATUS LIKE ‘wsrep_cluster_size’;| wsrep_cluster_size | 3 |MariaDB> SHOW STATUS LIKE ‘wsrep_local_state_comment’;| wsrep_local_state_comment | Synced |MariaDB> SHOW STATUS LIKE ‘wsrep_ready’;| wsrep_ready | ON |

• wsrep_cluster_size must equal the number of nodes you have. Anything less means a node has left, and the same value on every node still needs checking — two nodes each reporting size 2 in a three-node cluster is a partition, not a healthy cluster.
• wsrep_local_state_comment should read Synced. Donor/Desynced means this node is feeding a state transfer to another and will be slow. Joining means it is receiving one.
• wsrep_ready must be ON. When OFF, the node rejects queries entirely, and clients see connection errors rather than slow queries.

Losing quorum

Galera requires a majority to accept writes. A three-node cluster that loses two nodes has one survivor, which correctly refuses to write — it cannot know whether the others are dead or partitioned away and still serving.

The result is a cloud that reads fine and cannot change anything: instance listings work, instance creation fails.

Failures as the services report them

The database rarely announces itself. It appears as errors in whichever service noticed:

Service symptom	Database cause
Intermittent 500s across several services at once	Certification conflicts from multi-writer
“Deadlock found when trying to get lock” in logs	The same, surfacing directly
Reads work, writes fail cloud-wide	Lost quorum
One service consistently slow, others fine	A missing index or a query pattern specific to it
Everything slow, database CPU low	Connection pool exhaustion, not the database itself

That last row misleads reliably. When every service is slow and the database looks idle, the constraint is usually the number of available connections rather than the database’s capacity to answer.

SHOW STATUS LIKE 'Threads_connected';
SHOW VARIABLES LIKE 'max_connections';

OpenStack opens a lot of connections — each service, each worker, each host. max_connections sized for a small deployment becomes the ceiling for the whole cloud as it grows, and the failure appears everywhere except in the database’s own metrics.

Growth that becomes an incident

Two tables grow without bound unless something removes rows, and both are database-level problems that present as cloud-level ones:

Deleted rows. OpenStack soft-deletes. Instance records, network records, and volume records accumulate for the life of the deployment, slowing every query that scans them. The archive and purge tooling exists for this and is frequently never run.

Token tables, on deployments predating Fernet. Modern deployments do not have this problem; long-lived ones that were upgraded sometimes still carry the table.

The symptom of both is a cloud that has become gradually slower over months with no change to explain it.

The diagnostic path

1. Are workloads affected or only the control plane? Instances fine, APIs failing points here.
2. wsrep_cluster_size on every node — and compare them. Disagreement is a partition.
3. wsrep_local_state_comment and wsrep_ready on the node being written to.
4. Reads working, writes failing? Quorum.
5. Deadlock errors in service logs? Check whether the load balancer sends writes to one node.
6. Everything slow, database idle? Connection count against max_connections.

Exercise

Check wsrep_cluster_size and wsrep_local_state_comment on every node, not just one — the disagreement between nodes is the thing worth seeing.

Then examine your load balancer configuration for the database and establish, definitively, whether writes go to one node or several. Confirm it from the database side by comparing Threads_connected across nodes: a single-writer configuration shows connections concentrated on one node, and anything close to even distribution means you are running multi-writer, whatever the configuration was intended to do.