HAProxy — Where API Requests Actually Arrive

Every API request passes through something the projects do not document

The endpoints in the Keystone catalog point at a virtual IP, not at a service. That address belongs to a load balancer, which distributes across the controllers.

  client
    │  https://cloud.example/compute
    ▼
  VIP (keepalived) ──► HAProxy ──┬──► nova-api on controller-01
                                 ├──► nova-api on controller-02
                                 └──► nova-api on controller-03

Two components sit here, and they fail differently:

• keepalived owns the virtual IP and moves it between controllers. Its failures are total — the address is unreachable, or worse, claimed by two hosts at once.
• HAProxy distributes requests and decides which backends are eligible. Its failures are partial, which makes them much harder to recognise.

Partial failure is the theme of this lesson. A request that fails one time in three is nearly always a load balancer question, because nothing else in the stack naturally produces that pattern.

Health checks that keep broken backends in rotation

HAProxy removes a backend when its health check fails. What the check actually tests decides whether that ever happens.

This is the most common load-balancer-caused incident in OpenStack, and it is misdiagnosed persistently because the failures are attributed to the service that returned them.

Reading backend state

$ echo “show stat” | socat stdio /var/lib/kolla/haproxy/haproxy.sock > stat.csv$ grep ‘^nova_api,’ stat.csv | cut -d, -f2,18 # backend name and statuscontroller-01 UP 0controller-02 DOWN 0controller-03 UP 0

Two questions are worth asking of this output every time:

How many backends are up? A pool running on one of three is working and has no redundancy left. Nothing alerts on this by default, and the cloud behaves normally until that last backend fails.

Are any backends flapping? A backend alternating between UP and DOWN produces intermittent failures that correlate with nothing an application-level investigation would look at.

Timeouts, and the operations they cut short

HAProxy applies its own timeouts, and they are frequently shorter than the OpenStack operations passing through them.

The client sees the connection close. The service, unaware, carries on and completes the work. The result is an operation that reports failure and then succeeds — a volume that appears after the error, an image that uploads despite timing out, an instance that exists after the client gave up.

Operation	Why it exceeds a default timeout
Large image upload to Glance	Bounded by network throughput and image size
Volume creation from an image	The backend copies the whole image
Instance snapshot	Copies the disk
Live migration status polling	Long-running by nature

keepalived and the split-brain case

keepalived assigns the virtual IP to one controller and moves it if that one fails. When the controllers cannot see each other’s advertisements — a network problem, or blocked multicast — more than one may claim the address.

The result is traffic arriving at different hosts unpredictably, with connections resetting as ARP entries change. It presents as severe, erratic network instability affecting the API and nothing else.

# Run on every controller. Exactly one should hold the VIP.
ip addr show | grep <vip-address>

Two hosts answering is the whole diagnosis.

Internal versus public endpoints

The load balancer usually fronts both interfaces, and they are frequently configured differently — different ports, different TLS, sometimes different backend pools.

This is why a command can work from a controller and fail from a workstation, or the reverse. It is not a permissions problem and not a service problem; the two paths are genuinely different configurations, and only one of them is broken.

When a failure is reported, establish which interface the reporter was using before anything else. OS_INTERFACE decides it, and its default is not the same in every client.

The diagnostic path

1. Is the failure intermittent? If so, start here rather than at the service.
2. How many backends are up? One broken backend in rotation explains a consistent fraction of failures.
3. What does the health check test? A TCP check on a service that can fail while listening is the common cause.
4. Any backend flapping? Check the HAProxy log for state transitions.
5. Did the operation time out and then succeed? Load balancer timeout, not a service fault.
6. Does exactly one host hold the VIP? More than one is split-brain.
7. Which interface was the client using? Public and internal can fail independently.

Exercise

Read the backend state for one API pool and record how many backends are up and what the health check tests. If it is a TCP check, note what it would fail to catch.

Then stop one API service and time three things: how long until HAProxy marks it down, what clients see during that window, and how many requests fail before it is removed. That interval is the outage a backend failure actually causes, and it is usually longer than people assume — which is the argument for a health check that tests something real.