Keystone — Tokens, the Catalog, and Why Everything Fails at Once

Keystone does two jobs, and operators conflate them

Almost everyone describes Keystone as “the authentication service”. That is half of it, and the half that causes fewer incidents.

• Identity. Who are you, and what are you allowed to do? This produces a token.
• Catalog. Where does each service live? This produces a list of URLs.

The second job is the one that surprises people. When a client runs openstack server list, it does not know where Nova is. It asks Keystone, gets back a catalog, finds the compute entry, and calls the URL it finds there. Every command you run is two requests, and the first one is always to Keystone.

That is why a Keystone problem does not look like a Keystone problem.

What a token actually is

A token is a bearer credential with an expiry, a scope, and a set of roles. It proves that Keystone believed you at a particular moment, for a particular project.

+————+———————————+| Field | Value |+————+———————————+| expires | 2026-08-29T14:22:31+0000 || id | gAAAAABo… || project_id | 8f3c2b1a9d4e4f6b8c7a2d1e0f9b3c5a|| user_id | 2b7e1a4c6d8f0a2c4e6b8d0f2a4c6e8b|+————+———————————+

Three properties matter operationally:

Tokens are scoped. A token is issued for a project. A token scoped to project A is useless for project B, even for the same user with the same roles. “Permission denied” on a resource you own is frequently a token scoped somewhere else.

Tokens expire. The default lifetime is an hour. Long-running automation that acquires a token once and reuses it will work for an hour and then fail, which produces the distinctive pattern of a job that always dies about the same distance in.

Tokens are validated, not just presented. When you hand a token to Nova, Nova asks Keystone whether it is still good. So a Keystone outage breaks requests that are using tokens issued before the outage started.

The catalog is a lookup table, and it lies willingly

The catalog is just data. Keystone stores whatever endpoint URLs someone registered and hands them out. Nothing validates that a registered endpoint points at a running service.

+———–+———–+—————————————+| Name | Type | Endpoints |+———–+———–+—————————————+| nova | compute | RegionOne || | | public: https://cloud.example/compute|| | | internal: http://10.0.0.5:8774 || neutron | network | RegionOne || | | public: https://cloud.example/network|+———–+———–+—————————————+

Each service has up to three interfaces — public, internal, and admin — and clients choose which to use. This is a rich source of confusion: a command that works from a controller and fails from a laptop is often selecting internal in one place and public in the other, and only one of those is routable from where you are standing.

A wrong catalog entry produces an error from the client library, not from Keystone, and the error names the service you were trying to reach. openstack server list returning a connection refused looks like Nova being down. It can equally be Nova being fine and the catalog pointing at the wrong port.

Recognising a Keystone failure by its shape

Keystone failures have a characteristic signature: everything breaks at once, and nothing explains why.

What you see	What it usually means
Every command fails with an auth error, all at once	Keystone is down, or unreachable from the client
Auth fails intermittently, roughly half the time	Fernet keys diverged across nodes, or one node behind the load balancer is unhealthy
One project works, another does not	Token scope, or role assignment on that project
Works from the controller, fails elsewhere	Catalog interface selection, or a network path to the endpoint
Worked an hour ago, fails now, no changes	Token expiry in long-running automation

The first row is the one that wastes time, because the symptom appears in Nova, Neutron, Cinder, and Glance simultaneously and the natural instinct is to investigate the service you were using when you noticed.

Authentication and authorisation fail differently

These are separate stages and they produce different errors. Telling them apart immediately removes half the search space.

Authentication failure — Keystone does not believe you. HTTP 401. The credentials are wrong, the token has expired, or the user is disabled.

Authorisation failure — Keystone believes you, and you are not allowed to do this. HTTP 403. The user is real, the token is valid, and the role assignment does not permit the action on this project.

$ openstack server listThe request you have made requires authentication. (HTTP 401) → who you are is the problem: credentials, expiry, disabled user$ openstack server list –all-projectsYou are not authorized to perform the requested action. (HTTP 403) → who you are is fine: you lack the role for this action

A 403 is never fixed by re-authenticating, and a surprising amount of time is lost trying. Check the role assignment instead:

openstack role assignment list --user <user> --project <project> --names

Why Keystone problems are hard to see coming

Keystone is stateless per request and its load is dominated by token validation — every API call from every service generates one. That means Keystone is usually the busiest service in the cloud by request count while being invisible in every capacity discussion, which is organised around compute and storage.

Three failure modes follow from that:

1. Keystone saturates before anything else under a burst of automation, and the resulting timeouts appear as failures in whichever service the automation was driving.
2. A slow Keystone slows everything, because every request in the cloud waits on it. Latency here is multiplied across the whole system rather than isolated.
3. Keystone’s database is on the critical path for every request. A database problem that would be a minor annoyance elsewhere is a cloud-wide outage here.

The diagnostic path

When multiple services fail together, work in this order:

1. Can you get a token? openstack token issue. If not, Keystone or the path to it.
2. Is the catalog sane? openstack catalog list. Compare the endpoint against where the service actually listens.
3. Which interface is your client using? OS_INTERFACE or --os-interface. A mismatch here explains works-here-fails-there.
4. Is it 401 or 403? Authentication or authorisation — different problems, different fixes.
5. Is the failure intermittent? Suspect divergent Fernet keys or one unhealthy node behind the load balancer, and test repeatedly rather than once.

Only after all five does it make sense to investigate the service that reported the error.

Exercise

Issue a token and read the catalog it returns. Identify which interface your client is configured to use, and confirm by hand — with curl against the endpoint from the catalog — that it is reachable from where you are running commands.

Then deliberately break it. Update one endpoint to a port nothing is listening on and run a command against that service. Note carefully which component reports the error and what it says: the point of the exercise is to see that a catalog problem is reported as a failure of the service, and to recognise that shape next time.