AI Agents in Production
What changes when an agent loop moves from a demo to production: hard limits, tool authorisation, failure handling, and knowing when an agent is the wrong shape entirely.
Index
48 resources across 10 pillars. This page is a navigation index for readers, which is why it is not itself in a search index — the pillar hubs it links to are the pages that deserve that.
4 resources
Agents, RAG, model APIs, MCP, and evaluation in production.
What changes when an agent loop moves from a demo to production: hard limits, tool authorisation, failure handling, and knowing when an agent is the wrong shape entirely.
How to structure an AI application so that a slow model, a failed tool call, or a bad retrieval degrades one part of the system instead of the whole request path.
A working guide to retrieval-augmented generation: how to chunk, how to evaluate retrieval separately from generation, and how to diagnose a bad answer in the right order.
A layer-by-layer map of the modern AI stack, what each layer is actually responsible for, and where production systems tend to break in practice.
3 resources
GPUs, interconnects, storage, inference and training platforms.
Collective communication, RDMA, and rail topology explained from the operator's side, including why adding nodes to a training job can make it slower.
What determines accelerator performance in production: memory capacity versus bandwidth, interconnect topology, and the checks that find a misplaced workload.
Dataset streaming, checkpoint bursts, and model loading place completely different demands on storage. Designing for one and getting the others wrong is the usual outcome.
6 resources
Kubernetes, Docker, Terraform, Linux, and observability.
Exec format errors, permission denied, missing commands, bad entrypoints, immediate exits, and CrashLoopBackOff — diagnosed as one family rather than a page per error string.
Building container images for production: reproducible builds, correct signal handling, non-root runtime, layer strategy, and keeping secrets out of image history.
A single diagnostic path for workloads that will not schedule, will not stay up, or keep getting evicted — driven by events and previous logs rather than guesswork.
The operational settings that decide whether a workload survives a node failure or a rollout: requests and limits, the three probes, disruption budgets, and scheduling.
Running Terraform against production infrastructure: reviewing the plan mechanically, containing blast radius, structuring state, and avoiding the destroys nobody noticed.
The state commands that are genuinely dangerous, what each one actually does, and how to run them with a way back.
2 resources
Private cloud operations, with depth in OpenStack.
How Kolla-Ansible actually structures an OpenStack deployment, where configuration comes from, and how to make changes without discovering them during an outage.
Operating OpenStack in production: service state versus status, the message bus, placement disagreements, and the quiet failures that keep dashboards green.
2 resources
Incident response, diagnosis, and the operations workbench.
A repeatable method for diagnosing production failures: form hypotheses the evidence supports, order checks by what they eliminate, and know when to stop and mitigate.
Building observability that shortens incidents rather than producing dashboards: what to instrument, how to alert on symptoms, and why cardinality decides your bill.
21 resources
Structured learning paths for production engineers.
A sequenced learning path for engineers who operate OpenStack in production — architecture, service-by-service depth, deployment, and troubleshooting under pressure.
How Cinder models volumes and drivers, what backends report about themselves, and why the API stays green when storage is failing.
Diagnosing Cinder scheduling failures and attachment failures, including the capacity report that is accurate and still wrong.
How the load balancer sits in front of every OpenStack API, the health checks that lie, and the failures that appear as service errors.
How authentication and service discovery actually work in OpenStack, and why a Keystone problem presents as every other service failing simultaneously.
How Kolla-Ansible deploys OpenStack as containers, what the inventory controls, and why the running configuration is not the one you edited.
Planning an OpenStack upgrade as a sequence of stages, each with a validation and a defined position to stop at.
How OpenStack uses its database, what Galera guarantees and does not, and the failure modes that present as application errors.
How Neutron models networks as ports and bindings, what each agent does, and why a port stuck in DOWN is the most informative failure in OpenStack.
A layered method for OpenStack network failures, from the instance interface through the local bridge to the router and out.
The Nova services, what each one owns, and how to tell from a stuck instance which component to investigate first.
Working a compute failure from the instance record down through nova-compute to libvirt, and knowing which layer to stop at.
Reading real OpenStack capacity across compute, storage, and network, and the accounting errors that make a cloud full before it is.
Why process and API checks miss the failures that matter, and how to build functional checks that exercise the paths users depend on.
An ordered method for OpenStack incidents that eliminates whole categories of cause quickly, and the habits that keep a bad incident from getting worse.
The routine practices that decide how often OpenStack incidents happen and how long they last — change discipline, maintenance, and knowing your own baseline.
How OpenStack uses the message bus, what a broker problem looks like from the services, and the queue growth that predicts an outage.
How OpenStack services communicate, why the message bus and database are the real dependencies, and what each service actually owns.
Where OpenStack configuration actually comes from in a Kolla-Ansible deployment, how overrides are merged, and how to scope a change so it does not touch the whole cloud.
How Nova decides where an instance runs, what the filter chain does, and how to find which filter eliminated every host.
Status versus state, how liveness is actually determined, and why a service that is enabled and down is the most dangerous row in any OpenStack listing.
3 resources
Operational procedures with validation and rollback.
Diagnose and remediate OpenStack volume creation failures caused by the scheduler having no candidate backends, including the case where the API reports healthy.
A structured procedure for a pod that will not stay running, covering exit-code classification, probe-induced restarts, OOM kills, and validation that the fix held.
Find what is consuming a full filesystem, reclaim space safely including the deleted-but-open-file case, and validate before the host causes wider failures.
5 resources
Scenario-based incident practice with real signals.
A guided production incident where instance builds fail for lack of capacity on a cloud that is barely a third used. Find the capacity that does not exist.
A guided production incident where the whole OpenStack control plane reports down while every workload keeps running. Work the evidence and find what is actually wrong.
A service crash-looping after a routine config change, with logs that show a normal startup and no error anywhere. Work out what is killing it.
Instances stopped building on a cloud with visible free capacity. Work the evidence, form hypotheses, and find why the scheduler has no candidates.
A routine-looking Terraform pull request. Find the changes that would destroy data before you approve it, and work out which one is not what it appears to be.
2 resources
Original measurement and analysis of the AI stack.
A planned survey of how teams actually operate AI infrastructure in production — what breaks, what is measured, and where engineering time goes. Not yet conducted.
A planned recurring measurement of accelerator pricing across cloud providers, normalised so the comparison is meaningful. Methodology published before data collection begins.