AI Agents in Production
What changes when an agent loop moves from a demo to production: hard limits, tool authorisation, failure handling, and knowing when an agent is the wrong shape entirely.
Index
30 resources across 10 pillars. This page is a navigation index for readers, which is why it is not itself in a search index — the pillar hubs it links to are the pages that deserve that.
4 resources
Agents, RAG, model APIs, MCP, and evaluation in production.
What changes when an agent loop moves from a demo to production: hard limits, tool authorisation, failure handling, and knowing when an agent is the wrong shape entirely.
How to structure an AI application so that a slow model, a failed tool call, or a bad retrieval degrades one part of the system instead of the whole request path.
A working guide to retrieval-augmented generation: how to chunk, how to evaluate retrieval separately from generation, and how to diagnose a bad answer in the right order.
A layer-by-layer map of the modern AI stack, what each layer is actually responsible for, and where production systems tend to break in practice.
3 resources
GPUs, interconnects, storage, inference and training platforms.
Collective communication, RDMA, and rail topology explained from the operator's side, including why adding nodes to a training job can make it slower.
What determines accelerator performance in production: memory capacity versus bandwidth, interconnect topology, and the checks that find a misplaced workload.
Dataset streaming, checkpoint bursts, and model loading place completely different demands on storage. Designing for one and getting the others wrong is the usual outcome.
6 resources
Kubernetes, Docker, Terraform, Linux, and observability.
Exec format errors, permission denied, missing commands, bad entrypoints, immediate exits, and CrashLoopBackOff — diagnosed as one family rather than a page per error string.
Building container images for production: reproducible builds, correct signal handling, non-root runtime, layer strategy, and keeping secrets out of image history.
A single diagnostic path for workloads that will not schedule, will not stay up, or keep getting evicted — driven by events and previous logs rather than guesswork.
The operational settings that decide whether a workload survives a node failure or a rollout: requests and limits, the three probes, disruption budgets, and scheduling.
Running Terraform against production infrastructure: reviewing the plan mechanically, containing blast radius, structuring state, and avoiding the destroys nobody noticed.
The state commands that are genuinely dangerous, what each one actually does, and how to run them with a way back.
2 resources
Private cloud operations, with depth in OpenStack.
How Kolla-Ansible actually structures an OpenStack deployment, where configuration comes from, and how to make changes without discovering them during an outage.
Operating OpenStack in production: service state versus status, the message bus, placement disagreements, and the quiet failures that keep dashboards green.
2 resources
Incident response, diagnosis, and the operations workbench.
A repeatable method for diagnosing production failures: form hypotheses the evidence supports, order checks by what they eliminate, and know when to stop and mitigate.
Building observability that shortens incidents rather than producing dashboards: what to instrument, how to alert on symptoms, and why cardinality decides your bill.
5 resources
Structured learning paths for production engineers.
A sequenced learning path for engineers who operate OpenStack in production — architecture, service-by-service depth, deployment, and troubleshooting under pressure.
How OpenStack services communicate, why the message bus and database are the real dependencies, and what each service actually owns.
Where OpenStack configuration actually comes from in a Kolla-Ansible deployment, how overrides are merged, and how to scope a change so it does not touch the whole cloud.
How Nova decides where an instance runs, what the filter chain does, and how to find which filter eliminated every host.
Status versus state, how liveness is actually determined, and why a service that is enabled and down is the most dangerous row in any OpenStack listing.
3 resources
Operational procedures with validation and rollback.
Diagnose and remediate OpenStack volume creation failures caused by the scheduler having no candidate backends, including the case where the API reports healthy.
A structured procedure for a pod that will not stay running, covering exit-code classification, probe-induced restarts, OOM kills, and validation that the fix held.
Find what is consuming a full filesystem, reclaim space safely including the deleted-but-open-file case, and validate before the host causes wider failures.
3 resources
Scenario-based incident practice with real signals.
A service crash-looping after a routine config change, with logs that show a normal startup and no error anywhere. Work out what is killing it.
Instances stopped building on a cloud with visible free capacity. Work the evidence, form hypotheses, and find why the scheduler has no candidates.
A routine-looking Terraform pull request. Find the changes that would destroy data before you approve it, and work out which one is not what it appears to be.
2 resources
Original measurement and analysis of the AI stack.
A planned survey of how teams actually operate AI infrastructure in production — what breaks, what is measured, and where engineering time goes. Not yet conducted.
A planned recurring measurement of accelerator pricing across cloud providers, normalised so the comparison is meaningful. Methodology published before data collection begins.