Guide
AI Agents in Production
What changes when an agent loop moves from a demo to production: hard limits, tool authorisation, failure handling, and knowing when an agent is the wrong shape entirely.
intermediate· 4 minLLMAgents
Founder and Principal Engineer
James Joyner
Builds and operates the infrastructure layers underneath production AI systems.
- AI infrastructure
- OpenStack operations
- Kubernetes
- Terraform
- Linux systems engineering
- Observability
James founded Inside The AI Stack to publish the kind of infrastructure and operations material he wanted while running production systems: specific, tested where it claims to be tested, and written by someone who has had to fix the thing at 3am. He works across AI infrastructure, private cloud, and platform engineering, and reviews every technical resource published here before it is marked as verified.
Published here
Guide
AI Networking Fundamentals
Collective communication, RDMA, and rail topology explained from the operator's side, including why adding nodes to a training job can make it slower.
advanced· 4 minRDMAInfiniBand
Guide
Container Startup Troubleshooting
Exec format errors, permission denied, missing commands, bad entrypoints, immediate exits, and CrashLoopBackOff — diagnosed as one family rather than a page per error string.
intermediate· 4 minDockerContainers
Guide
Docker Production Engineering
Building container images for production: reproducible builds, correct signal handling, non-root runtime, layer strategy, and keeping secrets out of image history.
intermediate· 4 minDockerContainers
Guide
GPU Infrastructure for AI
What determines accelerator performance in production: memory capacity versus bandwidth, interconnect topology, and the checks that find a misplaced workload.
advanced· 6 minGPUNVIDIA
Guide
Incident Diagnosis Method
A repeatable method for diagnosing production failures: form hypotheses the evidence supports, order checks by what they eliminate, and know when to stop and mitigate.
intermediate· 5 minIncident ResponseObservability
Guide
Kolla-Ansible Production Architecture
How Kolla-Ansible actually structures an OpenStack deployment, where configuration comes from, and how to make changes without discovering them during an outage.
expert· 3 minOpenStackKolla-Ansible
Guide
Kubernetes Workload Troubleshooting
A single diagnostic path for workloads that will not schedule, will not stay up, or keep getting evicted — driven by events and previous logs rather than guesswork.
intermediate· 3 minKubernetes
Guide
OpenStack Production Operations
Operating OpenStack in production: service state versus status, the message bus, placement disagreements, and the quiet failures that keep dashboards green.
expert· 3 minOpenStackNova
Guide
Production AI Application Architecture
How to structure an AI application so that a slow model, a failed tool call, or a bad retrieval degrades one part of the system instead of the whole request path.
intermediate· 6 minLLMRAG
Guide
Production Kubernetes Operations
The operational settings that decide whether a workload survives a node failure or a rollout: requests and limits, the three probes, disruption budgets, and scheduling.
intermediate· 4 minKubernetesContainers
Guide
Production Observability
Building observability that shortens incidents rather than producing dashboards: what to instrument, how to alert on symptoms, and why cardinality decides your bill.
intermediate· 4 minPrometheusGrafana
Guide
RAG Architecture Guide
A working guide to retrieval-augmented generation: how to chunk, how to evaluate retrieval separately from generation, and how to diagnose a bad answer in the right order.
intermediate· 5 minRAGEmbeddings
Guide
Storage Architecture for AI
Dataset streaming, checkpoint bursts, and model loading place completely different demands on storage. Designing for one and getting the others wrong is the usual outcome.
advanced· 4 minStorageNVMe
Guide
Terraform Production Practices
Running Terraform against production infrastructure: reviewing the plan mechanically, containing blast radius, structuring state, and avoiding the destroys nobody noticed.
intermediate· 4 minTerraformOpenTofu
Guide
Terraform State Operations
The state commands that are genuinely dangerous, what each one actually does, and how to run them with a way back.
advanced· 3 minTerraformOpenTofu
Guide
The AI Stack Explained
A layer-by-layer map of the modern AI stack, what each layer is actually responsible for, and where production systems tend to break in practice.
foundational· 8 minLLMKubernetes
Runbook
Cinder — No Valid Backend
Diagnose and remediate OpenStack volume creation failures caused by the scheduler having no candidate backends, including the case where the API reports healthy.
expert· 20 minOpenStackCinder
Runbook
CrashLoopBackOff Triage
A structured procedure for a pod that will not stay running, covering exit-code classification, probe-induced restarts, OOM kills, and validation that the fix held.
intermediate· 15 minKubernetes
Runbook
Linux Disk Pressure Response
Find what is consuming a full filesystem, reclaim space safely including the deleted-but-open-file case, and validate before the host causes wider failures.
intermediate· 15 minLinuxKubernetes
LabPro
Phantom Capacity Incident
A guided production incident where instance builds fail for lack of capacity on a cloud that is barely a third used. Find the capacity that does not exist.
advanced· 40 minOpenStackNova
LabPro
RabbitMQ Partition Incident
A guided production incident where the whole OpenStack control plane reports down while every workload keeps running. Work the evidence and find what is actually wrong.
advanced· 35 minOpenStackRabbitMQ
Lab
Lab: CrashLoopBackOff
A service crash-looping after a routine config change, with logs that show a normal startup and no error anywhere. Work out what is killing it.
intermediate· 25 minKubernetes
Lab
Lab: Nova NoValidHost
Instances stopped building on a cloud with visible free capacity. Work the evidence, form hypotheses, and find why the scheduler has no candidates.
expert· 35 minOpenStackNova
Lab
Lab: Destructive Plan Review
A routine-looking Terraform pull request. Find the changes that would destroy data before you approve it, and work out which one is not what it appears to be.
intermediate· 20 minTerraformAWS
Learning path
OpenStack Production Engineer
A sequenced learning path for engineers who operate OpenStack in production — architecture, service-by-service depth, deployment, and troubleshooting under pressure.
intermediate· 5 minOpenStackNova
LessonPro
Cinder Architecture
How Cinder models volumes and drivers, what backends report about themselves, and why the API stays green when storage is failing.
intermediate· 45 minOpenStackCinder
LessonPro
Storage Troubleshooting
Diagnosing Cinder scheduling failures and attachment failures, including the capacity report that is accurate and still wrong.
advanced· 50 minOpenStackCinder
LessonPro
HAProxy and Endpoints
How the load balancer sits in front of every OpenStack API, the health checks that lie, and the failures that appear as service errors.
intermediate· 40 minOpenStackHAProxy
LessonPro
Keystone Identity
How authentication and service discovery actually work in OpenStack, and why a Keystone problem presents as every other service failing simultaneously.
intermediate· 45 minOpenStackKeystone
LessonPro
Kolla-Ansible Architecture
How Kolla-Ansible deploys OpenStack as containers, what the inventory controls, and why the running configuration is not the one you edited.
intermediate· 45 minOpenStackKolla-Ansible
LessonPro
Upgrade Strategy
Planning an OpenStack upgrade as a sequence of stages, each with a validation and a defined position to stop at.
expert· 55 minOpenStackKolla-Ansible
LessonPro
MariaDB Operations
How OpenStack uses its database, what Galera guarantees and does not, and the failure modes that present as application errors.
advanced· 45 minOpenStackMariaDB
LessonPro
Neutron Architecture
How Neutron models networks as ports and bindings, what each agent does, and why a port stuck in DOWN is the most informative failure in OpenStack.
advanced· 50 minOpenStackNeutron
LessonPro
Networking Troubleshooting
A layered method for OpenStack network failures, from the instance interface through the local bridge to the router and out.
advanced· 55 minOpenStackNeutron
LessonPro
Nova Architecture
The Nova services, what each one owns, and how to tell from a stuck instance which component to investigate first.
intermediate· 45 minOpenStackNova
LessonPro
Compute Troubleshooting
Working a compute failure from the instance record down through nova-compute to libvirt, and knowing which layer to stop at.
advanced· 55 minOpenStackNova
LessonPro
Capacity Management
Reading real OpenStack capacity across compute, storage, and network, and the accounting errors that make a cloud full before it is.
advanced· 45 minOpenStackNova
LessonPro
Health Checks
Why process and API checks miss the failures that matter, and how to build functional checks that exercise the paths users depend on.
advanced· 45 minOpenStackPrometheus
LessonPro
Incident Response
An ordered method for OpenStack incidents that eliminates whole categories of cause quickly, and the habits that keep a bad incident from getting worse.
advanced· 50 minOpenStack
LessonPro
Production Operations
The routine practices that decide how often OpenStack incidents happen and how long they last — change discipline, maintenance, and knowing your own baseline.
advanced· 45 minOpenStack
LessonPro
RabbitMQ Operations
How OpenStack uses the message bus, what a broker problem looks like from the services, and the queue growth that predicts an outage.
advanced· 45 minOpenStackRabbitMQ
Lesson
Control Plane Architecture
How OpenStack services communicate, why the message bus and database are the real dependencies, and what each service actually owns.
foundational· 45 minOpenStackRabbitMQ
LessonPro
Kolla Configuration
Where OpenStack configuration actually comes from in a Kolla-Ansible deployment, how overrides are merged, and how to scope a change so it does not touch the whole cloud.
intermediate· 40 minOpenStackKolla-Ansible
LessonPro
Nova Scheduling
How Nova decides where an instance runs, what the filter chain does, and how to find which filter eliminated every host.
intermediate· 50 minOpenStackNova
Lesson
Service State and Liveness
Status versus state, how liveness is actually determined, and why a service that is enabled and down is the most dangerous row in any OpenStack listing.
foundational· 35 minOpenStackNova
Research
AI Infrastructure Operations Survey
A planned survey of how teams actually operate AI infrastructure in production — what breaks, what is measured, and where engineering time goes. Not yet conducted.
foundational· 2 minAI InfrastructureGPU
Research
GPU Cloud Price Index
A planned recurring measurement of accelerator pricing across cloud providers, normalised so the comparison is meaningful. Methodology published before data collection begins.
foundational· 2 minGPUCloud