The AI Stack Explained — From Application Down to Silicon

Most descriptions of “the AI stack” are drawn by whoever is selling one layer of it. Vendors of vector databases draw a stack where retrieval is the centre of gravity. GPU cloud providers draw one where everything above the scheduler is a thin wrapper. Neither is much help at 2am when latency has tripled and you need to know which layer to look at first.

This is a map drawn from the operational side: eight layers, what each one is actually responsible for, what it assumes about the layer below it, and the specific way each one tends to fail. If you only take one thing from this page, take this — most AI production incidents are not model problems. They are capacity, networking, scheduling, or data problems wearing a model problem’s clothes.

The eight layers

L8  Applications          Agents            product surface, tool loops
L7  Models                RAG / Data        inference, retrieval, context
L6  AI Engineering        LLMOps            evaluation, tracing, cost
L5  Kubernetes            Containers        scheduling, isolation, rollout
L4  Terraform             Automation        provisioning, state, drift
L3  Cloud                 OpenStack         capacity, identity, quota
L2  GPUs                  Networking        accelerators, interconnect
L1  Compute               Storage           servers, disks, power, cooling

Each layer makes an assumption about the one beneath it, and every one of those assumptions is false under load. L8 assumes the model responds in bounded time. L5 assumes a GPU that is allocated is a GPU that works. L2 assumes the fabric delivers at line rate. Production is the process of discovering which assumption broke today.

L1 — Compute and storage

The physical layer: servers, local NVMe, distributed storage, power, and cooling. It is the layer everything above it silently treats as infinite and reliable.

Two things make this layer different for AI than for general compute. The first is power density. A conventional rack draws 5–15 kW; a dense accelerator rack can draw 40–120 kW, which changes cooling, power distribution, and floor loading from facilities details into design constraints that determine how many accelerators you can actually deploy.

The second is the shape of the I/O. Training reads the same dataset repeatedly in randomised order — throughput-bound, latency-tolerant, cache-hostile. Checkpointing writes very large files in bursts from every rank at once. Inference is different again: model weights are read once at load and then the working set is small, but cold-start time is dominated by how fast you can pull tens of gigabytes onto a node.

L2 — Accelerators and the fabric between them

At single-node scale this layer is about the accelerator: memory capacity, memory bandwidth, and the numeric formats it supports. Memory capacity sets what fits; memory bandwidth usually sets how fast inference runs, because token generation is bandwidth-bound rather than compute-bound.

At multi-node scale the interconnect becomes the limiting factor. Distributed training synchronises gradients every step through collective operations — all-reduce, all-gather, reduce-scatter. A collective completes when its slowest participant completes, which means one degraded link sets the pace for the entire job, and adding nodes to a job with an undersized fabric makes it slower rather than faster.

This is why interconnect topology, RDMA configuration, and rail alignment matter far more than their share of the budget suggests. Start here:

$ nvidia-smi topo -m# Shows GPU-to-GPU link types. NV# means NVLink; SYS means the# traffic crosses the CPU root complex, which is far slower.$ ibstat# Port state and rate per HCA. A port negotiated below its rated# speed is a common and easily missed cause of slow collectives.$ nvidia-smi –query-gpu=index,ecc.errors.uncorrected.volatile.total –format=csv# Uncorrected ECC errors on one GPU will stall the whole job.

L3 — Cloud and control plane

This layer turns hardware into schedulable capacity: identity, placement, quotas, block and object storage, and virtual networking. Whether it is a hyperscaler API or an OpenStack control plane you run yourself, the responsibilities are the same.

The operational characteristic that matters is that control planes fail quietly. The API stays up and keeps accepting requests while the thing that fulfils them has stopped participating. A scheduler with no candidate hosts does not return an error saying “my agents are gone” — it returns NoValidHost, which reads like a capacity problem and is usually a service-state problem.

$ openstack compute service list$ openstack network agent list$ openstack volume service list# Read two columns, not one: Status is administrative intent# (enabled/disabled), State is reality (up/down). A service that# is “enabled” and “down” is the one that will ruin your evening.

L4 — Provisioning and automation

Terraform, OpenTofu, Ansible, and everything that turns a described state into a real one. The value of this layer is reproducibility. Its risk is blast radius: automation applies a mistake to every host as efficiently as it applies a fix.

The single highest-leverage practice here is treating the plan as the review artifact. Not the code — the plan. Code review tells you what someone intended; the plan tells you what will happen to production.

# Produce a plan you can review mechanically rather than by eye.
terraform plan -out=tfplan
terraform show -json tfplan > plan.json

# Every resource this apply would destroy or replace:
jq -r '
  .resource_changes[]
  | select(.change.actions | index("delete"))
  | "\(.change.actions | join(",")): \(.address)"
' plan.json

L5 — Orchestration

Kubernetes and the container runtime: scheduling, isolation, resource limits, rollout, and recovery. For AI workloads, three things behave differently from ordinary services.

Accelerators are not divisible like CPU. GPUs are exposed through a device plugin as integer-valued extended resources. There is no 0.5 GPU in the standard scheduler, no overcommit, and no throttling — a pod either gets whole devices or it does not schedule at all.

Startup is slow and the defaults do not expect it. A container image carrying CUDA libraries plus tens of gigabytes of weights takes minutes to pull and load. Liveness probes tuned for a web service will kill the pod repeatedly before it ever becomes ready, producing a CrashLoopBackOff that looks like an application bug and is really a probe configuration bug.

Distributed jobs need all-or-nothing scheduling. Placing 6 of 8 required pods and waiting holds 6 GPUs idle while making no progress. This is the gang-scheduling problem, and the default scheduler does not solve it.

# Is the node actually advertising accelerators?
kubectl get nodes -o custom-columns=\
'NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

# Why is this pod pending? Read Events, not Status.
kubectl describe pod <pod> | sed -n '/Events:/,$p'

# Why did the last container exit? Previous logs survive the restart.
kubectl logs <pod> --previous --tail=50

L6 — AI engineering and LLMOps

Evaluation, tracing, cost control, and release safety for systems whose output is not deterministic. This layer exists because the practices that make ordinary software safe to change — unit tests, exact-match assertions, deterministic replay — do not transfer.

You cannot assert equality on a generated response. What you can do is measure a fixed suite of representative inputs against graded criteria on every change, and treat a drop as a regression the same way you would treat a failing test. Teams that skip building this are not moving faster; they are changing prompts and models with no ability to tell improvement from damage.

Tracing matters for the same reason. A slow agent response is a sum: retrieval time, plus prompt assembly, plus queueing at the inference endpoint, plus generation time proportional to output length, plus every tool call the agent decided to make. Without a trace that attributes latency across those spans, you are guessing, and the usual guess — “the model is slow” — is usually wrong.

L7 — Models, retrieval, and context

Model selection, prompt and context construction, retrieval, and embeddings. The dominant failure mode here is not the model being wrong. It is the model being asked the wrong question, because the context it received did not contain what it needed.

When a retrieval-augmented system gives a bad answer, the diagnostic order is almost always:

1. Was the correct document retrieved at all? If not, the problem is in chunking, embedding, or the query — not the model. Log retrieved chunk IDs on every request; without this you cannot answer the question.
2. Was it retrieved but ranked too low to survive truncation? A reranking or context-budget problem.
3. Was it present in context and still ignored? Only now is this a model or prompt problem.

Skipping straight to step 3 — swapping models to fix what is actually a chunking bug — is the most common wasted week in AI engineering.

L8 — Applications and agents

The product surface. An agent is a loop: the model chooses a tool, the tool runs, the result returns to the model, repeat until done. The engineering problem is that the loop is driven by a non-deterministic decision-maker with real credentials.

Three constraints belong in the loop itself, not in a prompt asking nicely:

• A hard iteration limit. Loops that fail to terminate are the default failure, not the exception. Cap iterations and fail loudly at the cap.
• A budget per task. Tokens, wall-clock time, and tool invocations. Enforce in code.
• Authorisation at the tool boundary. The tool must check permissions itself, because the model deciding to call it is not evidence that the caller was allowed to.

MCP has become the common way to expose tools to models, which makes the tool boundary a real security boundary rather than an implementation detail. Treat every tool as an authenticated API endpoint that happens to have a model as its client.

Which layer owns your problem?

A triage order that resolves most incidents faster than starting at the top:

Symptom: latency increased
  ├─ Is time going to generation, retrieval, or queueing?   → trace first (L6)
  │   ├─ queueing        → capacity or batching             (L5, L2)
  │   ├─ retrieval       → index size, ranking, backend     (L7, L1)
  │   └─ generation      → longer outputs or slower device  (L7, L2)
  └─ No trace available? Build that before guessing.

Symptom: throughput dropped, GPUs underutilised
  ├─ Data loading starved?          → storage throughput    (L1)
  ├─ Collectives slow?              → fabric, topology      (L2)
  └─ Fewer workers running?         → scheduling, evictions (L5)

Symptom: workload will not start
  ├─ Pending                        → scheduling, resources (L5)
  ├─ CrashLoopBackOff               → probes, image, config (L5)
  └─ NoValidHost / no valid backend → control plane state   (L3)

Symptom: answers got worse, nothing was deployed
  ├─ Retrieval logs show the right chunks?  → no  → data (L7)
  └─ Yes                                    → model, prompt, or context budget (L7, L6)

The pattern worth internalising: start where the evidence is, and prefer the layer that can be measured over the layer that is easiest to blame. Models are easy to blame and hard to measure. Storage throughput, link speed, scheduler events, and retrieval logs are all easy to measure — and that is usually where the answer is.

Where to go from here

If you own the application layer, the next thing worth reading is how these pieces assemble into something you can operate. If you own infrastructure, start at L2 and work down — that is where the leverage is, and where the least is written honestly.