GPU Infrastructure for AI Workloads — Memory, Bandwidth, and Topology

Accelerator specifications are marketed in FLOPS. Production performance is usually decided by memory bandwidth, interconnect topology, and whether the data arrives fast enough — none of which appear on the comparison chart.

This is what to reason about when you are sizing, buying, or debugging accelerator infrastructure.

Memory capacity sets what fits

Capacity is a hard wall. If the working set does not fit, the workload does not run — there is no graceful degradation, only an out-of-memory error.

For inference the working set is roughly: model weights, plus the KV cache for every concurrent request, plus activation memory, plus fragmentation.

Weights are the easy part. A model’s parameter count times the bytes per parameter gives a floor — 2 bytes for 16-bit, roughly 1 for 8-bit quantisation, roughly 0.5 for 4-bit. A 70B parameter model at 16-bit needs about 140 GB for weights alone, which is why it does not fit on one 80 GB device and why quantisation is not a nicety.

The part that surprises people is the KV cache, which grows linearly with both sequence length and concurrency. At long context and high concurrency it can exceed the size of the weights. Capacity planning that accounts only for weights will be wrong the first time real traffic arrives, and it will be wrong in the direction of an outage.

Bandwidth sets how fast it runs

Autoregressive generation reads the entire set of weights from memory for every token produced. That makes token generation memory bandwidth bound, not compute bound.

The consequence is direct: for single-stream generation, throughput scales with memory bandwidth, and a device with twice the FLOPS but the same bandwidth will not generate meaningfully faster.

Batching is what changes this. With a batch, the weights read once serve many sequences, moving the workload toward compute-bound and raising throughput per device substantially — at the cost of per-request latency. Almost every inference serving decision is somewhere on that trade.

Prefill behaves differently: processing the input prompt is compute-bound and parallel, which is why time-to-first-token and time-per-output-token have different scaling behaviour and should be measured separately.

Topology decides multi-GPU performance

When a workload spans devices, how those devices are connected matters more than what they are.

$ nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 CPU AffinityGPU0 X NV4 NV4 SYS 0-23GPU1 NV4 X NV4 SYS 0-23GPU2 NV4 NV4 X SYS 0-23GPU3 SYS SYS SYS X 24-47

Read that as a graph. NV4 is a high-bandwidth direct link. SYS means traffic crosses the CPU root complex — an order of magnitude slower and subject to contention with everything else on that path.

In this example GPU3 is the odd one out. A four-way job placed across all four devices runs at the speed of the GPU3 links, wasting most of the fabric it paid for. A three-way job on GPUs 0–2 would be substantially faster despite using less hardware.

NUMA affinity is the same problem one level down. A process pinned to CPU cores in one NUMA node while its GPU hangs off the other node’s PCIe complex pays a penalty on every host-to-device transfer. nvidia-smi topo -m shows the CPU affinity; use it.

Diagnosing low utilisation

Low nvidia-smi utilisation is a symptom with several causes, and they are distinguishable.

$ nvidia-smi –query-gpu=index,utilization.gpu,utilization.memory,memory.used,power.draw –format=csv -l 1# Sample over time, not once. A single reading tells you nothing about a# workload that alternates between compute and waiting.$ nvidia-smi dmon -s pucm# Power, utilisation, clocks, memory — per device, per second.

GPU utilisation is low. Where is the time going?

├─ Utilisation low, memory bandwidth low, power low
│   └─ The device is waiting for work
│       ├─ Data loading starved?    → check storage throughput and worker count
│       ├─ CPU preprocessing?       → check host CPU saturation
│       └─ Small batch?             → the device is underfed
│
├─ Utilisation low, but power high and clocks reduced
│   └─ Thermal or power capped     → check nvidia-smi -q -d TEMPERATURE,POWER
│
├─ Utilisation spiky, with regular gaps
│   └─ Synchronisation stalls      → collectives or gradient accumulation
│       └─ One slow rank sets the pace for all of them
│
└─ Utilisation high but throughput low
    └─ Doing work inefficiently    → precision, kernel selection, memory layout

The most common single cause on a new cluster is data loading. Training reads the dataset repeatedly in randomised order, which is close to the worst case for caching, and a storage path that benchmarks well sequentially can deliver a fraction of that under random access from many workers at once.

Collectives and the slowest rank

Distributed training synchronises gradients through collective operations. A collective completes when its slowest participant completes. That single property explains most large-cluster performance mysteries:

• One degraded network link slows the entire job, not one node
• One GPU running at reduced clocks due to heat slows the entire job
• Adding nodes to a job with an undersized fabric can make it slower

NCCL’s own diagnostics are the right starting point, and the NCCL documentation covers the environment variables that expose what it is actually doing. Running the bundled bandwidth tests between node pairs before running a real job is worth the hour it takes — it finds the one misconfigured link before it costs you a week of confusing training runs.

Health, before performance

Two failure classes produce misleading symptoms and are worth checking early.

ECC errors. Uncorrected errors on a device will fail or stall a job, often in ways that look like an application bug.

Thermal and power limits. A device that is throttling shows reduced clocks and lower throughput while reporting healthy status. In dense racks this is a facility problem presenting as a software problem.

$ nvidia-smi –query-gpu=index,ecc.errors.uncorrected.volatile.total –format=csv$ nvidia-smi -q -d TEMPERATURE,POWER,CLOCK | grep -E ‘Clocks Throttle|GPU Current Temp|Power Draw’# “Clocks Throttle Reasons” naming SW Power Cap or HW Thermal Slowdown# explains a performance drop that no code change caused.

Under an orchestrator

Kubernetes exposes accelerators as integer-valued extended resources through a device plugin. Three behaviours follow, and all three surprise people arriving from CPU workloads:

• No fractional allocation. A pod requests whole devices. There is no oversubscription and no throttling — either the devices are available or the pod stays Pending.
• No topology awareness by default. The scheduler will happily give you four devices with a slow path between them. Topology-aware placement requires additional configuration.
• No gang scheduling by default. A distributed job that needs eight pods can get six, hold six devices, and make no progress.

# Is the node advertising accelerators at all?
kubectl get nodes -o custom-columns=\
'NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

# If that column is empty, the problem is the device plugin, not capacity.
kubectl get pods -n kube-system -l k8s-app=nvidia-device-plugin-daemonset

Buying decisions

The questions that actually determine whether hardware performs, in rough order of how often they are the binding constraint:

1. Does the working set fit in memory? Capacity first — everything else is irrelevant if it does not fit.
2. What is the memory bandwidth? For inference this is your throughput ceiling.
3. What is the interconnect within a node? Direct high-bandwidth links versus PCIe changes multi-GPU scaling entirely.
4. What is the interconnect between nodes? For any job spanning nodes, this is usually the limit.
5. Can the storage path feed it? An accelerator waiting on data is an expensive idle device.
6. Can the facility power and cool it? Rack density is a real constraint on how many you can actually deploy.

Points 5 and 6 are the ones most often discovered after purchase.

Checklist

• nvidia-smi topo -m reviewed for the actual placement, not the intended one
• Memory sizing accounts for KV cache at target concurrency and context length, not just weights
• Utilisation sampled over time rather than read once
• Throttle reasons checked before investigating code
• ECC error counters monitored
• Collective bandwidth tested between node pairs before production jobs
• Device plugin health verified before diagnosing capacity
• Storage throughput measured under the access pattern the workload actually uses