Storage Architecture for AI — Three Access Patterns, Three Failure Modes

“The GPUs are only 30% utilised” is a storage problem more often than it is anything else, and it is diagnosed as a compute problem more often than it is diagnosed correctly.

The reason is that AI workloads place three completely different demands on storage, and a system sized for one of them will fail at the others in ways that never surface as storage errors.

The three patterns

Dataset streaming — training

Reads the same dataset repeatedly, in a different random order each epoch. Throughput-bound, latency-tolerant, and close to the worst possible case for caching, because randomisation defeats readahead and the working set is usually larger than any cache.

The demand is sustained aggregate throughput under many concurrent random readers. A storage system that streams beautifully in a sequential benchmark can collapse under this pattern, which is why the benchmark number in the vendor’s datasheet is rarely the number you get.

Checkpoint writes — training

Every rank writes a large file simultaneously, every N steps, and training is paused until it completes. Bursty, write-heavy, and extremely sensitive to the tail.

The demand is peak write throughput for short intervals. What matters is not average bandwidth but how long the burst takes, because that duration is multiplied by the number of checkpoints and subtracted directly from training time.

Weight loading — inference

Reads tens of gigabytes once, at container start, then almost nothing. Latency-critical in the sense that it determines cold-start time, and therefore how fast you can scale out under load.

The demand is fast first-read of large files, from many nodes at once when scaling. A model that takes four minutes to load caps how quickly you can respond to a traffic spike, no matter how much accelerator capacity is available.

Designing for all three

These pull in different directions, which is why a single tier rarely serves all of them well. The usual resolution is a layered arrangement:

┌─────────────────────────────────────────────────────┐
│ Local NVMe on each node                             │
│   working set, active shards, loaded weights        │
│   fastest, smallest, not durable                    │
├─────────────────────────────────────────────────────┤
│ Shared high-performance tier                        │
│   full dataset, recent checkpoints                  │
│   parallel filesystem or fast distributed storage   │
├─────────────────────────────────────────────────────┤
│ Object storage                                      │
│   dataset of record, checkpoint archive, artifacts  │
│   cheapest, durable, slowest                        │
└─────────────────────────────────────────────────────┘

The design question is what gets staged where and when. Copying a shard of the dataset to local NVMe at job start turns random reads across the network into random reads against local flash, and is frequently the single largest performance improvement available on a cluster that was built without it.

Diagnosing storage as the cause

The symptom is accelerator utilisation lower than expected with no obvious cause. Establish whether storage is the constraint before tuning anything else:

# Are the data loader processes waiting on I/O?
# High iowait alongside low GPU utilisation is the signature.
vmstat 1 10

# Per-device service times and queue depth. A device with high
# await and high utilisation is saturated.
iostat -x 1 5

# For a network filesystem, is the bottleneck the client or the server?
# Measure achieved throughput from the client, then from the server.
nfsiostat 1 5

The distinguishing test is simple: stage a subset of the data locally and rerun. If utilisation jumps, the shared path is the constraint and you now know where to spend. If it does not, storage is exonerated and you can move on to the network or the loader.

Accelerator utilisation is low.

├─ High iowait on the host?
│   ├─ yes → storage path
│   │        ├─ local device saturated?  → stage differently, or faster local media
│   │        └─ shared tier saturated?   → aggregate throughput limit, or a hot spot
│   └─ no  → continue
├─ Data loader workers saturated on CPU?
│   └─ yes → decode and augmentation on the critical path, not storage
├─ Utilisation drops at regular intervals?
│   └─ yes → checkpointing. Measure the pause duration.
└─ Otherwise → look at the fabric and collectives, not storage

Metadata is the constraint nobody sizes for

A dataset of millions of small files stresses metadata operations far more than data throughput. Every file open is a metadata round trip, and at high concurrency the metadata service saturates long before the data path does.

The symptom is distinctive: low throughput, low device utilisation, high latency, and a storage system that appears idle while nothing is progressing.

The fix is usually to stop storing millions of small files. Packing training data into large sequential archive formats — records, shards, or tars read in order with shuffling applied to a buffer rather than to the file list — converts a metadata problem into a throughput problem, which is the problem you actually built the storage to solve.

Checkpointing without stopping training

Checkpoint duration is training time you do not get back. Three things reduce it:

Write asynchronously. Copy state to host memory quickly, resume training, and flush to storage in the background. The pause becomes the memory copy rather than the storage write.

Shard the checkpoint. Each rank writes its own portion in parallel rather than gathering everything to rank zero. Parallel writes to a parallel filesystem are what it is for.

Keep fewer checkpoints in the fast tier. Retain recent ones where they are quick to write and restore, and move older ones to object storage on a schedule.

Inference cold start

Cold start is dominated by pulling weights, and it sets how fast you can scale under load.

• Cache weights on local NVMe on nodes that serve a given model, so a restart is a local read
• Pre-pull to nodes before they are needed, if traffic is predictable enough to anticipate
• Keep weights out of the container image — huge images are slow to distribute and the layer cache does not help across nodes as much as expected
• Measure the load time, and treat a regression in it as a real regression, because it is a capacity constraint even when nothing looks broken

Checklist

• Storage measured under the workload’s actual access pattern, not a sequential benchmark
• Local staging used for the training working set where the dataset allows
• Metadata operation rate measured, not just throughput
• Small files packed into sequential archives
• Checkpoint duration measured separately from step time
• Checkpoint writes sharded and asynchronous where the framework supports it
• Restore path exercised at least once per training campaign
• Model load time monitored as a capacity signal for inference