AI Networking Fundamentals — Why the Fabric Sets the Limit

A distributed training job is a synchronised system. Every step, every rank exchanges gradients with every other rank, and nobody proceeds until that exchange completes. The network is therefore not a supporting service — it is in the critical path of every single step.

Collectives, and the slowest-participant rule

Data-parallel training runs the same model on every device over different data, then averages gradients across all of them. That average is an all-reduce: every rank contributes and every rank receives the result.

The operational property that matters is that a collective completes when its slowest participant completes. Not on average — the slowest. Which means:

• One link negotiated at half speed slows every rank in the job
• One GPU thermally throttling slows every rank in the job
• One node with a misconfigured MTU slows every rank in the job

A single degraded component does not degrade the job proportionally to its share. It sets the pace for everything.

Why RDMA

Ordinary TCP networking moves data through the kernel, copies buffers, and consumes CPU on both ends. At the volumes and latencies collectives need, that overhead dominates.

RDMA lets one machine’s network adapter write directly into another machine’s memory, bypassing the kernel and the CPU. Two deployments are common:

InfiniBand — a purpose-built fabric with its own switching, addressing, and subnet manager. Highest performance, its own operational model, and its own tooling to learn.

RoCE (RDMA over Converged Ethernet) — RDMA over standard Ethernet. Reuses your Ethernet skills and hardware, and requires a correctly configured lossless fabric to perform. That configuration — priority flow control and explicit congestion notification, consistently applied across every switch in the path — is where RoCE deployments go wrong.

Rails and topology

Multi-accelerator nodes typically have several network adapters, each associated with specific GPUs by PCIe locality. A rail is the set of links connecting the same adapter position across every node, usually through the same switch.

        Rail 0 switch        Rail 1 switch
            │                     │
  ┌─────────┼─────────┐ ┌─────────┼─────────┐
  │         │         │ │         │         │
node0:HCA0  node1:HCA0  node0:HCA1  node1:HCA1
   │           │           │           │
 GPU0,1      GPU0,1      GPU2,3      GPU2,3

Rail-aligned traffic — GPU0 on node0 talking to GPU0 on node1 — crosses one switch. Traffic that is not rail-aligned crosses more switches, or worse, traverses the host’s PCIe complex to reach a different adapter first.

Collective libraries can exploit rail alignment when the topology is discoverable and the job is placed accordingly. Under an orchestrator that places pods without topology awareness, they frequently cannot, and the job silently runs on the slow path.

Checking the fabric

$ ibstatCA ‘mlx5_0’ Port 1: State: Active Physical state: LinkUp Rate: 400# A port negotiated below its rated speed is a common and easily# missed cause of slow collectives. Check every port, not a sample.$ ibdev2netdev# Maps RDMA devices to network interfaces — needed to correlate# fabric state with what the host and the scheduler see.

For error counters, rising symbol_error or link_downed on a port indicates a physical problem — a cable, a transceiver, or a port — and these degrade rather than fail. A link that retrains repeatedly will not appear as down in any dashboard while it destroys your collective performance.

Measure before you debug

Do not diagnose a training job’s performance from the training job. Measure the fabric on its own.

The standard collective benchmarks measure achieved bandwidth for all-reduce and other operations between a set of ranks. Run them:

• Within a node, to confirm the internal links perform as expected
• Between node pairs, to find the one pair that is slower than the rest
• Across the full job size, to establish the ceiling your job cannot exceed

That last number is the important one. If the benchmark achieves 40% of the expected bandwidth at full job size, no amount of application tuning will help, and you have saved yourself from optimising the wrong layer.

The NCCL documentation covers the environment variables that expose which transport and topology it selected — worth enabling once on a new cluster, because it will tell you plainly if it has fallen back to a slower path.

MTU, and the quiet 30%

Large frames matter for RDMA workloads, and MTU has to be consistent along the entire path. A mismatch does not produce an obvious error — it produces fragmentation, retransmission, and a throughput number that is inexplicably lower than the benchmark suggested.

Check it on the hosts, on every switch in the path, and at any router. “It is set correctly on the servers” is where most of these investigations stop, one hop short of the answer.

Diagnosing a slow job

Distributed job is slower than expected.

1. Does the collective benchmark reach expected bandwidth at this job size?
   ├─ no  → the fabric is the limit. Do not tune the application.
   │        ├─ all pairs slow?     → configuration: MTU, flow control, routing
   │        └─ one pair slow?      → physical: cable, transceiver, port, link rate
   └─ yes → continue

2. Is any rank consistently the straggler?
   ├─ yes → that node: check clocks, thermals, ECC, and CPU contention
   └─ no  → continue

3. Does time per step grow as ranks are added?
   ├─ yes → communication is dominating. Consider gradient compression,
   │        larger per-rank batch, or a different parallelism strategy.
   └─ no  → the problem is not the network. Look at data loading.

What to check on a new cluster

• Every port active and negotiated at its rated rate — every port, not a sample
• Error counters at zero, then monitored for growth rather than checked once
• MTU consistent across hosts, switches, and routers on the full path
• For RoCE: flow control and congestion notification configured on every switch in the path
• Collective bandwidth benchmarked within node, between pairs, and at full job size
• Topology visible to the placement layer, and jobs actually placed rail-aligned
• The benchmark numbers recorded, so a future regression is measurable rather than argued about