Runbook: NCCL / fabric performance regression, by layer¶
Scope: collectives are slower than they used to be and nothing is broken. Find the lowest layer at which measurement diverges from its recorded baseline, and change only that layer.
Run this when a distributed job is slower than a known-good run and the cause is not obvious:
busbwunder baseline, step time up with comms as the dominant phase, or a fleet that got slower after a driver, firmware, NCCL or cabling change. This is the dispatcher for the fabric runbook family: it decides which of them you are actually in. If the collective has stopped making progress entirely, that is the NCCL-hang runbook, not this one.Commands are reference templates on real APIs. Flag semantics for
nccl-testswere read from its source at commit717b683;rdma statistic show linkwas confirmed againstiproute2-6.1.0. Nothing here was run against a GPU fabric during authoring. The Python block was executed; its output is pasted verbatim.
The method is differential, not absolute. "Slow" has no meaning without a baseline, and the baseline has to be for this topology: an eight-node result compared against a two-node number, or a cross-rack pair compared against an intra-rack one, produces a confident wrong answer. The executed model at the end shows exactly that failure.
Related pages own the pieces and should not be re-derived here: DCGM run levels are in diagnostics and validation; the busbw correction factors are in NCCL collectives and algorithms; the perftest CUDA build and ibstat interpretation are in fabric bring-up and benchmarking; transport-string classification and the socket-fallback cause are in the socket-fallback runbook; nvidia-smi topo -m NUMA columns are in NUMA and CPU pinning; a standing per-pair matrix as a service is continuous NCCL fabric benchmarking.
Trigger¶
busbwat the large-message tail below the recorded baseline for the same rank count, topology and message range.- Training step time up with the collective phase dominant, and no hang (MFU regression is where you came from if the profiler pointed at comms).
- Tensor-parallel inference latency up on multi-node replicas but not on single-node ones.
- A platform change landed: driver, GPU firmware, NIC firmware, switch software, NCCL version, kernel, node image, or a recabling.
- One node or one rail looks worse than its peers in a per-pair sweep.
Pre-checks¶
- Find the baseline before touching anything. Same GPU model and count, same node count, same topology, same driver, same CUDA, same NCCL, same message range, same collective, same rank placement. If any of those differ you are comparing two systems, not detecting a regression.
- Confirm there is no hang. Progress that has stopped is a different runbook. This one is for progress that is slow.
- Confirm the job is not competing. A second workload on the same nodes or the same fabric explains a lot of regressions and none of the interesting ones.
- Do not start by setting NCCL environment variables. Tuning before localising changes the system you are trying to measure, and NVIDIA's own guidance is that a setting which helps one benchmark can hurt other workloads.
Flow¶
stateDiagram-v2
[*] --> Baseline
Baseline --> Node: baseline recorded
Node --> IntraNCCL: DCGM clean
Node --> HW: DCGM fails
IntraNCCL --> HostRDMA: intra-node at baseline
IntraNCCL --> Local: intra-node degraded
HostRDMA --> GpuRDMA: host RDMA at baseline
HostRDMA --> Fabric: host RDMA degraded
GpuRDMA --> MultiNCCL: GPU RDMA at baseline
GpuRDMA --> GDR: GPU RDMA degraded only
MultiNCCL --> App: multi-node at baseline
MultiNCCL --> Transport: multi-node degraded
App --> [*]: regression is above the fabricProcedure¶
The ladder has six rungs. Run them in order and stop at the first one that diverges from its baseline: the fault is in that layer or the one immediately below it. Every rung above a failing rung is uninterpretable, which is why running the application benchmark first tells you nothing.
Rung 0. Record what you are testing¶
A per-rung baseline is only comparable if the profile is pinned. Capture, per run:
GPU product and count per node driver version
node product and node count CUDA runtime
GPU topology (nvidia-smi topo -m) NCCL version
HCA product, count, and rail mapping NIC firmware and RDMA stack
fabric domain and switch path container image digest
CPU and NUMA placement launcher and rank mapping
NCCL_* environment variables message-size range and collective
Inside a container, confirm /sys exposes the real PCI topology. NCCL discovers GPU, PCI and NIC relationships through /sys, and a virtualised or trimmed view leads it to pick a worse transport for reasons that have nothing to do with the fabric.
A baseline is a specific sentence, not a number: "8 x H100 SXM, one node, NVSwitch", "16 x H100 SXM, two nodes, one rail", "64 x H100 SXM, eight nodes, four rails". Keep them separate. They are not versions of each other.
Rung 1. Each node, on its own¶
nvidia-smi topo -m # GPU-GPU, GPU-NIC, GPU-NUMA relationships
dcgmi diag -r 1 # quick gate
dcgmi diag -r 3 # deeper, on a drained node only
Run levels and what each covers are tabulated in diagnostics and validation. The active diagnostic needs exclusive access to the GPU, so the longer levels belong on a node removed from scheduling; that is operational practice in this knowledge base rather than a sentence in NVIDIA's diagnostics documentation, which does not say it. Read the output for PCIe replay and link errors, NVLink errors, ECC and pending page retirement, Xid events, thermal or power throttling, clock differences between GPUs, and memory-bandwidth outliers.
The long run level nominally includes NVIDIA's own nccl_tests plugin alongside pcie, memory, memory_bandwidth, diagnostic, targeted_stress, targeted_power and nvbandwidth. That is a useful shortcut and it is not a substitute for rung 2, because those plugins can be skipped rather than run: the NCCL plugin needs its test binary located, and some plugins are gated off by default. A skip is not a pass. Read the per-test result rather than the overall verdict, in JSON so a gate can check it:
If nccl_tests or nvbandwidth did not actually execute, run rung 2 by hand.
The comparison that matters here is between supposedly identical nodes. A single node whose topo -m shows GPU0 reaching NIC0 across a socket boundary while its siblings do not is a regression even though every device reports healthy.
If DCGM fails, stop. Route to the GPU-fault runbook, the NVLink visibility runbook, or the PCIe bandwidth runbook by fault class. Nothing above this rung means anything on a node with bad silicon.
Rung 2. Intra-node NCCL¶
This rung is the discriminator most triage skips, and it is the cheapest information in the whole procedure: if intra-node collectives have regressed, the external network is not yet a suspect.
git clone https://github.com/NVIDIA/nccl-tests.git && cd nccl-tests
make -j CUDA_HOME=/usr/local/cuda NCCL_HOME=/usr # see note on NCCL_HOME below
./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8 -w 20 -n 120 -K 20 -c 1 -I 1
Flags, read from src/common.cu at commit 717b683: -b minimum bytes, -e maximum bytes, -f step factor, -g GPUs per thread, -w warm-up iterations, -n measured iterations, -c the check-iteration count, and -I 1 per-iteration CUDA event timing. -c defaults to 1, so passing -c 1 documents the intent rather than enabling anything; the meaningful lever is -c 0 to switch correctness checking off, which also changes the buffer budget. Two things about -I that the flag name hides: it is incompatible with CUDA-graph mode (-G) and disables itself if both are set, and its i_p99 uses a nearest-rank percentile, so below 100 samples it can equal i_max. -K does not drop warm-up iterations, which is what -w controls; it excludes leading measured samples from the -I summary statistics. Raise -n to match whatever you pass to -K, or the summary set walks back under the 100-sample threshold.
Run the collectives your workload actually issues, not only all-reduce:
./build/all_gather_perf -b 8 -e 8G -f 2 -g 8 -w 20 -n 100 -c 1 -I 1
./build/reduce_scatter_perf -b 8 -e 8G -f 2 -g 8 -w 20 -n 100 -c 1 -I 1
./build/alltoall_perf -b 8 -e 8G -f 2 -g 8 -w 20 -n 100 -c 1 -I 1
./build/sendrecv_perf -b 8 -e 8G -f 2 -g 8 -w 20 -n 100 -c 1 -I 1
Pass NCCL_HOME explicitly even when NCCL is in the default location. The Makefile guards its include and library flags with a comparison against a literal two-character string, so the branch is always taken; with NCCL_HOME unset the build injects a bogus -I/include/ -L/lib. Confirmed by evaluating make -n at commit 717b683, with and without the variable set.
The build produces all_reduce, all_gather, broadcast, reduce_scatter, reduce, alltoall, alltoallv, scatter, gather, sendrecv and hypercube, each suffixed _perf (and _perf_mpi under NAME_SUFFIX=_mpi). Which ones matter depends on the workload: all-reduce and all-gather for tensor-parallel inference and DDP, reduce-scatter and all-gather for sharded training, all-to-all for mixture-of-experts routing, send/receive for pipeline parallelism. A fabric can pass all-reduce and regress on all-to-all, because the traffic distribution and the routing pressure are different. Compare on busbw, which normalises for the collective's pattern; the official correction factors are tabulated in NCCL collectives and algorithms, and nccl-tests applies factor 1 for send/receive in its own source.
What to look for, in order of how often it is the answer:
| Symptom against baseline | First suspect |
|---|---|
| Every message size slower | driver, clocks, topology discovery, or algorithm selection |
| Only large messages slower | NVLink or PCIe bandwidth, or protocol selection |
| Only small messages slower | launch overhead, CPU affinity, algorithm selection |
| One GPU pair slower | NVLink path, PCIe path, or that GPU |
| High per-iteration p99 with a normal mean | thermal throttling, a competing workload, or hardware errors |
#wrong non-zero |
stop everything; this is correctness, not performance |
Rung 3. Raw RDMA, without NCCL¶
Test the wire before blaming the collective. Pick node pairs deliberately: same switch, different rack, each rail, one known-good pair, one suspected pair.
A port can be Active and still have negotiated a lower width or rate than the fabric supports, which is a downshift, not a fault, and it will not appear anywhere else. Check every HCA the workload uses, not the default one.
Then host-memory bandwidth and latency, server side first:
# server # client
ib_write_bw -d mlx5_0 -F -a ib_write_bw -d mlx5_0 -F -a <server>
ib_write_lat -d mlx5_0 -F ib_write_lat -d mlx5_0 -F <server>
Flag meanings and the CUDA-enabled build are covered in fabric bring-up and benchmarking. Run each HCA and each rail separately rather than testing the default device and generalising.
Rung 4. GPU-memory RDMA¶
Repeat the bandwidth test against GPU memory to isolate the GPUDirect path from the network path. This comparison is NVIDIA's own recommendation, not an invention of this runbook: "When troubleshooting GPUDirect RDMA, it is often useful to compare host-memory results with GPU-memory results."1
# GPU memory. Note the flag takes a REQUIRED device index; a bare --use_cuda is invalid.
ib_write_bw -d mlx5_0 --use_cuda=<gpu_id> -a # server
ib_write_bw -d mlx5_0 --use_cuda=<gpu_id> <server> -a # client
# DMA-BUF path, where the kernel, driver and perftest build support it. Pass the same
# flags on BOTH ends; a one-sided registration compares two different paths.
ib_write_bw -d mlx5_0 --use_cuda=<gpu_id> --use_cuda_dmabuf -a # server
ib_write_bw -d mlx5_0 --use_cuda=<gpu_id> --use_cuda_dmabuf <server> -a # client
--use_cuda requires a CUDA-capable perftest build and takes a GPU index as an argument; --use_cuda_dmabuf takes none. Both are gated at runtime and only appear in the help output when the build supports them, which is why NVIDIA tells you to check first: "Check ib_write_bw --help on your system for the exact GPU-memory options supported by that build before comparing results."1 The CUDA build itself is covered in fabric bring-up and benchmarking.
The two-cell comparison is the whole point of separating rungs 3 and 4:
| Host-memory RDMA | GPU-memory RDMA | Diagnosis |
|---|---|---|
| slow | slow | HCA, cable, switch, routing, or congestion |
| at baseline | slow | GPUDirect, PCIe locality, DMA-BUF, or peer-memory |
| at baseline | at baseline | the wire is fine; go up a rung |
| variable | variable | congestion, retransmission, or a flapping link |
Rung 4b. Counters, measured across the test rather than read once¶
A non-zero lifetime counter is not evidence about this incident. A counter that grows during a controlled test is.
rdma statistic show link > rdma-before.txt
# run the bandwidth or NCCL test here
rdma statistic show link > rdma-after.txt
diff -u rdma-before.txt rdma-after.txt
rdma statistic show link is part of iproute2's rdma utility and prints the driver's default hardware counters. Watch for growth in retry and sequence-error counters (rnr_nak_retry_err, packet_seq_err, local_ack_timeout_err, implied_nak_seq_err on Mellanox hardware; the set is device-dependent, so read the names off an idle device rather than assuming). Note that rdma statistic mode supported lists the optional counters you can additionally switch on with rdma statistic set link ... optional-counters ..., not the defaults above, so an absence there is not evidence your device lacks them.
For InfiniBand port counters and physical link quality, and for a fabric-wide view:
sudo perfquery -x <lid> # extended port counters: symbol errors, link recovery, discards
mlxlink -d <mst_device> -c -e --show_fec # counters, eye opening, FEC; bare -d gives status only
sudo ibdiagnet # fabric-wide: missing links, bad ports, routing consistency
ibdiagnet needs subnet-manager access and is usually the network operator's tool, not the workload owner's; fabric bring-up and benchmarking and the NCCL-hang runbook cover its options and report location. NVIDIA ties it directly to the per-rail case this runbook reaches at rung 5: "if NCCL works on some rails but repeatedly fails or slows down on one specific rail, run ibdiagnet and inspect the generated report for missing links, bad ports, or routing inconsistencies affecting the switch ports connected to that rail before retrying the NCCL job."2 On RoCE, the equivalent evidence is on the NIC:
Correlate pause frames, ECN marks, CNP packets, queue drops and retransmissions. A RoCE regression is often a congestion-control or lossless-configuration problem with every link still reporting up (RDMA and RoCE tuning).
Rung 5. Multi-node NCCL, in two shapes¶
Rebuild with MPI and turn on diagnostics that survive being written by many ranks at once:
# `make clean` is not optional here. Object files built at rung 2 without MPI are not
# rebuilt by adding MPI=1, because their prerequisites have not changed. The binaries
# relink against libmpi, MPI_Init stays compiled out, and `mpirun -np 16` then launches
# 16 independent single-process runs that report plausible numbers for nothing.
make clean
make -j MPI=1 MPI_HOME=/path/to/mpi CUDA_HOME=/usr/local/cuda NCCL_HOME=/usr NAME_SUFFIX=_mpi
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,GRAPH,NET,TUNING,ENV
export NCCL_DEBUG_FILE=/tmp/nccl.%h.%p.log # per-host, per-process; ranks stop clobbering
export NCCL_TOPO_DUMP_FILE=/tmp/nccl-topology.xml
NCCL_DEBUG_FILE is the flag that makes NCCL_DEBUG=INFO usable past a handful of ranks: without it, a sixty-four-rank log is one interleaved stream nobody can read. %h and %p expand to host and process id.
Run two shapes, and the difference between them is the diagnosis:
# Shape A: one GPU per node. Isolates the inter-node path from everything local.
# -x forwards each variable to the REMOTE ranks. Without it only the launch node sees
# them, so the debug settings and any transport override apply to half the job.
mpirun -np 2 -H node-a:1,node-b:1 \
-x NCCL_DEBUG -x NCCL_DEBUG_SUBSYS -x NCCL_DEBUG_FILE -x NCCL_TOPO_DUMP_FILE \
./build/all_reduce_perf_mpi -b 8 -e 8G -f 2 -g 1 -w 20 -n 120 -c 1 -I 1
# Shape B: every GPU. Exercises multiple HCAs, rails, GPU-to-NIC affinity, and the
# full ring/tree construction at once.
mpirun -np 16 -H node-a:8,node-b:8 \
-x NCCL_DEBUG -x NCCL_DEBUG_SUBSYS -x NCCL_DEBUG_FILE -x NCCL_TOPO_DUMP_FILE \
./build/all_reduce_perf_mpi -b 8 -e 8G -f 2 -g 1 -w 20 -n 120 -c 1 -I 1
Two launcher traps. Open MPI's mpirun exports a variable to remote nodes only when you pass -x; nothing forwards your shell environment automatically, so a run without it measures default settings on every node but the first. Slurm's srun propagates the environment by default, which is why a procedure that works under Slurm silently degrades under mpirun. And rank-to-node mapping flags differ between distributions: verify the mapping that actually resulted rather than trusting -N.
NCCL_DEBUG_FILE writes on each host's own filesystem, so the logs from node-b are on node-b. Collect them before grepping.
If shape A is at baseline and shape B is not, the inter-node wire is fine and the fault is in how the ranks use it: GPU-to-NIC mapping, one degraded HCA or rail, ring and tree construction, CPU and NUMA affinity, multi-rail congestion, or algorithm selection.
Isolate one rail at a time by restricting NCCL to a single HCA and comparing across rails:
for hca in mlx5_0 mlx5_1 mlx5_2 mlx5_3; do
# The leading '=' forces an EXACT match. Unprefixed `mlx5_1` is a prefix selector and
# also picks up mlx5_10 and mlx5_11 on a node that has them, so the loop would not be
# isolating a rail at all.
mpirun -np 2 -H node-a:1,node-b:1 -x NCCL_IB_HCA="=$hca:1" \
./build/all_reduce_perf_mpi -b 1G -e 8G -f 2 -g 1 -w 5 -n 20 | tail -3
done
One rail well below its peers is a bad HCA, cable, port or leaf uplink, and it is invisible in an all-rails run that averages it away. Two caveats before you read the numbers. -g 1 uses whichever GPU rank 0 lands on, so an unpinned loop can be measuring the PCIe distance from that one GPU to each HCA rather than the health of each rail; bind the topology-local GPU for each HCA (nvidia-smi topo -m gives the pairing) and compare each rail against its own baseline, not against the others. NVIDIA ties the resulting per-rail asymmetry directly to the fabric diagnostic in rung 4b.
Then read what NCCL actually chose, rather than assuming it chose the fast thing:
Transport-string classification and the caveats around GPUDirect log lines are owned by the socket-fallback runbook. The one check to do here is for a forced fallback somebody left behind: NCCL_IB_DISABLE=1 pushes NCCL off InfiniBand and RoCE onto another transport, and it is a debugging lever that has a way of becoming permanent.
Rung 6. The real workload, last¶
If every rung above is at baseline and the application is still slow, the regression is above the fabric. Investigate rank-to-GPU assignment, CPU and NUMA binding, tensor-parallel degree, data and pipeline topology, gradient bucket or collective sizes, communication and computation overlap, dataloader starvation, checkpoint I/O, batch-shape imbalance, and CUDA synchronisation.
The trap at this rung: a collective's duration includes waiting for the slowest rank. NCCL can be working perfectly while one rank arrives late every step, and the symptom is indistinguishable from a slow fabric until you look at arrival times. On a live job, the NCCL RAS subsystem gives a global view of rank states cheaply; the NCCL-hang runbook documents the query and its version requirement.
Reading the per-pair matrix¶
Once a sweep produces a node-pair bandwidth matrix, the question is what it means. The model below is the rule, and it exists because the eyeball method gets three specific cases wrong. It was executed; the figures in the prose are its output.
# pair_localize.py -- validated: turn a node-pair busbw matrix into a blame verdict.
# The matrix is the artifact this runbook produces at rung 5; this is the rule that reads
# it. Three things the eyeball-the-heatmap method gets wrong and this does not:
# (1) a single global threshold on a fleet with more than one topology class condemns
# every cross-rack pair, because cross-rack is legitimately slower than intra-rack;
# (2) one bad leaf switch presents as every node on that leaf looking bad, which reads as
# N endpoint faults unless the shared boundary is tested for;
# (3) symmetrising an asymmetric pair still finds the node when the fault is severe, but
# reports the wrong SHAPE, and at moderate severity it hides the fault entirely;
# (4) the directional test has to require a CLEARLY healthy reverse direction, or one
# near-threshold measurement masks a real structural fault.
# Note the fabric here has THREE leaves. With exactly two, "leaf 0's uplink is bad" and
# "leaf 1's uplink is bad" produce an identical matrix and no rule can tell them apart.
# numpy only.
import numpy as np
def classify(bw, baseline, leaf, tol=0.85, clean_bar=0.95):
"""bw[i,j] = measured busbw GB/s for source i -> destination j, NaN on the diagonal.
baseline[i,j] = the recorded good value for that pair's topology class.
leaf[i] = leaf-switch id of node i.
Returns (verdict, detail, directional). Three rules the obvious version gets wrong:
an incomplete matrix must not read as clean; a pair that is degraded one way and only
mildly healthy the other is unresolved rather than silently ignored; and a boundary two
domains share cannot be attributed to either of them."""
n = bw.shape[0]
off = ~np.eye(n, dtype=bool)
# Refuse to grade an incomplete measurement. A NaN off the diagonal compares false
# against every threshold, so a missing pair would otherwise be reported as healthy.
missing = np.argwhere(off & ~np.isfinite(bw))
if missing.size:
return "incomplete", [(int(i), int(j)) for i, j in missing], []
slow = np.zeros_like(off)
slow[off] = bw[off] < baseline[off] * tol
clean = np.zeros_like(off)
clean[off] = bw[off] >= baseline[off] * clean_bar
# Directional: slow one way, CLEARLY healthy the other. Reported alongside the
# structural verdict, never instead of it, so noise cannot mask a shared-path fault.
directional = [(int(i), int(j)) for i, j in np.argwhere(slow & clean.T)]
# Slow one way and only middling the other is neither directional nor structural.
ambiguous = [(int(i), int(j)) for i, j in np.argwhere(slow & ~slow.T & ~clean.T)]
struct = slow & slow.T
if not struct.any():
if ambiguous:
return "unresolved", ambiguous, directional
return ("clean" if not directional else "directional-only"), [], directional
if struct[off].all():
return "fleet", [], directional
# Shared boundary. With more than two domains, "all traffic crossing leaf L is down and
# nothing avoiding L is" identifies L. With exactly two, both leaves satisfy it and the
# boundary is the inter-leaf path itself, which cannot be attributed to either switch.
hits = []
for lid in np.unique(leaf):
member = leaf == lid
crosses = off & (member[:, None] ^ member[None, :])
avoids = off & ~(member[:, None] | member[None, :])
if crosses.any() and struct[crosses].all() and not struct[avoids].any():
hits.append(int(lid))
if len(hits) == 1:
return "leaf", hits, directional
if len(hits) > 1:
return "inter-domain-boundary", hits, directional
bad = [i for i in range(n) if struct[i].sum() == n - 1]
if bad:
others = off.copy()
for i in bad:
others[i, :] = False
others[:, i] = False
if not struct[others].any():
return "endpoint", bad, directional
return "unresolved", [(int(i), int(j)) for i, j in np.argwhere(struct)], directional
def build(n, leaf, intra=48.0, cross=24.0):
"""A two-leaf fabric: pairs on one leaf run at `intra`, pairs across leaves at `cross`."""
same = leaf[:, None] == leaf[None, :]
base = np.where(same, intra, cross).astype(float)
np.fill_diagonal(base, np.nan)
return base.copy(), base.copy()
LEAF = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])
N = LEAF.size
# --- Baseline sanity: a healthy fleet is clean, and the two topology classes differ.
bw, base = build(N, LEAF)
assert classify(bw, base, LEAF) == ("clean", [], [])
print(f"0 healthy 9-node, 3-leaf fabric -> {classify(bw, base, LEAF)[0]} "
f"(intra {base[0,1]:.0f} GB/s, cross {base[0,4]:.0f} GB/s)")
# --- (1) The trap: a single global threshold. Take the healthy matrix above and judge every
# pair against the FASTEST class. Nothing is broken, and most of the fabric is condemned.
flat = np.full((N, N), 48.0)
np.fill_diagonal(flat, np.nan)
verdict, detail, _ = classify(bw, flat, LEAF)
cross_pairs = int((LEAF[:, None] != LEAF[None, :]).sum())
assert verdict == "unresolved" and len(detail) == cross_pairs == 54, (verdict, len(detail))
print(f"1 same healthy matrix, one global threshold -> {verdict!r} on {len(detail)} of "
f"{N * (N - 1)} pairs, every cross-leaf pair, with zero real faults")
# --- (2) One bad endpoint: node 5's HCA is degraded in both directions.
bw, base = build(N, LEAF)
bw[5, :] *= 0.30
bw[:, 5] *= 0.30
np.fill_diagonal(bw, np.nan)
assert classify(bw, base, LEAF) == ("endpoint", [5], []), classify(bw, base, LEAF)
print(f"2 node 5 HCA degraded -> {classify(bw, base, LEAF)[:2]}")
# --- (3) Two independent bad endpoints must stay two endpoints, not escalate to "fleet".
bw, base = build(N, LEAF)
for k in (1, 7):
bw[k, :] *= 0.30
bw[:, k] *= 0.30
np.fill_diagonal(bw, np.nan)
assert classify(bw, base, LEAF) == ("endpoint", [1, 7], []), classify(bw, base, LEAF)
print(f"3 two independent bad HCAs -> {classify(bw, base, LEAF)[:2]} (not 'fleet')")
# --- (4) Adversarial: one bad leaf, on a three-leaf fabric so the answer is evidence and
# not the lowest id. Traffic crossing leaf 1's boundary is down; traffic between leaves 0
# and 2, and traffic inside every leaf, is fine. Naive per-node counting blames nobody.
bw, base = build(N, LEAF)
member = LEAF == 1
crosses = (member[:, None] ^ member[None, :])
bw[crosses] *= 0.40
np.fill_diagonal(bw, np.nan)
assert classify(bw, base, LEAF) == ("leaf", [1], []), classify(bw, base, LEAF)
naive = [int((bw[i] < base[i] * 0.85).sum()) for i in range(N)]
print(f"4 leaf-1 uplink degraded -> {classify(bw, base, LEAF)[:2]}; naive per-node slow-peer "
f"counts {naive} rank six nodes equally and name no culprit")
# --- (5) Adversarial: directional. Node 2 transmits fine but receives at a third of rate.
bw, base = build(N, LEAF)
bw[:, 2] *= 0.33
np.fill_diagonal(bw, np.nan)
verdict, _, pairs = classify(bw, base, LEAF)
assert verdict == "directional-only" and all(j == 2 for _, j in pairs), (verdict, pairs)
print(f"5 node 2 RX-only degraded -> {verdict!r} on {len(pairs)} pairs, all into node 2")
# --- (6) What symmetrising costs, measured rather than asserted. It does NOT hide the
# fault: averaging the two directions still puts every pair into node 2 under tolerance, so
# the same rule still names node 2. What is lost is the SHAPE: an inbound-only fault is
# reported as a symmetric endpoint fault, which is the difference between suspecting one
# receive path and replacing the card.
sym = np.where(np.isnan(bw), np.nan, (bw + bw.T) / 2)
assert classify(sym, base, LEAF) == ("endpoint", [2], []), classify(sym, base, LEAF)
print(f"6 symmetrised, the same matrix reads {classify(sym, base, LEAF)[:2]}: right node, "
f"wrong shape; pair (0,2) shows {sym[0, 2]:.1f} GB/s against a true "
f"{bw[0, 2]:.1f} inbound and {bw[2, 0]:.1f} outbound")
# --- (7) Boundary: exactly at the tolerance. 85% of baseline must pass, a hair under fails.
bw, base = build(N, LEAF)
bw[3, 4] = bw[4, 3] = base[3, 4] * 0.85
np.fill_diagonal(bw, np.nan)
assert classify(bw, base, LEAF)[0] == "clean"
bw[3, 4] = bw[4, 3] = base[3, 4] * 0.85 - 1e-9
np.fill_diagonal(bw, np.nan)
assert classify(bw, base, LEAF)[0] == "unresolved"
print("7 tol boundary: exactly 0.85x baseline passes, 0.85x minus epsilon does not")
# --- (8) Adversarial: the directional test must not hijack the verdict. A real leaf fault
# plus ONE pair 1% under tolerance in a single direction. If the directional check fired on
# any asymmetry, the structural fault would be masked; requiring the reverse direction to be
# clearly healthy keeps it visible.
bw, base = build(N, LEAF)
bw[crosses] *= 0.40
bw[0, 1] = base[0, 1] * 0.84 # noisy, one direction, reverse still degraded-free
np.fill_diagonal(bw, np.nan)
verdict, detail, directional = classify(bw, base, LEAF)
assert (verdict, detail, directional) == ("leaf", [1], [(0, 1)]), (verdict, detail, directional)
print(f"8 real leaf fault plus one noisy directional pair -> still {verdict!r} {detail}, "
f"with the noisy pair reported alongside as {directional}")
# --- (9) Adversarial: symmetrising is worse than case 6 suggests. Case 6 used a severe
# 0.33 inbound path, where the average still fell under tolerance. At 0.75 inbound the
# average is 0.875, above the 0.85 bar, and the fault disappears completely.
bw, base = build(N, LEAF)
bw[:, 2] *= 0.75
np.fill_diagonal(bw, np.nan)
sym75 = np.where(np.isnan(bw), np.nan, (bw + bw.T) / 2)
assert classify(bw, base, LEAF)[0] == "directional-only", classify(bw, base, LEAF)[0]
assert classify(sym75, base, LEAF)[0] == "clean", classify(sym75, base, LEAF)
print(f"9 moderate 0.75 inbound fault: directional matrix -> "
f"{classify(bw, base, LEAF)[0]!r}, symmetrised -> {classify(sym75, base, LEAF)[0]!r} "
f"(pair (0,2) averages to {sym75[0,2] / base[0,2]:.3f} of baseline, over the 0.85 bar)")
# --- (10) Adversarial: an incomplete matrix must not read as clean. A NaN off the diagonal
# compares false against every threshold, so the naive rule reports a healthy fabric for a
# sweep that never measured node 5 at all.
bw, base = build(N, LEAF)
bw[5, :] = np.nan
bw[:, 5] = np.nan
np.fill_diagonal(bw, np.nan)
verdict, detail, _ = classify(bw, base, LEAF)
assert verdict == "incomplete" and len(detail) == 2 * (N - 1), (verdict, len(detail))
print(f"10 node 5 never measured -> {verdict!r} on {len(detail)} pairs, not 'clean'")
# --- (11) Adversarial: slow one way, only middling the other. It is neither a clean
# directional fault nor a symmetric one, and it must not fall through to 'clean'.
bw, base = build(N, LEAF)
bw[0, 1] = base[0, 1] * 0.40
bw[1, 0] = base[1, 0] * 0.90
np.fill_diagonal(bw, np.nan)
verdict, detail, directional = classify(bw, base, LEAF)
assert (verdict, detail, directional) == ("unresolved", [(0, 1)], []), (verdict, detail, directional)
print(f"11 pair 0.40 one way, 0.90 the other -> {verdict!r} {detail}, not silently clean")
# --- (12) Adversarial: exactly two leaves. "Leaf 0's uplink is bad" and "leaf 1's uplink is
# bad" produce an identical matrix, so attributing it to either is a coin flip. The rule has
# to name the boundary instead.
leaf2 = np.array([0, 0, 0, 0, 1, 1, 1, 1])
bw2, base2 = build(leaf2.size, leaf2)
cross2 = leaf2[:, None] != leaf2[None, :]
bw2[cross2] *= 0.40
np.fill_diagonal(bw2, np.nan)
verdict, detail, _ = classify(bw2, base2, leaf2)
assert (verdict, detail) == ("inter-domain-boundary", [0, 1]), (verdict, detail)
print(f"12 two-leaf fabric, inter-leaf path degraded -> {verdict!r} {detail}: both domains "
"satisfy the test, so neither switch can be named")
# --- (13) Fleet-wide: a driver or NCCL change that hurt every path. Must not be pinned on
# one endpoint or one switch.
bw, base = build(N, LEAF)
bw *= 0.50
np.fill_diagonal(bw, np.nan)
assert classify(bw, base, LEAF) == ("fleet", [], []), classify(bw, base, LEAF)
print(f"13 every pair at half baseline -> {classify(bw, base, LEAF)[0]!r} "
"(look at what changed on every node, not at the fabric)")
print("all assertions passed")
Executed output:
0 healthy 9-node, 3-leaf fabric -> clean (intra 48 GB/s, cross 24 GB/s)
1 same healthy matrix, one global threshold -> 'unresolved' on 54 of 72 pairs, every cross-leaf pair, with zero real faults
2 node 5 HCA degraded -> ('endpoint', [5])
3 two independent bad HCAs -> ('endpoint', [1, 7]) (not 'fleet')
4 leaf-1 uplink degraded -> ('leaf', [1]); naive per-node slow-peer counts [3, 3, 3, 6, 6, 6, 3, 3, 3] rank six nodes equally and name no culprit
5 node 2 RX-only degraded -> 'directional-only' on 8 pairs, all into node 2
6 symmetrised, the same matrix reads ('endpoint', [2]): right node, wrong shape; pair (0,2) shows 31.9 GB/s against a true 15.8 inbound and 48.0 outbound
7 tol boundary: exactly 0.85x baseline passes, 0.85x minus epsilon does not
8 real leaf fault plus one noisy directional pair -> still 'leaf' [1], with the noisy pair reported alongside as [(0, 1)]
9 moderate 0.75 inbound fault: directional matrix -> 'directional-only', symmetrised -> 'clean' (pair (0,2) averages to 0.875 of baseline, over the 0.85 bar)
10 node 5 never measured -> 'incomplete' on 16 pairs, not 'clean'
11 pair 0.40 one way, 0.90 the other -> 'unresolved' [(0, 1)], not silently clean
12 two-leaf fabric, inter-leaf path degraded -> 'inter-domain-boundary' [0, 1]: both domains satisfy the test, so neither switch can be named
13 every pair at half baseline -> 'fleet' (look at what changed on every node, not at the fabric)
all assertions passed
Case 1 is the one to internalise. It uses the healthy matrix from case 0, unchanged, judged against one global threshold instead of per-topology-class baselines, and it condemns 54 of 72 measured pairs on a fabric with nothing wrong with it. That is what a single number does to a fleet with more than one topology class, and it is why rung 0 insists the baseline is a sentence rather than a figure.
Case 4 is the case naive triage inverts. One degraded leaf uplink makes the six nodes on either side of that boundary look slow against the same number of peers, so per-node ranking names nobody. Testing the shared boundary names the switch.
Cases 5, 6 and 9 are a correction to the obvious intuition, arrived at in two stages. Symmetrising a directional fault does not necessarily hide it: at the severe 0.33 inbound rate of case 6 the average still falls under tolerance and the rule still names node 2, losing only the shape of the fault, which is the difference between suspecting one receive path and replacing the card. But case 9 is the one that matters operationally. At a moderate 0.75 inbound rate the average is 0.875 of baseline, above the 0.85 bar, and the fault disappears entirely: the directional matrix reports it, the symmetrised matrix reports clean. Recording a single number per pair costs you the repair on severe faults and the detection on moderate ones.
Case 8 is why directional pairs are reported alongside the structural verdict rather than instead of it. A real leaf fault plus one pair a percent under tolerance in one direction: if any asymmetry short-circuited the analysis, the structural fault would vanish behind measurement noise, and near-threshold noise is asymmetric by nature in a real sweep.
Cases 10 and 11 are the guards a sweep needs before it is allowed to say clean. A NaN off the diagonal compares false against every threshold, so an unmeasured node reads as healthy under the naive rule; the classifier now refuses to grade an incomplete matrix at all. And a pair degraded one way but only middling the other is neither a clean directional fault nor a symmetric one, so it is unresolved rather than falling through to clean.
Case 12 is the honesty case. On a two-leaf fabric, "leaf 0's uplink is degraded" and "leaf 1's uplink is degraded" produce an identical matrix. Both domains satisfy the shared-boundary test, so attributing it to either switch is a coin flip dressed as a diagnosis. The verdict is inter-domain-boundary, naming the path rather than a device. Three or more domains is what makes attribution possible, which is worth knowing before you build the sweep.
Cases 3 and 13 are the boundaries of escalation: two independent bad endpoints stay two endpoints, and a uniform halving across every path is a change common to all nodes rather than a fabric fault.
Verification¶
- The rung that was fixed is back at its baseline, measured with the identical command and profile that produced the baseline.
- Every rung above it re-run and at baseline. A fix at rung 3 does not entitle you to assume rung 5 recovered.
#wrongis zero on every collective run with-c 1. Performance work on a fabric returning wrong answers is misdirected work.- Error and retry counter deltas are zero across the verification run, not merely small. Traffic counters such as write, read and atomic request counts necessarily rise during a bandwidth test; it is the retry, sequence-error and link-error counters that must not move.
- The transport NCCL selected is the intended one, read from the log rather than inferred from the devices being present.
- The real workload recovered, in step time or tokens per second, not only the microbenchmark.
Rollback¶
If a change is what caused the regression, revert that change, singly, and re-measure the same rung:
# Resolve the node explicitly. An unset or inherited NODE targets the wrong machine.
NODE=gpu-node-17
kubectl get node "$NODE" -o wide # confirm it is the one you mean
kubectl get pods -A --field-selector spec.nodeName="$NODE" -o wide
kubectl get pdb -A # a PDB can block or stall the drain
# driver or node image: cordon, drain, revert the profile, re-qualify.
# --delete-emptydir-data DISCARDS emptyDir contents on this node. Confirm nothing above
# holds state you need there before running it.
kubectl cordon "$NODE"
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data
# NCCL or engine version: redeploy the previous pinned image digest
# NCCL environment: unset what was added, do not add a counter-tuning
Revert one variable at a time. A rollback that reverts a driver, a NCCL version and three environment variables together tells you the fleet is healthy again and nothing about which of them mattered, so the next upgrade repeats the incident.
Do not leave diagnostic settings behind. NCCL_DEBUG=INFO at scale is a real logging cost, NCCL_DEBUG=TRACE more so, and a forced transport set during triage is exactly the kind of residue this runbook exists to find.
Turning the ladder into a gate¶
Run the suite when commissioning nodes, after driver, CUDA, NCCL, NIC or switch firmware changes, after any recabling or topology change, on a rotating sample of the fleet, and immediately after a distributed-workload regression. Record per rung: median and p99 operation time, algbw and busbw, correctness failures, per-rank minimum and maximum, RDMA bandwidth and latency, counter deltas, the transport and algorithm NCCL selected, the topology profile, and every software and firmware version.
An example policy, which is a starting point for your fleet and not a hardware specification:
Correctness failures (#wrong): zero tolerance, block
Link width or rate downshift: block
Newly growing error counters: block
Transport fallback to sockets: block
Median busbw regression: warn at 5%, block on a confirmed 10%
p99 per-iteration regression: workload-specific gate
Unexplained topology change: manual review
Use repeated runs and a known-good control cohort before blocking on a small delta. A single result moves with thermal state, competing workloads and transient fabric traffic. Where a standing per-pair matrix already exists, consume it as the baseline source rather than rebuilding one (continuous NCCL fabric benchmarking).
Related runbooks¶
- NCCL hang / collective stall: progress stopped rather than slowed.
- NCCL socket fallback: the specific cause where the transport is wrong.
- PCIe / P2P bandwidth regression: rung 1 and 2 faults inside a node.
- NVLink visibility / P2P failure: peers not visible over NVLink.
- Fabric Manager failure: NVSwitch unconfigured under a healthy-looking node.
- Topology-unaware scheduling: the fabric is fine and the placement is not.
- Training MFU regression: where you came from if the profiler pointed at comms.
- operational runbooks: the runbook index.
References¶
- NVIDIA nccl-tests, argument parsing and usage text at the commit cited above: https://github.com/NVIDIA/nccl-tests/blob/717b68318278e93f371d8ffb46b076069d7c7851/src/common.cu
- NVIDIA nccl-tests, binary list and build variables at the same commit: https://github.com/NVIDIA/nccl-tests/blob/717b68318278e93f371d8ffb46b076069d7c7851/src/Makefile
- nccl-tests PERFORMANCE.md (algbw, busbw, per-collective correction factors): https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md
- NCCL environment variables (
NCCL_DEBUG,NCCL_DEBUG_SUBSYS,NCCL_DEBUG_FILE,NCCL_TOPO_DUMP_FILE,NCCL_IB_HCA,NCCL_IB_DISABLE): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html - NCCL troubleshooting, networking: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html
- NCCL troubleshooting, performance and tuning: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/performance_and_tuning.html
- NCCL troubleshooting, GPU issues: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/gpu_troubleshooting.html
- NVIDIA DCGM diagnostics (run levels and plugins): https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/dcgm-diagnostics.html
- linux-rdma perftest (
ib_write_bw,ib_write_lat, CUDA support): https://github.com/linux-rdma/perftest - iproute2 rdma utility (
rdma statistic show link): https://github.com/iproute2/iproute2
Related: Fabric bring-up and benchmarking · NCCL collectives and algorithms · Fabric validation recipe · Continuous fabric benchmarking · NCCL hang · NCCL socket fallback · RDMA and RoCE tuning · Diagnostics and validation · NUMA and CPU pinning · Operational runbooks · Glossary
-
NVIDIA NCCL networking troubleshooting: "When troubleshooting GPUDirect RDMA, it is often useful to compare host-memory results with GPU-memory results", followed by
ib_write_bw -d mlx5_0 --use_cuda=<gpu_id> <server_hostname_or_ip> -aand the DMA-BUF variant, and the caution to "Checkib_write_bw --helpon your system for the exact GPU-memory options supported by that build before comparing results." https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html ↩↩ -
NVIDIA NCCL networking troubleshooting, on fabric-wide diagnostics: "if NCCL works on some rails but repeatedly fails or slows down on one specific rail, run
ibdiagnetand inspect the generated report for missing links, bad ports, or routing inconsistencies affecting the switch ports connected to that rail before retrying the NCCL job." https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html ↩