Skip to content
Markdown

Runbook: NCCL socket fallback despite InfiniBand/RoCE present

Scope: diagnose and fix NCCL choosing TCP sockets, or running IB/RoCE without GPUDirect RDMA, on a node where RDMA hardware is present and expected to be used. Distinct from a full hang: the job keeps running, just one to two orders of magnitude slower, with no crash and no XID.

Run this when a job is progressing but far slower than its established baseline, nccl-tests busbw sits well under the fabric's line rate, and there is no collective timeout. Severity: job-degraded, not job-down. Because nothing crashes and nothing times out, this fails silently: a training run can burn a full day at a fraction of expected throughput before anyone notices. If step time has actually gone to infinity and a collective never returns, this is the wrong page; go to the NCCL-hang runbook.

Reference templates on real APIs; pin versions and validate before production use. Nothing here was hardware-tested.

Fabric background is in networking fabric and RDMA and RoCE performance tuning; the collective algorithms and the busbw/algbw numbers are in NCCL collectives & algorithms; the Kubernetes device/network plumbing a container needs for RDMA is in the NicClusterPolicy manifest; the live monitoring tools are in diagnostics tools and nvidia-smi reference; an ACS misconfiguration blocking the GPUDirect path is the PCIe/P2P bandwidth-regression runbook.

Trigger

  • Throughput or step time far below the established baseline, but the job is progressing: no exit, no XID, no collective timeout.
  • nccl-tests busbw sits well under the topology's line rate (fabric bring-up and benchmarking, recipe: fabric validation) for a link that should carry tens of GB/s (IB/RoCE) or more.
  • NCCL_DEBUG=INFO shows NET/Socket where an IB/RoCE transport was expected, or shows an IB/RoCE transport with no GDRDMA tag on the channel.
  • GPU sm utilisation is low relative to a compute/comm-bound step while a host CPU core is pegged during the collective: the classic fingerprint of NCCL staging data through the CPU instead of DMA'ing GPU-to-GPU.1
  • Symptom appears freshly after: a new node or container image, a Kubernetes manifest change, a driver/OFED/NCCL upgrade, or ACS re-enabled by a BIOS reset.

Pre-checks

  • Confirm it is a fallback, not a hang. The job must be making progress, just slowly. A full stall (step time to infinity, collective never returns) is the NCCL-hang runbook, not this one.
  • Confirm the hardware is actually RDMA-capable on this node. GPUDirect RDMA needs a Tesla/data-center-class GPU and an RDMA-capable NIC; a workstation/consumer card with no such NIC path is expected to run over sockets, not a fallback to diagnose.
  • Confirm the IB/RoCE port is Active, not just cabled: 
    ssh "$NODE" 'ibstat | grep -A2 "State:"'
  • Record the established busbw baseline for this exact topology and message size before changing anything; without a recorded number, "slow" is a guess (fabric bring-up and benchmarking, MFU regression runbook).

Flow

flowchart TB
    A["Job runs but busbw far below baseline, no hang"] --> B["NCCL_DEBUG=INFO: read the NET line"]
    B -->|"NET/Socket"| C["No RDMA device usable at all"]
    B -->|"NET/IB or NET/RoCE, no GDRDMA"| D["RDMA transport up, GPU-direct disabled"]
    C --> E["Check container device access, HCA filter, IB_DISABLE, verbs lib, subnet manager"]
    D --> F["Check ACS, GID/container mismatch, GDR level, nvidia_peermem or DMA-BUF"]
    E --> G["Fix + relaunch"]
    F --> G
    G --> H["Verify: nccl-tests busbw + dmon"]
    H -->|"pass"| I["Return to service"]
    H -->|"still degraded"| B

Procedure

NODES="gpu-20 gpu-21"   # ranks in the degraded job
  1. Read which transport NCCL is actually using right now. Relaunch (or tail live logs from) the job with debug scoped to init and net, and grep the channel lines: 
    NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,NET <launch cmd> 2>&1 | grep -E "NET/IB|NET/Socket|GDRDMA|GPU Direct RDMA Disabled"
    Three outcomes, each pointing at a different branch below:
  2. Channel 00 : 1[...] -> 0[...] [send] via NET/IB/0/GDRDMA (or the RoCE equivalent): the fast path is engaged. If busbw is still low, the transport is not the problem; re-check the baseline and message size, or suspect a degraded link (fabric bring-up and benchmarking).
  3. NET/Socket (or using Ethernet interface eth0 with no IB/RoCE line at all): no RDMA device is usable by this process at all. Go to step 5.

  4. An IB/RoCE line with no GDRDMA suffix, or an explicit GPU Direct RDMA Disabled for HCA ... message: the RDMA transport itself came up, but GPUDirect is disabled, so the NIC and GPU are staging through host memory instead of DMA'ing directly. Go to step 6.

  5. Read busbw against the topology's line rate, not algbw. busbw applies the collective's correction factor (for AllReduce, algbw * 2*(n-1)/n) so the figure is comparable to hardware peak independent of rank count.2 

    mpirun -np 16 -H gpu-20:8,gpu-21:8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1
    Read the tail (largest message size). As a rough triage signature, not a hard threshold: a full socket fallback typically lands in the low single-digit GB/s or below (the book's worked example: a 400 MB all-reduce on 800 Gb/s / 100 GB/s hardware runs at roughly 2 GB/s over a CPU-bound Gloo-style path versus roughly 100 GB/s, line rate, with GPUDirect RDMA engaged);1 a GDR-disabled path (RDMA up, host-staged) usually sits meaningfully below line rate but above that socket-fallback floor, since it still moves data over the RDMA fabric, just via an extra host-memory copy. Treat both as directional; record and use your own measured fleet baseline rather than a borrowed number (recipe: fabric validation).

  6. Corroborate with nvidia-smi dmon, paired with a host CPU monitor. dmon reports the GPU side only (SM utilisation, PCIe Rx/Tx throughput); it has no host-CPU column, so pair it with a host tool to see the full fallback signature:3 

    ssh "$NODE" 'nvidia-smi dmon -s put -d 1'      # power/temp, utilisation (incl. sm%), PCIe throughput, 1s
ssh "$NODE" 'mpstat -P ALL 1'                   # per-core host CPU, alongside dmon
    During a healthy GPUDirect collective, sm tracks the compute/comm-bound step and t (PCIe Rx/Tx, MB/s) reflects real DMA traffic. In a socket (or GDR-disabled) fallback, sm drops or flatlines low relative to that baseline while one or more host cores in mpstat sit pegged near 100%, the NCCL socket transport doing the copy on the CPU instead of the GPU DMA'ing directly.1

  7. Confirm the port and subnet manager are actually healthy, since a down port or an unconverged fabric can itself present as "no usable IB device" (step 1's NET/Socket branch): 

    ssh "$NODE" 'ibstat | grep -E "State:|Physical state:"'   # expect Active / LinkUp
ssh "$NODE" 'echo "verbose status" | nc localhost 28028'  # NCCL RAS, if enabled (see the NCCL-hang runbook)
    Down or Polling physical state means no subnet manager has converged on this port; that is a fabric fault, not a container/config problem, escalate to fabric bring-up and benchmarking rather than continuing here.

  8. Work the "no RDMA device usable" checklist (step 1 showed NET/Socket, no IB/RoCE line at all):

  9. Container/pod has no device access. The process needs /dev/infiniband (or the RoCE equivalent) inside the container. On Kubernetes this needs both the RDMA device-plugin resource and the secondary network attachment; either alone is not enough, the resource without an attached interface yields no verbs handle with an IP, the interface without the resource yields no verbs handle at all (the NicClusterPolicy manifest): 
    kubectl exec "$POD" -- ls /dev/infiniband       # expect uverbsN, rdma_cm
kubectl get pod "$POD" -o jsonpath='{.spec.containers[0].resources}'  # rdma/<name> present?
kubectl get pod "$POD" -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/networks}'
  10. NCCL_IB_DISABLE=1 set somewhere in the job's env (often copied in from a laptop/dev config and never removed). Grep the launch environment for it explicitly.
  11. NCCL_IB_HCA filters out every real device. Compare the filter against what the host/container actually exposes: 
    ssh "$NODE" 'ibv_devices'          # real device names as the verbs layer sees them
echo "$NCCL_IB_HCA"                # must match, or be unset to let NCCL pick
  12. Verbs libraries missing from the container image. ibv_devices/ibv_devinfo failing or absent (not just empty) means libibverbs/rdma-core was never installed in the image; NCCL's IB plugin cannot even attempt the fast path.
  13. IPC_LOCK capability or memlock ulimit missing. Verbs need to pin memory; without IPC_LOCK (Kubernetes) or an unlimited memlock (bare container), registration fails and NCCL falls back: 
    kubectl get pod "$POD" -o jsonpath='{.spec.containers[0].securityContext.capabilities}'
ssh "$NODE" 'docker exec <container> bash -c "ulimit -l"'   # expect unlimited, not a small default

  14. Port down / subnet manager not converged: covered in step 4 above; this is a fabric fault, not a config fix.

  15. Work the "GPUDirect disabled" checklist (step 1 showed an IB/RoCE line with no GDRDMA):

  16. ACS P2P Request Redirect on the NIC-to-GPU PCIe path forces the transfer up through the Root Complex instead of straight across, blocking GPUDirect entirely even though the RDMA transport itself is healthy. Reuse the redirect-bit check and fix from the PCIe/P2P bandwidth-regression runbook: 
    ssh "$NODE" "sudo lspci -vvv 2>/dev/null | grep -E 'ReqRedir\+|CmpltRedir\+|UpstreamFwd\+|SrcValid\+'"
    Any bridge printing a + flag needs the ACS-disable service.
  17. GID mismatch between container and host (RoCE). A container that sees a different GID table than the host blocks GPUDirect registration and forces a CPU-driven RDMA copy instead of true GPU RDMA:4 
    ssh "$NODE" 'show_gids'                       # host view
kubectl exec "$POD" -- show_gids               # container view; compare
  18. NCCL_NET_GDR_LEVEL set too restrictive for the actual NIC-to-GPU PCIe distance on this node. Values, increasing permitted distance: LOC, PIX, PXB, PHB, SYS. Leave it auto-selected unless you have measured the correct value for this topology; a hand-set PIX on a node where the NIC and GPU sit two PCIe switches apart disables GDR outright.
  19. nvidia_peermem not loaded (legacy path) or the DMA-BUF prerequisites unmet (Open Kernel Module driver, CUDA >= 11.7, Turing-or-newer data-center GPU, kernel >= 5.12):5 
    ssh "$NODE" 'lsmod | grep nvidia_peermem'
ssh "$NODE" 'dmesg | grep -i nvidia_peermem'

  20. NCCL/OFED/driver version mismatch after an upgrade on only some nodes. Compare across the job's node set; a stale OFED on one rank can silently disable GDR for the pair even when the rest of the fleet is fine.

  21. Fix, then relaunch the job. Do not declare it fixed on the config change alone; every fix here is confirmed by the same transport line and a re-measured busbw in Verification below.

Verification

  • NCCL_DEBUG=INFO now shows NET/IB/.../GDRDMA (or the intended RoCE equivalent) on every channel, not NET/Socket and not an IB/RoCE line missing GDRDMA.
  • nccl-tests busbw at the large-message tail returns to the fleet's recorded baseline for this topology, not just "better than before": 
    mpirun -np 16 -H gpu-20:8,gpu-21:8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1
  • nvidia-smi dmon shows the healthy pattern: sm utilisation consistent with a compute/comm-bound step and PCIe t throughput consistent with real DMA traffic, with the host CPU (mpstat) no longer pegged during the collective.
  • The job's step time returns to its established baseline (observability).

Rollback

A silent fallback is not a change to revert; recovery is making sure the fix sticks and the fast path stays proven:

  • Bake the fix into the launch template, manifest, or image, not a one-off env override on this run. A NCCL_IB_HCA correction, an IPC_LOCK capability, or a NicClusterPolicy fix that only lives in this job's command line regresses on the very next launch (SRE and MLOps practices).
  • If an ACS fix was applied, confirm it is the managed boot-time service, not a manual setpci, so it survives the next reboot (the ACS-disable service).
  • If the root cause was a subnet manager or port fault (step 4), that is a fabric issue, not a job config issue; hand off to fabric bring-up and benchmarking and keep the node out of the pool until it is proven, not just "probably fine now."
  • If nothing here resolves it and busbw stays low with a clean transport line, the fabric itself may be degraded (a bad cable, an asymmetric link); escalate to fabric bring-up and benchmarking rather than continuing to iterate on NCCL env vars.

References

  • NCCL environment variables (NCCL_IB_HCA, NCCL_IB_DISABLE, NCCL_NET_GDR_LEVEL, NCCL_DEBUG): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html
  • NCCL networking troubleshooting (transports, GDR, fallback): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html
  • nccl-tests performance metrics (algbw/busbw, AllReduce correction factor): https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md
  • nvidia-smi manual (dmon metric groups, sm and PCIe-throughput column semantics): https://docs.nvidia.com/deploy/nvidia-smi/index.html
  • NVIDIA GPU Operator, GPUDirect RDMA (DMA-BUF vs legacy nvidia_peermem): https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html
  • Mellanox/NVIDIA k8s-rdma-shared-dev-plugin (RDMA device-plugin resources): https://github.com/Mellanox/k8s-rdma-shared-dev-plugin
  • PCI-SIG ACS Engineering Change Notice (P2P Request Redirect): https://pdos.csail.mit.edu/~sbw/links/ECN_access_control_061011.pdf
  • Chris Fregly, AI Systems Performance Engineering (O'Reilly), Ch. 4, "Tuning Distributed Networking Communication": the silent socket-fallback behaviour without container access to /dev/infiniband, the GID-mismatch trap, and the CPU-spike/GPU-idle fallback signature.

Related: NCCL Hang / Collective Stall · RDMA and RoCE Performance Tuning · NCCL Collectives & Algorithms · Fabric Bring-Up, Validation and Benchmarking · Diagnostics Tools · nvidia-smi Reference · PCIe / P2P Bandwidth Regression · Manifest: NicClusterPolicy · Operational Runbooks · Glossary

  1. Fregly, Ch. 4: without container access to /dev/infiniband, NCCL silently falls back to TCP, throughput drops from tens of GB/s to a few Gb/s with no error message; a red flag during a collective is GPU utilisation dropping while CPU utilisation spikes, since the CPU is copying data for communications. The 400 MB / 800 Gb/s worked example: a CPU-bound Gloo-style path runs approximately 200 ms (approximately 2 GB/s, CPU near 100%) versus approximately 4 ms (approximately 100 GB/s, line rate) with GPUDirect RDMA engaged. Illustrative figures from the book, not hardware-tested here. ↩↩↩

  2. nccl-tests PERFORMANCE.md: algbw = size/time; for AllReduce, busbw = algbw * 2*(n-1)/n; busbw reflects hardware utilisation independent of rank count, for comparison against hardware peak. https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md ↩

  3. nvidia-smi manual: dmon metric groups selected with -s; u reports utilisation including sm (percentage of time at least one SM was busy); t reports PCIe Rx/Tx throughput in MB/s. dmon has no host-CPU column; pair with a host tool (mpstat, top) to see host-side load. https://docs.nvidia.com/deploy/nvidia-smi/index.html ↩

  4. Fregly, Ch. 4: mismatched container/host GID assignments block GPUDirect registration and force CPU-driven RDMA copies instead of true GPU RDMA. ↩

  5. NVIDIA GPU Operator, GPUDirect RDMA: DMA-BUF is the current recommended path (Open Kernel Module driver, CUDA >= 11.7, Turing-or-newer data-center/RTX/Quadro-RTX GPU, Linux kernel >= 5.12) and the helm-install default; the legacy nvidia_peermem kernel module needs --set driver.rdma.enabled=true explicitly. https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html ↩