Markdown

Runbook: NCCL socket fallback despite InfiniBand/RoCE present¶

Scope: diagnose a progressing NCCL workload that selected TCP sockets, or selected the built-in verbs transport without GPUDirect RDMA, when an InfiniBand or RoCE path is expected. This runbook separates transport selection, RDMA initialization failures, and GPUDirect eligibility because each class has different evidence and remediation.

Use this runbook when the job still makes progress but throughput or step time is materially worse than the recorded baseline. A collective that stops making progress belongs in the NCCL-hang runbook. Severity: job-degraded, not job-down.

Commands are reference templates that use documented interfaces. Replace the example node, pod, container, and HCA values, then validate each change in staging before production use.

Fabric background is in networking fabric and RDMA and RoCE performance tuning. Collective bandwidth metrics are in NCCL collectives and algorithms. Kubernetes device and network plumbing is covered by the NicClusterPolicy manifest. ACS diagnosis belongs to the PCIe/P2P bandwidth-regression runbook.

Trigger¶

The workload progresses, but throughput or step time is below the established baseline for the same nodes, rank count, message size, and NCCL test.
NCCL_DEBUG=INFO reports NET/Socket where the built-in verbs transport was expected.
An expected inter-node channel reports NET/IB without the /GDRDMA suffix. The built-in verbs transport uses the IB name for both InfiniBand and RoCE.
The symptom appeared after a node, image, workload manifest, BIOS, driver, NIC driver, or NCCL change.

CPU load, GPU utilization, and PCIe counters can corroborate a regression, but none of them identifies the selected NCCL transport by itself.

Pre-checks¶

Set the local variables once. The values below are examples:

export NODE="gpu-20"
export NAMESPACE="ml"
export POD="trainer-0"
export CONTAINER="trainer"

Confirm progress. A collective that never returns is a hang, even if the same network fault is suspected.
Confirm platform support. Check the current GPUDirect RDMA support matrix for the GPU, driver, CUDA runtime, kernel, and NIC path. NVIDIA's current DMA-BUF matrix includes supported Turing-or-newer data-center, Quadro RTX, and RTX GPUs; product branding alone is not a functional test.
Record the exact baseline. Capture rank count, hosts, message range, NCCL and nccl-tests revisions, and the large-message busbw result before changing the system.
Confirm that the host exposes an RDMA link:
```
ssh "$NODE" 'rdma link show'
```

Flow¶

flowchart TB
    A["Job progresses below its measured baseline"] --> B["Read NCCL INFO network lines"]
    B -->|"NET/Socket"| C["Built-in verbs transport was not selected"]
    B -->|"NET/IB without GDRDMA"| D["RDMA selected; direct GPU path not used"]
    B -->|"NET/IB with GDRDMA"| E["Transport is not the primary suspect"]
    C --> F["Check pod access, NCCL config, HCA filter, verbs runtime"]
    F --> G["Force NCCL_NET=IB for one diagnostic run"]
    G -->|"verbs or QP error"| H["Fix RDMA initialization; do not call it fallback"]
    D --> I["Check relevant ACS path, GDR level, DMA-BUF or peermem"]
    H --> J["Relaunch and measure"]
    I --> J
    E --> J
    J -->|"baseline restored"| K["Return to service"]
    J -->|"still degraded"| B

Procedure¶

Read the selected network from a short, reproducible run. This example uses the same two-node nccl-tests command used later for verification:

LAUNCH_CMD=(
  mpirun -np 16 -H gpu-20:8,gpu-21:8
  ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1
)

NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,NET \
  "${LAUNCH_CMD[@]}" 2>&1 | tee nccl-net.log

grep -E 'NET/(IB|Socket)|GDRDMA|Using network' nccl-net.log

Classify the result:

NET/Socket means NCCL selected its IP socket transport. Continue at step 5.
NET/IB/.../GDRDMA means the built-in verbs transport registered the direct GPU path for that channel. This string also applies to RoCE. If performance is still low, validate the link and baseline instead of changing GPUDirect settings.
NET/IB without /GDRDMA on an inter-node path that should support it means RDMA was selected but the direct GPU path was not used for that path. Continue at step 7.
ibv_create_qp, ibv_modify_qp, or ibv_reg_mr failures are RDMA initialization or registration errors. Continue at step 6.

GPU Direct RDMA Disabled is not a reliable INFO-level message in current NCCL source. A short TRACE capture may expose a reason on builds that include that trace point, but TRACE is high-volume and must not remain enabled for the workload:

NCCL_DEBUG=TRACE NCCL_DEBUG_SUBSYS=GRAPH,NET \
  "${LAUNCH_CMD[@]}" 2>&1 | tee nccl-net-trace.log

grep -E 'GPU Direct RDMA Disabled|NET/(IB|Socket)|GDRDMA' nccl-net-trace.log

Compare busbw with the recorded baseline. For AllReduce, nccl-tests computes busbw = algbw * 2 * (n - 1) / n. Hardware peak is a ceiling; the expected result depends on topology, rank placement, protocol, message size, PCIe, NVLink, and fabric configuration.
```
mpirun -np 16 -H gpu-20:8,gpu-21:8 \
  ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1
```
Compare the largest common message sizes with a known-good run that used the same command. Do not diagnose socket fallback from a generic GB/s threshold.
Collect GPU and host counters as supporting evidence. nvidia-smi dmon has no host CPU column, so pair it with a host monitor:
```
ssh "$NODE" 'nvidia-smi dmon -s put -d 1'
ssh "$NODE" 'mpstat -P ALL 1'
```
The t group reports PCIe receive and transmit throughput where supported. A dash means the metric is unsupported. MIG, virtualized, and non-PCIe platforms can limit these counters. Compare the same benchmark before and after remediation; high CPU load or low SM utilization alone does not prove that NCCL selected sockets.
Validate the fabric according to its link layer.

For InfiniBand, require physical LinkUp and logical Active:

ssh "$NODE" 'ibstat'
ssh "$NODE" 'if command -v sminfo >/dev/null 2>&1; then sminfo; fi'

Physical Polling or another non-LinkUp state points to link, cable, peer, or port configuration. Physical LinkUp with logical Init or Armed points to subnet-manager or fabric initialization. Route either case to fabric bring-up and benchmarking.

RoCE has no InfiniBand subnet manager. Check the RDMA link, Ethernet carrier, addresses, routing, VLAN, and lossless-fabric configuration:

ssh "$NODE" 'rdma link show'
ssh "$NODE" 'ip -brief link show; ip -brief address show'

If NCCL selected sockets, determine why the verbs transport was unavailable.
Inspect the workload's actual container. Kubernetes needs verbs-device access plus a reachable RDMA-capable interface in the pod network namespace. The interface may come from a secondary CNI, the primary CNI, or hostNetwork; a secondary attachment is not universal. InfiniBand and RoCE verbs devices normally appear under /dev/infiniband.
```
kubectl get pod -n "$NAMESPACE" "$POD" \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.limits}{"\n"}{end}'

kubectl exec -n "$NAMESPACE" "$POD" -c "$CONTAINER" -- \
  sh -c 'ls -l /dev/infiniband; rdma link show; ip -brief address show'
```
An RDMA device-plugin resource is a common way to grant device access, but its resource name and the network attachment model are cluster-specific.
Inspect every NCCL configuration source. NCCL reads the process environment, /etc/nccl.conf, and the file named by ${NCCL_CONF_FILE}:
```
kubectl exec -n "$NAMESPACE" "$POD" -c "$CONTAINER" -- sh -c '
env | grep "^NCCL_" || true
for file in /etc/nccl.conf "${NCCL_CONF_FILE:-}"; do
  if [ -n "$file" ] && [ -r "$file" ]; then
    echo "### $file"
    sed -n "1,200p" "$file"
  fi
done
'
```
Remove an unintended NCCL_IB_DISABLE=1. Check NCCL_IB_HCA against the devices and ports visible inside the same container. Use an exact selector such as =mlx5_0:1 when exact matching is intended; an unprefixed mlx5_1 also matches names that begin with that string.

Separate utilities, devices, and the runtime library. Missing ibv_devices proves only that the utility is absent. NCCL loads libibverbs.so or libibverbs.so.1 dynamically:

kubectl exec -n "$NAMESPACE" "$POD" -c "$CONTAINER" -- sh -c '
command -v ibv_devices || true
command -v ibv_devinfo || true
if command -v ibv_devices >/dev/null 2>&1; then
  ibv_devices
fi
if command -v ldconfig >/dev/null 2>&1; then
  ldconfig -p | grep "libibverbs\.so" || true
fi
'

If the image lacks ldconfig, inspect its package manifest or dynamic-loader paths instead of inferring library absence from the missing CLI.

Force the verbs transport for one diagnostic test. NCCL_NET=IB makes missing or broken verbs support fail closed instead of allowing a socket result:
```
NCCL_NET=IB NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,NET \
  mpirun -np 16 -H gpu-20:8,gpu-21:8 \
  ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1
```
Remove NCCL_NET=IB after the test. Its failure text determines whether step 6 applies.
Treat RDMA initialization and registration errors as errors, not silent socket fallback.
ibv_reg_mr or ibv_create_qp with a memory-lock warning requires an adequate memlock limit. NVIDIA's Kubernetes transfer example adds IPC_LOCK; the applicable container runtime and security policy determine how the limit is supplied:
```
kubectl exec -n "$NAMESPACE" "$POD" -c "$CONTAINER" -- \
  sh -c 'ulimit -l'

kubectl get pod -n "$NAMESPACE" "$POD" \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.securityContext.capabilities}{"\n"}{end}'
```
Current NCCL propagates registration failure after selecting the RDMA network. Correct the limit and rerun the forced-verbs test; do not describe this path as automatic socket reselection.
On RoCE, investigate GID selection only when logs report ibv_modify_qp Invalid argument, address-family problems, or failed RoCE connectivity:
```
ssh "$NODE" 'show_gids 2>/dev/null || rdma link show'
kubectl exec -n "$NAMESPACE" "$POD" -c "$CONTAINER" -- \
  sh -c 'show_gids 2>/dev/null || rdma link show'
```
Different host and container tables are not proof of a fault. NCCL 2.21 and later dynamically select a GID and advise against setting NCCL_IB_GID_INDEX. Remove a stale manual index unless it was validated for the deployed NCCL and network. A wrong GID causes QP or connectivity failure; it does not convert GPUDirect RDMA into a host-staged RDMA copy.
If RDMA was selected without the expected /GDRDMA suffix, validate the direct GPU path.
Inspect ACS only on the relevant GPU-to-NIC PCIe path. First identify that path, then inspect its bridges:
```
ssh "$NODE" 'nvidia-smi topo -m'
ssh "$NODE" \
  "sudo lspci -vvv | grep -E 'ReqRedir\\+|CmpltRedir\\+|UpstreamFwd\\+|SrcValid\\+'"
```
On bare metal, redirect-enabled ACS on a relevant PCIe switch can redirect peer traffic through the root complex, reducing performance or causing a hang. Follow the PCIe/P2P bandwidth-regression runbook before changing it. Virtual machines require ACS isolation and a working ATS configuration; do not disable ACS in a guest or hypervisor from this runbook.
Remove an over-restrictive NCCL_NET_GDR_LEVEL. The documented distances are LOC, PIX, PXB, PHB, and SYS. Leave the variable unset for automatic selection unless a topology-specific override has been measured and recorded.
Identify the configured GPUDirect mechanism. NVIDIA recommends DMA-BUF when its prerequisites are met. The legacy path uses nvidia_peermem:
```
ssh "$NODE" 'uname -r'
ssh "$NODE" 'nvidia-smi --query-gpu=name,driver_version --format=csv,noheader'
ssh "$NODE" 'lsmod | grep "^nvidia_peermem " || true'
```
Absence of nvidia_peermem is expected on a DMA-BUF deployment. Check the current support matrix rather than treating module absence as failure.

Run a direct GPU RDMA transfer test. The commands below follow NVIDIA's DMA-BUF verification procedure and assume two diagnostic pods whose image and perftest version are pinned in their manifests.

Terminal 1, server:

NAMESPACE="default"
SERVER_POD="rdma-gpu-test-1"
HCA="mlx5_0"

kubectl exec -n "$NAMESPACE" "$SERVER_POD" -- \
  ib_write_bw --use_cuda=0 --use_cuda_dmabuf \
  -d "$HCA" -a -F --report_gbits -q 1

Terminal 2, client:

NAMESPACE="default"
SERVER_POD="rdma-gpu-test-1"
CLIENT_POD="rdma-gpu-test-2"
HCA="mlx5_0"
SERVER_IP="$(
  kubectl get pod -n "$NAMESPACE" "$SERVER_POD" \
    -o jsonpath='{.status.podIP}'
)"

kubectl exec -n "$NAMESPACE" "$CLIENT_POD" -- \
  ib_write_bw -n 5000 --use_cuda=0 --use_cuda_dmabuf \
  -d "$HCA" -a -F --report_gbits -q 1 "$SERVER_IP"

A successful transfer validates the CUDA-to-RDMA data path independently of NCCL. Compare its bandwidth with a known-good result for the same HCA, port, MTU, and topology.

Check version alignment against the supported stack. Compare NCCL, CUDA/driver, kernel, NIC driver, firmware, and container image revisions across the affected ranks. A version difference is evidence to investigate, not proof that GPUDirect was disabled.
Apply one controlled fix and relaunch. Save the currently deployed manifest, image digest, NCCL configuration, security context, and platform setting before changing them. Confirm the result with the same INFO log and the same benchmark; a configuration edit alone is not verification.

Verification¶

The expected inter-node paths report NET/IB/.../GDRDMA; a RoCE path still uses the NET/IB transport name.
The large-message busbw result returns to the recorded baseline for the same topology, hosts, rank count, message range, NCCL, and nccl-tests revisions:
```
mpirun -np 16 -H gpu-20:8,gpu-21:8 \
  ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1
```
If the direct GPU path changed, the pinned ib_write_bw --use_cuda_dmabuf test succeeds and returns to its topology-specific baseline.
Supporting dmon and host CPU counters no longer show the incident's measured regression. They remain corroborating evidence, not the pass criterion.
The workload's step time or throughput returns to its established baseline.
After verification, commit the durable fix to the launch template, manifest, image, or node configuration. Remove temporary TRACE, NCCL_NET=IB, and one-off diagnostic overrides.

Rollback¶

Revert the exact manifest, image digest, NCCL configuration, security context, driver/NIC-driver bundle, or node setting changed during the procedure. Use the saved pre-change state or the deployment system's versioned rollback.
Remove temporary NCCL_NET=IB, TRACE logging, HCA filters, GID variables, and GDR-level overrides introduced for diagnosis.
If ACS was changed, restore the platform's previous setting and service state. Do not apply a bare-metal ACS policy to a virtualized host.
Relaunch the same benchmark after rollback. If the prior baseline does not return, drain the node and route the incident to fabric bring-up and benchmarking or the platform owner.
Keep the node out of the workload pool when the known-good state cannot be restored. A socket transport that lets the job progress is not proof that the node is healthy.

NCCL hang: the collective stops making progress or times out.
PCIe/P2P bandwidth regression: inspect the relevant ACS and PCIe path.
Fabric Manager failure: diagnose a degraded NVLink/NVSwitch path.
GPU fault and RMA: route fatal XIDs and hardware faults.
Operational runbooks: browse the runbook index.

References¶

NCCL environment variables (NCCL_NET, NCCL_IB_HCA, NCCL_IB_DISABLE, NCCL_NET_GDR_LEVEL, debug settings, and configuration files): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html
NCCL networking troubleshooting (InfiniBand states, memory locking, RoCE GID selection): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html
NCCL GPU troubleshooting (GPU-to-NIC DMA-BUF/peermem and ACS): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/gpu_troubleshooting.html
NCCL source, network channel logging and /GDRDMA suffix: https://github.com/NVIDIA/nccl/blob/5067397c2676d5aed50042fc39e5c8ee96eb0027/src/transport/net.cc#L323-L330
NCCL source, dynamic libibverbs loading: https://github.com/NVIDIA/nccl/blob/5067397c2676d5aed50042fc39e5c8ee96eb0027/src/misc/ibvsymbols.cc#L74-L115
nccl-tests performance metrics (algbw, busbw, and collective correction factors): https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md
NVIDIA GPU Operator, GPUDirect RDMA support and DMA-BUF transfer verification: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html
nvidia-smi manual (dmon metric groups and limitations): https://docs.nvidia.com/deploy/nvidia-smi/index.html
RDMA shared-device plugin examples (device resources and pod networking): https://github.com/Mellanox/k8s-rdma-shared-dev-plugin
rdma-core libibverbs documentation: https://github.com/linux-rdma/rdma-core/blob/master/Documentation/libibverbs.md