Skip to content
Markdown

SHARP: in-network reduction

Scope: NVIDIA SHARP offloading all-reduce / reduce-scatter / all-gather into the InfiniBand switch ASIC (and NVLink SHARP / NVLS in the NVSwitch) to halve endpoint data movement and free GPU SMs: what it is, when it pays off, and the exact NCCL flags and fabric services that turn it on.

What it is

SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) performs the arithmetic of a collective inside the network silicon instead of on the GPUs. As partial results from multiple endpoints flow into a switch, the switch reduces (e.g. sums) them and returns only the aggregated result, so each GPU stops shuttling intermediate buffers around the fabric.115

Two distinct hardware paths share the SHARP name:

  • IB SHARP: switch-resident reduction inside Quantum-class InfiniBand switches. Reached from NCCL through the CollNet algorithm provided by the nccl-rdma-sharp-plugins package.111
  • NVLink SHARP (NVLS): the analogous capability inside the NVSwitch fabric of an NVLink domain (e.g. a 72-GPU GB200/GB300 NVL72 rack). NVLS accelerates collectives and enables efficient all-to-all and broadcast across the domain, reached through the NVLS / NVLSTree algorithms.1216

SHARPv1 shipped on EDR switches; SHARPv2 added streaming aggregation for large-vector reductions at line rate, 16/32/64-bit integer and floating-point ops, and GPUDirect RDMA.15 On the fabric side, a SHARP Aggregation Manager (sharp_am) runs on a management server alongside the Subnet Manager and builds the in-network reduction (aggregation) trees that the switches execute.314

flowchart LR
  G0["GPU 0"] --> SW["IB switch ASIC<br/>(SHARP reduction engine)"]
  G1["GPU 1"] --> SW
  G2["GPU 2"] --> SW
  G3["GPU 3"] --> SW
  SW -->|"reduce-scatter case: B/n slice"| G0
  SW -->|"reduce-scatter case: B/n slice"| G1
  SW -->|"reduce-scatter case: B/n slice"| G2
  SW -->|"reduce-scatter case: B/n slice"| G3
  AM["sharp_am + Subnet Manager<br/>(management host)"] -.->|"builds reduction tree"| SW

The diagram illustrates the reduce-scatter case specifically, where each GPU's return is a distinct B/n slice of the reduced total (see "Why it matters" below); a true all-reduce instead returns the full, identical B-sized reduced result to every GPU.

Why it matters

The win is reduced data movement per endpoint, not just faster links. For a ring reduce-scatter, each GPU normally receives B*(n-1)/n bytes across n-1 hops; with in-network reduction the switch aggregates and returns only B/n to each GPU, roughly a 1/(n-1) per-endpoint receive versus full-ring.1 For all-gather, NVLS hardware multicast lets each GPU send its B/n segment once while the network replicates it, cutting the sender's volume by 1/(n-1) versus the full ring.1 Overlapping a multicast all-gather with an in-network reduce-scatter can drop the bandwidth-bound phase time by ~1/2, since wall time becomes the max of the two operations instead of the sum.1

All-gather has no arithmetic reduction, so its NVLS benefit comes purely from multicast replication and is smaller than the gains for all-reduce and reduce-scatter.1

The effect is to move reduction off the GPU SMs and onto the switch. NCCL 2.27 extended SHARP to AllGather and ReduceScatter across both NVLink and IB fabrics, cutting the SM footprint of those collectives from 16 SMs to 6 or fewer per GPU.17 NVIDIA reports 2x–5x speedups for large-scale all-reduce with SHARP.110

Gains scale with the cluster. On two-to-four-node jobs the improvement may be marginal; at 32 nodes and beyond, where the network is the bottleneck, SHARP cuts the number of serialized communication steps and the latency falls substantially.1

When it is needed (and when not)

Needed:

  • Large-scale all-reduce / reduce-scatter on a Quantum-class InfiniBand fabric where the network, not the GPU, bounds the collective.1
  • NVLink-domain collectives on NVSwitch (NVL72 and similar) where NVLS offload and multicast apply.116

Not worth it (or unavailable):

  • Ethernet / RoCE fabrics. As of writing SHARP is primarily an InfiniBand technology. Spectrum-X improves all-reduce with congestion control and adaptive routing but does not expose switch-resident reduction engines analogous to SHARP.15
  • Small clusters (two-to-four nodes), where the network is not the bottleneck and the offload barely registers.1
  • Very large messages that exceed switch buffer limits. Switch reduction buffers are finite; collectives in the many-MB/GB range may fall back to regular (non-SHARP) methods. Monitor NCCL logs and alert if SHARP fallbacks begin appearing due to switch memory pressure.6

How: implement, integrate, maintain

Fabric prerequisites. SHARP firmware on the switches, a sharp_am Aggregation Manager running on a management host alongside the Subnet Manager, and the nccl-rdma-sharp-plugins plugin discoverable by NCCL. Each host must load the GPUDirect RDMA kernel module so collectives reach GPU memory directly.34

Confirm GPUDirect RDMA is active. NVIDIA now recommends DMA-BUF over the legacy nvidia_peermem kernel module where the prerequisites are met (Open Kernel Module driver, CUDA >= 11.7, Turing-or-newer GPU, Linux kernel >= 5.12); DMA-BUF is the GPU Operator's helm-install default, with nvidia_peermem as an explicit legacy opt-in.13 On the legacy path, check the module is loaded:

lsmod | grep nvidia_peermem

(absence on the legacy path means NCCL may use CPU-staged RDMA copies instead of true GPUDirect; on the DMA-BUF path, absence of this module is expected and not itself a problem).4

Enable IB SHARP (CollNet). With the plugin present, the documented variables that route eligible collectives through the switch are:11

export NCCL_COLLNET_ENABLE=1     # allow the CollNet (SHARP) algorithm
export NCCL_ALGO=CollnetChain    # or CollnetDirect; current NCCL split the pre-2.14 "CollNet" spelling in two

Upstream inconsistency, flagged rather than silently resolved: NCCL's own current environment-variable docs retired the bare CollNet spelling after version 2.13, replacing it with CollnetChain/CollnetDirect from 2.14 onward (already-current NCCL is far past that). NVIDIA's own HPC-X and SHARP documentation, however, still shows NCCL_ALGO=CollNet as the example value, with the SHARP page's own annotation explaining it was "required to overcome a bug in NCCL <= 2.7.8," i.e. guidance frozen from before the 2.14 split and never updated. If you are following an HPC-X or SHARP guide that says CollNet, use CollnetChain/CollnetDirect instead and verify against the core NCCL docs.12

NCCL_ALGO is normally left unset so NCCL auto-selects per message size and topology; pin it only to force or A/B-test the SHARP path.216

Enable NVLink SHARP (NVLS). On third-generation NVSwitch (NVLink4, Hopper and later), NVLS is selected via the NCCL_ALGO NVLS / NVLSTree options and gated by NCCL_NVLS_ENABLE. The documented default is 2 (use NVLS where supported; do not fail if unsupported, but fail if resources cannot be allocated); 0 disables it. Note the default was 1 in NCCL 2.17–2.20.16

export NCCL_NVLS_ENABLE=2        # default: enable NVLS where the NVSwitch domain supports it
# optional, to force the NVLS path for inspection:
export NCCL_ALGO=NVLSTree

NVL72-class symmetric-memory low-latency kernels (NCCL 2.27+) apply within a single NVLink domain and are gated separately by NCCL_WIN_ENABLE (default on); set NCCL_WIN_ENABLE=0 to disable.17

Disable for A/B testing. SHARP is not enabled by default; it requires the plugin selection / policy above. NCCL_SHARP_DISABLE does not exist in current NCCL or HPC-X documentation; do not use it. To measure IB SHARP's impact, disable the CollNet path specifically, which is finer-grained than tearing down the whole RDMA plugin:9

export NCCL_COLLNET_ENABLE=0     # bypass IB SHARP/CollNet only; revert (=1) for production
# Coarser alternative: disables the entire nccl-rdma-sharp-plugins RDMA path, not just SHARP.
# export NCCL_IBEXT_DISABLE=1

For NVLink SHARP (NVLS), use NCCL_NVLS_ENABLE=0 instead, documented above; it is a separate knob from the IB-side variables here.

Integration. Using SHARP requires no application code changes; it is a fabric-and-environment configuration. PyTorch DDP/FSDP and any NCCL-based stack inherit it once the fabric and env are configured.1 For best results, register persistent NCCL user buffers (ncclCommRegister / ncclCommDeregister): zero-copy registration is essential to the best SHARP paths for both on-node NVLS and off-node IB, and it cuts SM/channel usage.7

Verify and maintain. SHARP usage shows up in NCCL logs:

export NCCL_DEBUG=INFO           # logs will name SHARP / CollNet / NVLS when active

Device-level support can be checked with ibv_devinfo.8 Continuously monitor the logs and alert on SHARP-to-non-SHARP fallbacks, which signal switch buffer pressure or a degraded Aggregation Manager.6

Reference-template guidance only. The flags, defaults, and prerequisites above are drawn from the cited book chapter and NVIDIA documentation; they have not been hardware-validated here. Confirm exact behavior against your NCCL version's release notes and your fabric's SHARP/UFM configuration before relying on them.

References

Related: NCCL Collectives and Algorithm Selection · NVSHMEM: GPU-Initiated Communication · Communication-Computation Overlap · RDMA and RoCE Performance Tuning · BlueField DPUs for AI Networking · HPC Networking Fabric · NVSwitch and NVLink · Fabric Manager · Ansible Role: rdma_fabric · Distributed Training Platform · FSDP · NCCL Hang / Collective Stall · Glossary

  1. Fregly, Ch. 4, "In-Network SHARP Aggregation." ↩↩↩↩↩↩↩↩↩↩↩↩↩↩

  2. Fregly, Ch. 4, "NCCL Communication Algorithms" (Tree/NVLSTree, CollNet, CollTree; NCCL_ALGO override). ↩↩

  3. Fregly, Ch. 4: NCCL offloads via the NCCL RDMA SHARP plugin with SHARP firmware on the switches and a SHARP Aggregation Manager running alongside the Subnet Manager. ↩↩

  4. Fregly, Ch. 4: each host must load the GPUDirect RDMA kernel module; verify with lsmod | grep nvidia_peermem. ↩↩

  5. Fregly, Ch. 4: SHARP is primarily an InfiniBand technology; Spectrum-X Ethernet does not expose switch-resident reduction engines analogous to SHARP. ↩

  6. Fregly, Ch. 4: finite switch reduction buffers; very large collectives may fall back to regular methods — monitor NCCL logs and alert on fallbacks. ↩↩

  7. Fregly, Ch. 4, "Persistent NCCL User Buffers and Zero-Copy Registration" (ncclCommRegister / ncclCommDeregister). ↩

  8. Fregly, Ch. 4: verify with NCCL logs (NCCL_DEBUG=INFO) and ibv_devinfo; the book's suggested NCCL_SHARP_DISABLE=1 for A/B testing does not exist in current NCCL/HPC-X docs, see 9 for the documented alternative. ↩

  9. NVIDIA HPC-X, "NCCL-RDMA-SHARP Plugins": NCCL_IBEXT_DISABLE=1 "disables the use of the plugin," falling back to NCCL's native internal communication (broader than SHARP alone); NCCL_COLLNET_ENABLE=0 disables the CollNet/SHARP path specifically while leaving the rest of the plugin's RDMA transport active. NCCL_SHARP_DISABLE does not appear in this documentation. https://networking-docs.nvidia.com/hpcxum/2221/NCCL-RDMA-SHARP+Plugins ↩↩

  10. Fregly, Ch. 4, "Key Takeaways": in-network computing like SHARP can accelerate collectives by 2x–5x, especially at scale. ↩

  11. NVIDIA HPC-X, "NCCL-RDMA-SHARP Plugins": NCCL_COLLNET_ENABLE=1 enables the CollNet plugin path for SHARP aggregation with NCCL; the page's own example still pairs this with the pre-2.14 NCCL_ALGO=CollNet spelling, see 12. ↩↩

  12. NVIDIA NCCL, Environment Variables (NCCL_ALGO, current version 2.30.7): bare Collnet was valid NCCL 2.5 to 2.13 only; split into CollnetChain and CollnetDirect from 2.14 onward. NVIDIA's own HPC-X/SHARP documentation was not updated for this split and still shows the retired spelling, one SHARP page annotating it as "Required to overcome a bug in NCCL <= 2.7.8." https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html ↩↩

  13. NVIDIA GPU Operator, GPUDirect RDMA: DMA-BUF is now recommended over the legacy nvidia_peermem kernel module and is the helm-install default (requires an Open Kernel Module driver, CUDA >= 11.7, Turing-or-newer GPU, Linux kernel >= 5.12); nvidia_peermem remains available as an explicit legacy opt-in. https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html ↩

  14. NVIDIA SHARP user manual: the Aggregation Manager (sharp_am) receives SHARP job requests and manages reduction trees. ↩

  15. NVIDIA, "Advancing Performance with NVIDIA SHARP In-Network Computing": SHARPv2 streaming aggregation, 16/32/64-bit integer and FP ops, GPUDirect RDMA. ↩↩

  16. NCCL documentation, Environment Variables: NCCL_NVLS_ENABLE (default 2; 1 in NCCL 2.17–2.20), NVLS/NVLSTree via NCCL_ALGO. ↩↩↩↩

  17. NVIDIA, "Enabling Fast Inference and Resilient Training with NCCL 2.27": SHARP for AllGather and ReduceScatter on NVLink and IB fabrics; SM usage 16 -> 6 or fewer; symmetric memory and NCCL_WIN_ENABLE. ↩↩