Markdown

GPU communication protocols: a map from GPU-to-GPU to DC-to-DC¶

Scope: a reference map of every protocol and technology this knowledge base documents for moving data between GPUs, organized by physical scope, intra-node, intra-datacenter cluster fabric, and inter-datacenter, and cross-referenced by the four things a platform team actually builds with them: distributed training, RL post-training, inference serving, and standing up a cluster in the first place. This page carries no procedure of its own; it routes to the focused pages that do, and states, explicitly, which layer answers which question, so the right page is one hop away rather than found by trial and error.

The layered answer¶

"How many protocols connect GPUs" does not have one number; it depends which layer is being counted. This KB documents four layers, each solving a structurally different problem:

flowchart TB
  L1["Layer 1: GPU-to-GPU, one chassis<br/>NVLink, NVSwitch, PCIe P2P"]
  L2["Layer 2: node-to-node, one datacenter<br/>InfiniBand, RoCEv2, RDMA, GPUDirect RDMA"]
  L3P["Layer 3a: DC-to-DC, physical<br/>dark fiber, DWDM (400ZR/OpenZR+), private backbone"]
  L3F["Layer 3b: DC-to-DC, fabric extension<br/>MetroX, IB router, ATSW-style RoCE aggregation"]
  L3O["Layer 3c: DC-to-DC, overlay<br/>WireGuard, NCCL-over-TCP/Gloo"]
  L4["Layer 4: the algorithm that decides<br/>whether Layer 3 needs to be fast at all<br/>DiLoCo, geo-distributed placement"]

  L1 --> L2
  L2 --> L3P
  L3P --> L3F
  L3P --> L3O
  L3F --> L4
  L3O --> L4

Layers 1 and 2 are almost always both present in a real cluster (a node has intra-node links and a NIC onto the cluster fabric). Layer 3 is three different, non-interchangeable answers to "how do two datacenters talk," picked by whether the link is owned end to end and how latency-sensitive the traffic crossing it is. Layer 4 is not a network layer at all: it is the decision of whether the workload needs Layer 3 to behave like Layer 2 (in which case, pay for fabric extension) or can tolerate a slow, infrequent link (in which case, change the algorithm instead of the network).

Layer 1: GPU-to-GPU, one chassis¶

Technology	What it is	Page
NVLink	Direct point-to-point GPU interconnect, datacenter SXM only. 600 GB/s (Ampere, NVLink3) to 900 GB/s (Hopper) to 1.8 TB/s (Blackwell) to 3.6 TB/s (Rubin)	NVSwitch & NVLink
NVSwitch	Crossbar ASIC turning NVLink's point-to-point links into a non-blocking all-to-all fabric, up to a 72-GPU domain (NVL72), run by Fabric Manager	NVSwitch & NVLink, Fabric Manager
PCIe peer-to-peer	Universal fallback; the only multi-GPU path on consumer/most workstation cards, no NVSwitch domain	HPC Networking Fabric

Two real protocols (NVLink, PCIe), plus NVSwitch as the switching fabric that scales NVLink to all-to-all and, via IMEX, to rack scale.

Layer 2: node-to-node, one datacenter¶

Technology	What it is	Page
InfiniBand	Native lossless fabric, SHARP-capable, the default for large training fabrics	HPC Networking Fabric
RoCEv2	RDMA over routable Ethernet, needs PFC+ECN+DCQCN to stay lossless	RDMA & RoCE Tuning
RDMA / GPUDirect RDMA	Kernel-bypass memory access, and its GPU-specific extension (NIC-to-GPU direct DMA)	RDMA & RoCE Tuning
SHARP / NVLS	In-network collective reduction on IB switches and NVSwitch respectively	SHARP In-Network Reduction
NCCL	The collective library that picks an algorithm (Ring/Tree/CollNet/NVLS) across whichever of the above is present	NCCL Collectives & Algorithms

Two real transport choices (InfiniBand vs. RoCEv2), both carrying RDMA/GPUDirect RDMA, with NCCL as the software layer that decides how to use whichever is present.

Layer 3: DC-to-DC, three different answers¶

This is where "how many protocols" gets genuinely three-way, because the right answer depends on whether the link is owned end to end and how latency-sensitive the traffic is.

3a. Physical: how the datacenters are actually wired together. Dark fiber, DWDM coherent optics (400ZR up to 120 km, OpenZR+ to 2,000 km at reduced rate), and the private backbones built on top (Meta's 10x Backbone and Prometheus/BAG, Google's B4). This layer has to exist before anything above it can work; see Datacenter Interconnect (DCI) for the full inventory, the optics-tier decision, and the speed-of-light latency floor.

3b. Fabric extension: making two buildings share one tightly-coupled RDMA fabric. NVIDIA MetroX (InfiniBand, up to 40 km, DWDM-based) and the IB router (subnet segregation and fault isolation, a different mechanism from extension) are the InfiniBand answers; Meta's ATSW aggregation layer is the production RoCE answer, accepting oversubscription rather than re-provisioning PFC for metro distance. See RDMA/InfiniBand Fabric Extension Across Metro Distances for why RoCEv2 does not simply stretch, and the buffer-headroom math that shows why.

3c. Overlay: securing traffic over a link you do not own end to end. WireGuard, the "de facto" tool for multi-provider and multi-cloud GPU clusters, laid over the public internet, with NCCL pinned to TCP/socket transport (or Gloo) since NCCL-over-IB cannot cross an overlay. See Overlay & Mesh Networking, plus the transport layer purpose-built for inference specifically in P2P Transport for Decentralized Inference.

Three structurally different answers, not a maturity ladder: 3b is faster and more expensive and requires owning or leasing the physical path; 3c is cheaper, works over commodity internet, and is fundamentally not fast enough for a per-step synchronous collective.

Layer 4: the algorithm that decides whether Layer 3 needs to be fast¶

A slow or oversubscribed Layer 3 link is not automatically a blocker; it changes which training and inference patterns are viable:

DiLoCo trades per-step synchronization for infrequent outer-loop sync, tolerant of exactly the WAN-overlay link 3c provides. The runnable version is Recipe: DiLoCo (geo-distributed).
Geo-Distributed Training Placement solves rank placement on a slow, heterogeneous fabric directly, using measured link costs rather than assuming a topology.
Cross-WAN Model-Parallel Inference and Region-Replica Routing are the inference-serving counterparts: shard a model across non-colocated sites when no site holds a full replica, or route requests to whichever full replica is closest, respectively.

By use case¶

Distributed training, single datacenter. Layer 1 and Layer 2 only. Read HPC Networking Fabric then NVSwitch & NVLink, validate with Fabric bring-up, validation & benchmarking, run with Recipe: FSDP (single datacenter).

Distributed training, multiple datacenters or a gigawatt-scale multi-building campus. Start at Datacenter Interconnect (DCI) to know what physically links the sites. If the workload needs per-step synchronous collectives across buildings within metro reach, read RDMA/InfiniBand Fabric Extension Across Metro Distances next. If it does not, or the sites are farther apart or not owned end to end, go straight to DiLoCo and Geo-Distributed Training Placement, carried over Overlay & Mesh Networking.

RL post-training. RL rollouts and policy updates ride the same Layer 1/2 fabric as any other distributed training job inside one datacenter (see RL libraries overview for the colocated-vs-disaggregated architecture choice); when rollout workers and the trainer are not colocated, the same Layer 3 decision applies as for training generally, DiLoCo-style infrequent sync over an overlay for a geo-distributed rollout pool, or fabric extension if the workers are metro-close and need tight coupling. Delta Weight Sync covers the sparse, bandwidth-efficient weight-update pattern that makes an infrequent-sync topology practical for RL specifically.

Inference serving. Layer 1/2 inside one datacenter for a normal disaggregated deployment (Disaggregated Inference, KV cache transfer with NIXL). Across sites: Region-Replica Routing when a full replica fits at every site, Cross-WAN Model-Parallel Inference when it does not, both riding the transport documented in P2P Transport for Decentralized Inference rather than the training-oriented WireGuard overlay.

Standing up a cluster from bare metal. Follow Start here's own "Stand up a GPU cluster from bare metal" reading path for the intra-datacenter build; add Datacenter Interconnect (DCI) and, if applicable, RDMA/InfiniBand Fabric Extension Across Metro Distances to that path the moment the design spans more than one building.

Don't-miss checklist¶

Do not reach for Layer 3b (fabric extension) before confirming the workload actually needs per-step synchronous collectives across the link; most cross-DC and cross-cloud workloads are better served by Layer 4 (DiLoCo-class algorithms) than by an expensive metro appliance.
Do not assume a cloud interconnect product (AWS Direct Connect, Azure ExpressRoute) links a customer's own two sites by default; only a specific feature (ExpressRoute Global Reach, or the equivalent elsewhere) re-exports the provider's backbone for that purpose, detailed on Datacenter Interconnect (DCI).
Do not confuse an InfiniBand router (subnet segregation, fault isolation) with fabric extension (one unified domain across buildings); they solve different problems, detailed on RDMA/InfiniBand Fabric Extension Across Metro Distances.
Do not put a decode-loop or per-step-synchronous hot path on the training-oriented WireGuard overlay; Overlay & Mesh Networking says explicitly not to, and inference has its own purpose-built transport in P2P Transport for Decentralized Inference.

References¶

NVIDIA NVLink and NVLink Switch: https://www.nvidia.com/en-us/data-center/nvlink/
NVIDIA Quantum-X800 InfiniBand platform: https://www.nvidia.com/en-us/networking/products/infiniband/quantum-x800/
NVIDIA GPUDirect RDMA: https://docs.nvidia.com/cuda/gpudirect-rdma/index.html
WireGuard protocol and docs: https://www.wireguard.com/
DiLoCo, "Distributed Low-Communication Training of Language Models," arXiv:2311.08105: https://arxiv.org/abs/2311.08105
NVIDIA Developer Blog, cross-datacenter training case study: https://developer.nvidia.com/blog/turbocharge-llm-training-across-long-haul-data-center-networks-with-nvidia-nemo-framework/