Skip to content
Markdown

RDMA/InfiniBand fabric extension across metro distances

Scope: making GPUs in two separate datacenter buildings share one RDMA fabric, rather than two independently-fabricked clusters stitched by a software overlay. Covers the two genuinely different NVIDIA mechanisms (the IB router, which segments a fabric into independent subnets, versus MetroX, which extends one fabric's reach), why RoCEv2's lossless mechanism (PFC) does not scale to metro distance by itself, transceiver reach limits, and what breaks in NCCL collectives as hop latency grows. This sits between two other pages: the physical path itself is Datacenter interconnect (DCI); the intra-building fabric this extends is HPC networking fabric and RDMA & RoCE tuning. The reading map across all three layers is GPU communication protocols: a map from GPU-to-GPU to DC-to-DC.

What it is

Two NVIDIA product lines answer superficially similar questions with different mechanisms, and conflating them is a real accuracy defect:

  • IB router (SB7780/SB7880): NVIDIA's own description is that "the router segregates the data center network into several subnets. Each subnet runs its own subnet manager, effectively isolating each subnet from the others' availability or instability."1 This is fault isolation and scale beyond one subnet manager's node-count ceiling ("enable[s] the fabric to scale up to an unlimited number of nodes," versus a managed switch's roughly 2,000-node single-subnet limit),1 not fabric extension. Two subnets behind an IB router are two separate InfiniBand networks that can route traffic to each other; they are not one NVLink-style unified domain.
  • MetroX-3 XC: "extends the reach of InfiniBand up to 40 kilometers," over "dense wavelength-division multiplexing (DWDM) infrastructures," with built-in "encryption over long distances."2 This is the fabric-extension answer: it exists specifically to connect "remote data centers" and data-center-to-remote-storage.2 The prior generation, MetroX-2, used named long-reach transceivers (LR4 at roughly 10 km, ER4 at roughly 40 km) on dedicated long-haul switch hardware distinct from standard ConnectX/Quantum port optics.

RoCEv2 has no equivalent metro-extension appliance in this KB's research; the practical answer is that its lossless mechanism does not stretch to metro distance without re-engineering. RoCEv2's losslessness rests on Priority Flow Control (PFC, IEEE 802.1Qbb): a switch asserts XOFF before a queue overflows, but frames already in flight when the sender receives that pause must land somewhere, so every lossless port reserves headroom buffer sized from the cable's round-trip propagation delay. Juniper's own PFC configuration guide states the mechanism plainly: "the system uses the MRU and the length of the attached physical cable to calculate buffer headroom allocation... A higher MRU or a longer cable length increases the amount of headroom buffer space required."3 The default assumption Juniper's own PFC buffer calculations bake in is a 100-meter cable, a normal in-rack or in-row distance, not a metro one; this KB has not independently confirmed the same default across other vendors' switch ASICs, but 100 m is a common in-rack cable-length assumption industry-wide, so treat this as illustrative of the general problem rather than a universally documented constant. The validated model below quantifies exactly how far off a 40 km metro hop is from that assumption.

Meta's own production network answers a related but different question, extending the reach of a RoCE domain within a metro region rather than lighting a single link across it: an Aggregator Training Switch (ATSW) layer "connects the CTSWs in a data center building, expanding the RoCE domain beyond a single AI Zone," explicitly accepting that "the cross-AI Zone connectivity is oversubscribed by design, with network traffic balanced using ECMP."4 That is architecturally closer to routing between fabric segments (like an IB router) than to lighting one lossless link across 40 km (like MetroX); Meta pays for domain reach with oversubscription rather than re-provisioning PFC headroom for metro distance.

flowchart TB
  subgraph BLDG_A["Building A"]
    R1["Rail-aligned IB/RoCE fabric<br/>(networking-fabric.md)"]
  end
  subgraph BLDG_B["Building B, up to ~40 km"]
    R2["Rail-aligned IB/RoCE fabric"]
  end
  R1 -->|"MetroX-3 XC over DWDM:<br/>ONE extended fabric"| R2
  R1 -->|"IB router (SB7780/SB7880):<br/>TWO routed, isolated subnets"| R3["Building B, alternate design<br/>independent subnet manager"]
  R1 -->|"ATSW aggregation layer:<br/>ONE oversubscribed RoCE domain"| R4["Building B, Meta-style design<br/>ECMP, oversubscribed by design"]

Why use it

  • NCCL collectives assume a fabric, not a routed hop. NCCL collectives and algorithm selection already documents Ring (bandwidth-optimal, O(N) steps, latency-sensitive to every hop) and Tree (O(log N) steps, chosen when latency dominates); SHARP/CollNet in-network reduction additionally assumes the switches in the data path can participate in the reduction tree. Stretching the fabric with MetroX (one extended domain) preserves the assumptions those algorithms already make; a software overlay or an IB-router hop changes the topology NCCL sees and can force a fallback the algorithm was not tuned for.
  • It is the tightly-coupled alternative to a WAN overlay. Overlay & mesh networking is explicit that "NCCL-over-IB is unavailable across an overlay" and pins collectives to TCP/socket transport instead, appropriate for DiLoCo's low-frequency sync but not for a per-step all-reduce. Metro RDMA extension exists for the case in between: two buildings close enough, and important enough, to justify a dedicated long-haul appliance instead of falling back to sockets.
  • It is how gigawatt-scale campuses stay one training system. Meta's own architecture (ATSW expanding the RoCE domain beyond one AI Zone) is the production answer to the same siting pressure documented on Datacenter interconnect (DCI): no single building holds a gigawatt-class GPU count, so something has to extend the RDMA domain across buildings if the campus is to train as one cluster rather than several.

When to use it (and when not)

  • Use MetroX-class fabric extension when two buildings are within its stated reach (up to 40 km) and the workload needs the same synchronous, lossless RDMA semantics inside both buildings, not a slower cross-building link.
  • Use an IB router when the goal is fault isolation and scale past one subnet manager's ceiling, not fabric extension; accept that traffic between subnets is routed, not fused into one domain.
  • Accept oversubscription (the Meta ATSW pattern) or run separate fabric islands per building when full non-blocking metro RDMA is not worth the cost. This KB's own research turned up vendor reach and product claims for MetroX, but no independently documented, named case of a live single-fabric deployment spanning multiple buildings at a stated production scale; treat "one unified fabric across buildings" as a real, shipping capability on paper, and verify a specific vendor's deployment guidance before committing a design to it.
  • Do not assume RoCEv2 stretches to metro distance by raising cable length alone. The headroom model below shows why: the buffer a lossless port needs grows with distance, and a 40 km hop needs roughly two orders of magnitude more headroom than the 100 m default most switch ASICs are provisioned for.
  • Do not put a per-step synchronous collective across an under-provisioned or oversubscribed cross-building hop and expect in-building performance. If the link cannot carry it at the needed rate, the answer is DiLoCo's infrequent sync, not a stretched Ring all-reduce.

How to use it: does the distance and the traffic pattern justify a dedicated appliance

# pfc_headroom_metro.py -- validated: how much buffer headroom a switch port needs to stay
# lossless under Priority Flow Control (PFC) as cable length grows, and why that constrains
# stretching RoCEv2 to metro distances without re-engineering. Grounded in the mechanism
# Juniper's own PFC config guide states: headroom is sized from cable length and MRU, with a
# 100 m DEFAULT assumption baked into most switch ASIC buffer allocations.
# Run: python3 pfc_headroom_metro.py

def propagation_delay_us(distance_km: float, refractive_index: float = 1.468) -> float:
    """One-way propagation delay in microseconds for single-mode fiber."""
    C_KM_S = 299_792.458
    v_km_s = C_KM_S / refractive_index
    return (distance_km / v_km_s) * 1e6


def pfc_headroom_bytes(link_rate_gbps: float, distance_km: float,
                        reaction_time_us: float = 1.0) -> float:
    """Bytes of headroom buffer a PFC-lossless port must reserve to absorb frames already
    in flight when a PAUSE (XOFF) is asserted: one full round trip of propagation delay
    (the sender must see the pause before it stops) plus a fixed switch/NIC reaction time.
    This is the mechanism, not a vendor-exact formula; real ASICs add MTU-sized margins."""
    rtt_us = 2 * propagation_delay_us(distance_km) + reaction_time_us
    bits_in_flight = link_rate_gbps * 1e9 * (rtt_us / 1e6)
    return bits_in_flight / 8.0


if __name__ == "__main__":
    for km in (0.1, 1, 10, 40, 100):
        h100 = pfc_headroom_bytes(100.0, km)
        h400 = pfc_headroom_bytes(400.0, km)
        print(f"{km:>6.1f} km: 100G headroom = {h100/1024:8.1f} KiB, 400G headroom = {h400/1024:8.1f} KiB")

    # The Juniper-documented default assumption (100 m) vs a real metro hop (40 km, MetroX's
    # own advertised max reach): headroom required scales by roughly the distance ratio.
    default_headroom = pfc_headroom_bytes(400.0, 0.1)
    metro_headroom = pfc_headroom_bytes(400.0, 40.0)
    ratio = metro_headroom / default_headroom
    print(f"\n400G headroom at Juniper's 100 m default: {default_headroom/1024:.2f} KiB")
    print(f"400G headroom at MetroX's 40 km max reach: {metro_headroom/1024:.2f} KiB")
    print(f"Required headroom grows {ratio:.0f}x -- far past typical on-chip buffer budgets, "
          f"which is why metro RDMA is a dedicated appliance (MetroX/IB router), not 'run RoCE "
          f"over a longer fiber.'")

    # Adversarial assertions.
    assert pfc_headroom_bytes(400.0, 0.0, reaction_time_us=0.0) == 0.0
    assert pfc_headroom_bytes(400.0, 40.0) > pfc_headroom_bytes(100.0, 40.0)
    assert pfc_headroom_bytes(100.0, 100.0) > pfc_headroom_bytes(100.0, 1.0)
    far = pfc_headroom_bytes(400.0, 1000.0, reaction_time_us=0.0)
    near = pfc_headroom_bytes(400.0, 10.0, reaction_time_us=0.0)
    assert abs(far / near - 100.0) < 0.5, (far, near)

    print("\nALL ASSERTIONS PASSED")

Executed output:

   0.1 km: 100G headroom =     24.2 KiB, 400G headroom =     96.6 KiB
   1.0 km: 100G headroom =    131.8 KiB, 400G headroom =    527.0 KiB
  10.0 km: 100G headroom =   1207.7 KiB, 400G headroom =   4830.8 KiB
  40.0 km: 100G headroom =   4794.2 KiB, 400G headroom =  19176.6 KiB
 100.0 km: 100G headroom =  11967.1 KiB, 400G headroom =  47868.4 KiB

400G headroom at Juniper's 100 m default: 96.65 KiB
400G headroom at MetroX's 40 km max reach: 19176.64 KiB
Required headroom grows 198x -- far past typical on-chip buffer budgets, which is why metro RDMA is a dedicated appliance (MetroX/IB router), not 'run RoCE over a longer fiber.'

ALL ASSERTIONS PASSED

Read the 198x figure as the reason a dedicated long-haul appliance exists at all: a standard switch ASIC's on-chip buffer is a fixed, shared resource, and reserving tens of megabytes of headroom on every metro-facing port for one link is not a config change, it is a different hardware budget. MetroX and the IB router both solve this by moving the metro hop off the standard fabric switch entirely, onto purpose-built hardware with its own buffering and (for MetroX) its own DWDM optics.

How to develop with it: match the mechanism to the goal

  1. Fault isolation and scale past one subnet manager: IB router. Traffic between subnets is routed, and the two sides can differ in topology.
  2. One synchronous fabric across buildings, within reach: MetroX-class long-haul appliance over DWDM.
  3. RoCE domain reach across a metro campus, accepting oversubscription: an aggregation layer in the Meta ATSW shape, with the scheduler placing latency-sensitive rank groups to minimize cross-zone hops rather than assuming uniform bandwidth.
  4. Beyond metro reach, or no dedicated appliance budget: do not attempt to extend the RDMA fabric at all; treat the buildings as separate islands and move to DiLoCo or the geo-distributed rank-placement model for the cross-building communication pattern instead.

How to maintain it

  • Validate the extended link with the same discipline as an in-building fabric, not a lighter one: Fabric bring-up, validation & benchmarking's bandwidth-at-line-rate proof still applies, but run it end to end across the metro hop specifically, since a healthy in-building fabric on both sides says nothing about the extension link's actual behavior under load.
  • Watch SHARP/CollNet reduction-tree membership across the extended domain. This KB's research could not confirm whether MetroX passes through or terminates SHARP's aggregation tree; treat that as an open question to validate against current NVIDIA guidance before relying on in-network reduction across a metro hop, and fail back to Ring/Tree if it does not.
  • Monitor headroom and PFC pause counters on every metro-facing port specifically, not just the fabric average; a port under-provisioned for its actual cable length will show as intermittent, distance-correlated frame loss that an in-building baseline will not catch.

How to run it in production

Treat cross-building RDMA extension as a placement-aware resource, not a transparent extension of local bandwidth. Meta's own production pattern is instructive precisely because it does not pretend otherwise: it accepts oversubscription at the ATSW layer by design and compensates with a scheduler enhanced to find a "minimum cut" when dividing training nodes into AI Zones, learning each GPU server's position in the logical topology to recommend a rank assignment that reduces cross-zone traffic.4 The operational lesson generalizes past Meta's specific RoCE architecture: whatever mechanism extends the fabric (MetroX, IB router, or an oversubscribed aggregation layer), the scheduler and the workload's parallelism layout need to know which ranks are local and which crossed a building boundary, and should bias toward keeping the most latency-sensitive collective (tensor-parallel all-reduce) fully local while letting a less frequent one (data-parallel gradient sync, or a DiLoCo outer-loop sync) cross the extension.

Failure modes

  • Assuming RoCEv2 "just works" over a longer cable. It does not without re-provisioned headroom; see the 198x figure above.
  • Confusing an IB router with fabric extension. A router gives two isolated, routed subnets, not one unified NVLink-style domain; a design that needs one fabric across buildings and gets a router instead will see routed-hop behavior it did not plan for.
  • Assuming SHARP/in-network reduction survives an extension hop unmodified. This is genuinely unverified in this KB's own research; do not assume it works, and do not assume it fails, without checking current vendor guidance for the specific appliance in the path.
  • Placing a per-step synchronous collective across an oversubscribed cross-zone hop. Meta's own architecture accepts oversubscription and compensates in the scheduler; a design that skips the scheduler-side compensation will see the straggler effect the oversubscription creates.
  • No named, independently verified production case for "one InfiniBand fabric across multiple buildings" beyond vendor product claims. Validate against a specific vendor's current deployment guidance before committing a design to full fabric unification at metro scale; this KB found strong evidence for the underlying products (MetroX, IB router) and for a RoCE-domain-extension production case (Meta ATSW), but not for a named single-InfiniBand-fabric-across-buildings deployment at stated scale.

References

  • NVIDIA Networking, SB7780/SB7880 InfiniBand Router (subnet segregation, fault isolation, unlimited-node scaling): https://www.nvidia.com/en-eu/networking/infiniband/sb7780/
  • NVIDIA Networking, InfiniBand long-haul systems (MetroX-3 XC, up to 40 km, DWDM, encryption): https://www.nvidia.com/en-us/networking/infiniband-long-haul-systems/
  • NVIDIA Networking, MetroX-2 (legacy long-haul reach and transceivers): https://www.nvidia.com/en-eu/networking/infiniband/metrox-2/
  • Juniper Networks, "Understanding CoS IEEE 802.1p Priorities for Lossless Traffic Flows" (PFC headroom sized from cable length and MRU, 100 m default): https://www.juniper.net/documentation/us/en/software/junos/traffic-mgmt-qfx/topics/concept/cos-qfx-series-lossless-ieee8021p-priority-config-understanding.html
  • Meta Engineering, "RoCE networks for distributed AI training at scale" (Clos topology, ATSW layer, cross-AI-Zone oversubscription and ECMP): https://engineering.fb.com/2024/08/05/data-center-engineering/roce-network-distributed-ai-training-at-scale/
  • NVIDIA, 100G-PAM4 transceiver and fiber reach reference (SR4/DR4/FR4 distances): https://docs.nvidia.com/networking/display/400g100gpam4ovdev/transceivers-and-fiber-details-100g-pam4
  • IEEE 802.1Qbb, "Priority-based Flow Control" (the standard PFC is defined against): https://standards.ieee.org/ieee/802.1Qbb/4361/

Related: Datacenter Interconnect (DCI) · HPC Networking Fabric · RDMA & RoCE Tuning · SHARP In-Network Reduction · NCCL Collectives & Algorithms · Overlay & Mesh Networking · GPU Communication Protocols: a Map from GPU-to-GPU to DC-to-DC · Glossary

  1. NVIDIA Networking, SB7780/SB7880 InfiniBand Router: https://www.nvidia.com/en-eu/networking/infiniband/sb7780/ ↩↩

  2. NVIDIA Networking, InfiniBand long-haul systems: https://www.nvidia.com/en-us/networking/infiniband-long-haul-systems/ ↩↩

  3. Juniper Networks, "Understanding CoS IEEE 802.1p Priorities for Lossless Traffic Flows": https://www.juniper.net/documentation/us/en/software/junos/traffic-mgmt-qfx/topics/concept/cos-qfx-series-lossless-ieee8021p-priority-config-understanding.html ↩

  4. Meta Engineering, "RoCE networks for distributed AI training at scale": https://engineering.fb.com/2024/08/05/data-center-engineering/roce-network-distributed-ai-training-at-scale/ ↩↩