Markdown

Datacenter interconnect (DCI): dark fiber, DWDM, and private backbones¶

Scope: the physical and optical layer that links two or more datacenter buildings, campuses, or regions so their GPU clusters can be operated as one system: dark fiber, DWDM coherent pluggable optics (400ZR/OpenZR+), and the private backbones hyperscalers build on top (Meta's 10x Backbone and Prometheus/BAG, Google's B4), plus the cloud-provider products that offer a slice of the same thing without owning fiber. This is the physical substrate beneath two different upper layers documented elsewhere: when GPUs must share one tightly-coupled RDMA fabric across buildings, see RDMA/InfiniBand fabric extension across metro distances; when a link is not owned end to end (public internet, multi-cloud, decentralized pools), see Overlay & mesh networking, which assumes a physical path already exists and secures it in software. The reading map across all three layers is GPU communication protocols: a map from GPU-to-GPU to DC-to-DC.

What it is¶

Three distinct things get bundled under "DCI," and conflating them is the most common mistake:

Dark fiber: unlit glass strands, spliced end to end between two points, with no active equipment until the lessee installs their own optics. Leased from carriers or colocation providers, or owned outright by a hyperscaler that trenches its own conduit.
DWDM / coherent optics: the equipment that lights the fiber, multiplexing many independent wavelengths onto one fiber pair. Current pluggable coherent optics: 400ZR (an OIF standard, whose own spec is titled "Implementation Agreement OIF-400ZR-01.0" and mandates fixed DP-16QAM modulation for cross-vendor interoperability; Juniper's own coherent-optics documentation states reach "up to 120 km")¹ and OpenZR+ (an MSA, not an OIF standard, trading line rate for reach on the same QSFP-DD form factor: "400 Gbps can support a nominal reach of 500 km, 300 Gbps up to 900 km, and 200 Gbps up to 2000 km" on Juniper's QDD-400G-ZR-M/JCO400-QDD-ZR-M transceivers).¹ Both eliminate the standalone transponder chassis DWDM systems used to need; Meta states its own ZR pluggables cut power "80 to 90% less" versus the transponders they replaced (10-15 W per pluggable versus up to 2 kW per standalone transponder).²
The backbone network: the routing and traffic-engineering layer running over the lit fiber. Google's B4 (SIGCOMM 2013) was an early public example, an OpenFlow/SDN WAN with a centralized traffic-engineering service driving Google's own inter-datacenter links.³ Meta's Express Backbone (2017) is the same idea for a different reason: it exists because "machine-to-machine traffic often occurs in large bursts that may interfere with and impact the regular user traffic, affecting our reliability goals," so Meta runs a dedicated network for inter-datacenter machine traffic, separate from the internet-facing backbone, on Open/R distributed routing and MPLS segment routing with a centralized TE controller.⁴ Meta's newer 10x Backbone is the AI-era scale-up of the same architecture, with a distance-tiered optics strategy (below) and the claim that "a single AI Backbone site-pair is twice the size of the global backbone that we've been building for the last 10 years."²

Meta's own optics-tiering, stated directly in its 10x Backbone engineering post, is the clearest public disclosure of how a hyperscaler actually picks optics by distance: FR plugs for buildings roughly 3 km apart, LR plugs extending to roughly 10 km, and ZR plugs over DWDM beyond 10 km, with each fiber pair on the ZR tier carrying "64x 800G (51.2T)."²

flowchart LR
  A["Building A<br/>GPU racks"] --> OA["DCI optics<br/>(FR/LR/ZR pluggable)"]
  OA --> F["Dark fiber<br/>(leased or owned)"]
  F --> OB["DCI optics"]
  OB --> B["Building B<br/>GPU racks"]
  OA -.->|"DWDM: many wavelengths,<br/>one fiber pair"| DW["DWDM line system<br/>(beyond ~10 km)"]
  DW -.-> OB
  B --> BB["Backbone routing<br/>(TE, segment routing / SDN)"]
  BB --> C["Building C (another region)"]

Why use it¶

There is no bit path between buildings without it. Overlay & mesh networking secures and addresses traffic over a link that already exists; it does not create the link. WireGuard over the public internet is a real, cheaper substitute when a company does not own end-to-end fiber, but it inherits whatever DCI (or lack of it) the carriers between two points actually built.
Power and space force multi-building designs at gigawatt scale. Meta states its Prometheus cluster "will deliver 1-gigawatt of capacity," with infrastructure that will "span several data center buildings in a single larger region, interconnecting tens of thousands of GPUs."⁵ No single building's electrical service covers a gigawatt; DCI is what lets that capacity be operated as one cluster rather than several unconnected ones.
Cloud tenants can rent a slice of a provider's own DCI instead of leasing fiber. Azure's ExpressRoute Global Reach is explicit about this: "suppose you have two private data centers, one in California and one in Texas, each connected to an ExpressRoute circuit in their respective regions. You can use ExpressRoute Global Reach to link your data centers with these circuits, and your cross data-center traffic uses the Microsoft network."⁶ That is a customer's own sites linked through a cloud provider's backbone, the direct cloud-product answer to "how do I connect two of my own GPU clusters without owning fiber."

When to use it (and when not)¶

Use it when a single-region footprint has outgrown one building's power or space budget (the Prometheus case), or when two sites need deterministic, provider-graded bandwidth and latency rather than best-effort internet.
Metro (roughly under 100 km) is where DCI pays for tightly-coupled training. NVIDIA's own published case trained Nemotron-4 340B across Chicago (ORD) and Ashburn (IAD), Oracle Cloud sites "approximately 1,000 km" apart with a "measured 21 ms" round trip, and still reached "over 96% compared to single-site baseline" throughput.⁷ Read that result as evidence for what careful engineering buys at that specific distance and RTT, not as a general endorsement of any inter-DC distance; the physics below explains why the same approach degrades as distance grows.
Do not expect DCI to erase distance. Round-trip latency has a hard speed-of-light floor. The validated model below shows the ORD/IAD floor is 9.79 ms against NVIDIA's 21 ms measured figure, meaning routing, switching, and serialization already account for more than half the measured RTT at just 1,000 km; a transcontinental or intercontinental link makes the physical floor alone dominate. Past that point the answer is not more DCI bandwidth, it is a communication-efficient algorithm: DiLoCo and the geo-distributed rank-placement cost model.
Do not build DCI you do not need. If a link genuinely does not need to be owned end to end (bursting to a neocloud, a decentralized pool, multi-cloud), Overlay & mesh networking over commodity internet is cheaper and faster to stand up than leasing dark fiber.

How to use it: pick the optics tier by distance¶

# dci_reach_and_delay.py -- validated: which DCI optics tier fits a given inter-DC distance,
# and the physical speed-of-light floor on round-trip latency, cross-checked against NVIDIA's
# own published Chicago-Ashburn measurement (~1000 km, 21 ms RTT).
# Run: python3 dci_reach_and_delay.py

def optics_tier(distance_km: float) -> str:
    """Which pluggable-optics tier reaches this distance, using the verified reach figures:
    FR (~3 km, Meta 10x Backbone), LR (~10 km, Meta 10x Backbone), 400ZR (up to 120 km,
    OIF/Juniper), OpenZR+ (400G to 500 km, 300G to 900 km, 200G to 2000 km), MetroX-3 XC
    (InfiniBand-specific long-haul appliance, up to 40 km, DWDM-based)."""
    if distance_km <= 3:
        return "FR direct-attach pluggable"
    if distance_km <= 10:
        return "LR direct-attach pluggable (or MetroX for InfiniBand)"
    if distance_km <= 40:
        return "MetroX-3 XC (InfiniBand) or 400ZR (Ethernet), both over DWDM"
    if distance_km <= 120:
        return "400ZR over DWDM (InfiniBand needs an IB router hop, not MetroX)"
    if distance_km <= 500:
        return "OpenZR+ at 400G, or a leased/carrier DWDM line system"
    if distance_km <= 900:
        return "OpenZR+ at 300G (rate falls as reach grows)"
    if distance_km <= 2000:
        return "OpenZR+ at 200G, or a carrier long-haul transport service"
    return "beyond single-hop coherent optics reach; needs regeneration or a carrier backbone"


def one_way_delay_ms(distance_km: float, refractive_index: float = 1.468) -> float:
    """Speed-of-light floor for one-way propagation in single-mode fiber (SMF-28-class,
    refractive index ~1.468). This is a physical lower bound, not a prediction of real
    latency: real paths are never straight lines and add switching/serialization delay."""
    C_KM_S = 299_792.458
    v_km_s = C_KM_S / refractive_index
    return (distance_km / v_km_s) * 1000.0


if __name__ == "__main__":
    for km in (3, 10, 40, 120, 500, 900, 2000, 5000):
        print(f"{km:>5} km -> {optics_tier(km)}")

    # Cross-check against NVIDIA's own published measurement: Chicago (ORD) <-> Ashburn (IAD),
    # ~1000 km apart, MEASURED 21 ms round-trip.
    measured_rtt_ms = 21.0
    distance_km = 1000.0
    theoretical_one_way = one_way_delay_ms(distance_km)
    theoretical_rtt = theoretical_one_way * 2

    print(f"\nORD<->IAD: ~{distance_km:.0f} km great-circle, {measured_rtt_ms:.1f} ms measured RTT")
    print(f"Speed-of-light floor: {theoretical_one_way:.2f} ms one-way, {theoretical_rtt:.2f} ms RTT")
    overhead_ms = measured_rtt_ms - theoretical_rtt
    overhead_pct = (overhead_ms / measured_rtt_ms) * 100
    print(f"Non-physics overhead: {overhead_ms:.2f} ms ({overhead_pct:.1f}% of measured RTT) "
          f"-- real fiber routes, switching, and serialization, not a straight line")

    # Adversarial assertions: the physical floor must be a LOWER bound on any real measurement,
    # never the other way round -- if this failed, either the measured figure or the distance
    # assumption would be wrong.
    assert theoretical_rtt < measured_rtt_ms, (theoretical_rtt, measured_rtt_ms)
    assert 0 < overhead_pct < 100

    # Edge case: at 0 km, delay is exactly 0, not a divide-by-zero or negative artifact.
    assert one_way_delay_ms(0.0) == 0.0

    # Sanity bound: fiber propagation is always slower than vacuum light speed (index > 1).
    assert one_way_delay_ms(1000.0, refractive_index=1.0) < one_way_delay_ms(1000.0, refractive_index=1.468)

    print("\nALL ASSERTIONS PASSED")

Executed output:

    3 km -> FR direct-attach pluggable
   10 km -> LR direct-attach pluggable (or MetroX for InfiniBand)
   40 km -> MetroX-3 XC (InfiniBand) or 400ZR (Ethernet), both over DWDM
  120 km -> 400ZR over DWDM (InfiniBand needs an IB router hop, not MetroX)
  500 km -> OpenZR+ at 400G, or a leased/carrier DWDM line system
  900 km -> OpenZR+ at 300G (rate falls as reach grows)
 2000 km -> OpenZR+ at 200G, or a carrier long-haul transport service
 5000 km -> beyond single-hop coherent optics reach; needs regeneration or a carrier backbone

ORD<->IAD: ~1000 km great-circle, 21.0 ms measured RTT
Speed-of-light floor: 4.90 ms one-way, 9.79 ms RTT
Non-physics overhead: 11.21 ms (53.4% of measured RTT) -- real fiber routes, switching, and serialization, not a straight line

ALL ASSERTIONS PASSED

Read the 53.4% figure carefully: it is not a claim that routing is inefficient, only that the straight-line speed-of-light number is a floor, not an estimate. Real fiber follows road and right-of-way corridors, passes through regeneration and switching sites, and serializes frames onto the wire; all of that is additive to the physical floor, never subtractable from it.

How to develop with it: do you own the path end to end¶

Before comparing optics part numbers, answer one question: do you control both endpoints and everything between them?

Yes, and it is metro-scale (roughly under 40 km). Light your own or leased dark fiber with FR/LR pluggables or a DWDM line system; for InfiniBand specifically, this is the regime RDMA/InfiniBand fabric extension across metro distances covers.
Yes, but it is regional-to-continental. This is a carrier or hyperscaler-backbone problem, not a pluggable-optics problem: lease wavelengths or IP transit from a carrier, or, if operating at hyperscaler scale, build the traffic-engineered backbone Google's B4 and Meta's Express/10x Backbone exemplify.
No, at least one endpoint is a cloud region or a site you do not physically control. Use the cloud provider's own interconnect product: AWS Direct Connect (dedicated ports at "1 Gbps, 10 Gbps, 100 Gbps, and 400 Gbps"),⁸ Azure ExpressRoute (standard circuits at 50 Mbps to 10 Gbps, or ExpressRoute Direct for "dual 10-Gbps, 100-Gbps or 400-Gbps" ports, with Global Reach linking a customer's own sites through Microsoft's backbone as quoted above),⁶ or GCP Cloud Interconnect. These products terminate a customer's circuit into the provider's network; they are not a substitute for owning fiber between two of a customer's own non-cloud sites, except through a Global-Reach-style feature that explicitly re-exports the provider's backbone for that purpose.
No, and the link does not need to be owned or provider-graded at all. This is the Overlay & mesh networking case: run WireGuard over whatever internet path exists.

How to maintain it¶

DCI sits below the GPU cluster's own operational surface and is usually run by a network or optical-transport team, not the platform team that owns HPC networking fabric. Three things matter operationally regardless of who owns them:

Protection and diversity. A single fiber path is a single point of failure for an entire inter-DC link; production backbones typically run physically diverse paths with protection switching for exactly this reason. Meta's own metro fiber is pre-built as two rings rather than point-to-point runs, though Meta's own stated rationale for the ring design is standardized, scalable capacity, not explicitly fault protection; treat the protection benefit as this page's own inference from the topology, not a claim Meta itself makes.²
Optical-layer telemetry is a different signal set than a GPU fabric's. Bit-error rate, optical power budget, and DWDM channel health live in the transport layer, not in nccl-tests or ibdiagnet; a degraded but still "up" optical link can silently raise packet loss under an RDMA-lossless fabric long before a training job's own metrics point at the network.
Encryption at the optical or IP layer. NVIDIA's MetroX-3 XC advertises encryption "over long distances" as a built-in feature,⁹ and Meta secures its BAG inter-datacenter links with MACsec;⁵ a link leaving a building's physical security boundary should not be assumed trusted the way an in-rack cable is.

How to run it in production¶

The clearest published production shape is Meta's: a dedicated Ethernet layer purpose-built for inter-DC machine traffic (not shared with user-facing internet traffic, the original 2017 Express Backbone rationale),⁴ scaled for AI with distance-tiered optics and an aggregation layer, BAG (Backend Aggregation), described as "a centralized Ethernet-based super spine network layer that primarily functions to interconnect multiple spine layer fabrics across various data centers and regions," reaching "16-48 Pbps per region pair" of inter-BAG capacity, built on Jericho3 ASIC line cards ("up to 432x800G ports"), routed with eBGP and secured with MACsec, at roughly "4.5:1" L2-to-BAG oversubscription (4.98:1 for at least one named fabric).⁵ Read that oversubscription ratio as the load-bearing production fact: even a gigawatt-scale, purpose-built AI backbone is not built non-blocking end to end, so workload placement (which racks and which BAG a given job's ranks land on) still matters as much on the DCI layer as it does inside one building's fabric.

Failure modes¶

Treating DCI as a drop-in bandwidth upgrade with no latency cost. The floor is physical; see the 53.4% overhead figure above and budget accordingly for any step-synchronous collective crossing the link.
Conflating dark fiber, DWDM optics, and backbone routing. A dark-fiber lease alone carries nothing until it is lit; a lit wavelength alone carries nothing intelligently routed until a backbone (or a simple point-to-point config) exists on top of it.
Assuming a cloud interconnect product links a customer's own sites by default. AWS Direct Connect and standard Azure ExpressRoute circuits terminate into the provider's cloud, not into each other; only a specific feature (ExpressRoute Global Reach, or the analogous transit-gateway pattern on other clouds) re-exports the backbone for site-to-site traffic.
No protection path. A DCI link built as a single fiber run has no failure isolation; a fiber cut takes the inter-DC fabric down with it, exactly the failure a rail-aligned or ring topology is built to avoid at the metro layer.
Ignoring oversubscription at the aggregation layer. Even hyperscaler AI backbones run oversubscribed by design at the DCI aggregation tier; a placement that assumes uniform bandwidth to every remote rack will see stragglers that a single-building fabric would not.

References¶

Meta Engineering, "10x Backbone: How Meta is scaling backbone connectivity for AI" (distance-tiered FR/LR/ZR optics, 64x800G per fiber pair, 80-90% power reduction, AI site-pair vs global backbone scale): https://engineering.fb.com/2025/10/16/data-center-engineering/10x-backbone-how-meta-is-scaling-backbone-connectivity-for-ai/
Meta Engineering, "Building Prometheus: How Backend Aggregation enables gigawatt-scale AI clusters" (1 GW / multi-building region, BAG architecture, 16-48 Pbps inter-BAG, Jericho3 432x800G, MACsec): https://engineering.fb.com/2026/02/09/data-center-engineering/building-prometheus-how-backend-aggregation-enables-gigawatt-scale-ai-clusters/
Meta Engineering, "Building Express Backbone: Facebook's new long-haul network" (dedicated machine-traffic backbone rationale, Open/R, MPLS segment routing): https://engineering.fb.com/2017/05/01/data-center-engineering/building-express-backbone-facebook-s-new-long-haul-network/
NVIDIA Developer Blog, "Turbocharge LLM Training Across Long-Haul Data Center Networks With NVIDIA NeMo Framework" (ORD/IAD ~1,000 km, 21 ms measured RTT, Nemotron-4 340B on 3,072 GPUs, >96% throughput, 49% vs 51% MFU): https://developer.nvidia.com/blog/turbocharge-llm-training-across-long-haul-data-center-networks-with-nvidia-nemo-framework/
Juniper Networks, "400ZR and 400G OpenZR+ Transceivers" (400ZR up to 120 km; OpenZR+ 400G/500 km, 300G/900 km, 200G/2000 km): https://www.juniper.net/documentation/us/en/hardware/coherent-optics-cables-guide/optics-coherent/topics/topic-map/400zr-400openzr-transceiver.html
NVIDIA Networking, InfiniBand long-haul systems (MetroX-3 XC, up to 40 km, DWDM, encryption): https://www.nvidia.com/en-us/networking/infiniband-long-haul-systems/
Jain, Kumar, Mandal, et al., "B4: Experience with a Globally-Deployed Software Defined WAN" (SIGCOMM 2013; SDN/OpenFlow centralized traffic engineering, cited here for architecture only): https://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf
AWS Direct Connect, dedicated connection port speeds (1/10/100/400 Gbps): https://docs.aws.amazon.com/directconnect/latest/UserGuide/dedicated_connection.html
Microsoft Learn, Azure ExpressRoute overview (bandwidth options, ExpressRoute Direct, ExpressRoute Global Reach site-to-site quote): https://learn.microsoft.com/en-us/azure/expressroute/expressroute-introduction

Juniper Networks, "400ZR and 400G OpenZR+ Transceivers": https://www.juniper.net/documentation/us/en/hardware/coherent-optics-cables-guide/optics-coherent/topics/topic-map/400zr-400openzr-transceiver.html ↩↩
Meta Engineering, "10x Backbone: How Meta is scaling backbone connectivity for AI": https://engineering.fb.com/2025/10/16/data-center-engineering/10x-backbone-how-meta-is-scaling-backbone-connectivity-for-ai/ ↩↩↩↩
Jain, Kumar, Mandal, et al., "B4: Experience with a Globally-Deployed Software Defined WAN," SIGCOMM 2013: https://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf ↩
Meta Engineering, "Building Express Backbone: Facebook's new long-haul network": https://engineering.fb.com/2017/05/01/data-center-engineering/building-express-backbone-facebook-s-new-long-haul-network/ ↩↩
Meta Engineering, "Building Prometheus: How Backend Aggregation enables gigawatt-scale AI clusters": https://engineering.fb.com/2026/02/09/data-center-engineering/building-prometheus-how-backend-aggregation-enables-gigawatt-scale-ai-clusters/ ↩↩↩
Microsoft Learn, "Azure ExpressRoute Overview": https://learn.microsoft.com/en-us/azure/expressroute/expressroute-introduction ↩↩
NVIDIA Developer Blog, "Turbocharge LLM Training Across Long-Haul Data Center Networks With NVIDIA NeMo Framework": https://developer.nvidia.com/blog/turbocharge-llm-training-across-long-haul-data-center-networks-with-nvidia-nemo-framework/ ↩
AWS Direct Connect, dedicated connections: https://docs.aws.amazon.com/directconnect/latest/UserGuide/dedicated_connection.html ↩
NVIDIA Networking, InfiniBand long-haul systems: https://www.nvidia.com/en-us/networking/infiniband-long-haul-systems/ ↩