Skip to content
Markdown

Datacenter interconnect (DCI): dark fiber, DWDM, and private backbones

Scope: the physical and optical layer that links two or more datacenter buildings, campuses, or regions so their GPU clusters can be operated as one system: dark fiber, DWDM coherent pluggable optics (400ZR/OpenZR+), and the private backbones hyperscalers build on top (Meta's 10x Backbone and Prometheus/BAG, Google's B4), plus the cloud-provider products that offer a slice of the same thing without owning fiber. This is the physical substrate beneath two different upper layers documented elsewhere: when GPUs must share one tightly-coupled RDMA fabric across buildings, see RDMA/InfiniBand fabric extension across metro distances; when a link is not owned end to end (public internet, multi-cloud, decentralized pools), see Overlay & mesh networking, which assumes a physical path already exists and secures it in software. The reading map across all three layers is GPU communication protocols: a map from GPU-to-GPU to DC-to-DC. Classification: a vendor-neutral design and reference page, not a commissioning cookbook; provisioning dark fiber, DWDM line systems, and carrier backbones is a physical, multi-vendor field-engineering process this KB cannot execute or validate end to end. It gives you the vocabulary, the optics-tier decision, the latency physics, and the acceptance artifacts to demand from whoever builds the link; the vendor-specific line-system turn-up (NMS configuration, per-channel power balancing, OTDR shot interpretation) belongs to the optical-transport team and its vendor's documentation, so the HOW sections below cover choosing an optics tier and ownership model, not turning one up end to end.

What it is

Three distinct things get bundled under "DCI," and conflating them is the most common mistake:

  1. Dark fiber: unlit glass strands, spliced end to end between two points, with no active equipment until the lessee installs their own optics. Leased from carriers or colocation providers, or owned outright by a hyperscaler that trenches its own conduit.
  2. DWDM / coherent optics: the equipment that lights the fiber, multiplexing many independent wavelengths onto one fiber pair. Current pluggable coherent optics: 400ZR (an OIF standard, whose own spec is titled "Implementation Agreement OIF-400ZR-01.0" and mandates fixed DP-16QAM modulation for cross-vendor interoperability; Juniper's own coherent-optics documentation states reach "up to 120 km")1 and OpenZR+ (an MSA, not an OIF standard, trading line rate for reach on the same QSFP-DD form factor: "400 Gbps can support a nominal reach of 500 km, 300 Gbps up to 900 km, and 200 Gbps up to 2000 km" on Juniper's QDD-400G-ZR-M/JCO400-QDD-ZR-M transceivers).1 Both eliminate the standalone transponder chassis DWDM systems used to need; Meta states its own ZR pluggables cut power "80 to 90% less" versus the transponders they replaced (10-15 W per pluggable versus up to 2 kW per standalone transponder).2
  3. The backbone network: the routing and traffic-engineering layer running over the lit fiber. Google's B4 (SIGCOMM 2013) was an early public example, an OpenFlow/SDN WAN with a centralized traffic-engineering service driving Google's own inter-datacenter links.3 Meta's Express Backbone (2017) is the same idea for a different reason: it exists because "machine-to-machine traffic often occurs in large bursts that may interfere with and impact the regular user traffic, affecting our reliability goals," so Meta runs a dedicated network for inter-datacenter machine traffic, separate from the internet-facing backbone, on Open/R distributed routing and MPLS segment routing with a centralized TE controller.4 Meta's newer 10x Backbone is the AI-era scale-up of the same architecture, with a distance-tiered optics strategy (below) and the claim that "a single AI Backbone site-pair is twice the size of the global backbone that we've been building for the last 10 years."2

Meta's own optics-tiering, stated directly in its 10x Backbone engineering post, is the clearest public disclosure of how a hyperscaler actually picks optics by distance: FR plugs for buildings roughly 3 km apart, LR plugs extending to roughly 10 km, and ZR plugs over DWDM beyond 10 km, with each fiber pair on the ZR tier carrying "64x 800G (51.2T)."2

Architecture

flowchart LR
  A["Building A<br/>GPU racks"] --> OA["DCI optics<br/>(FR/LR/ZR pluggable)"]
  OA --> F["Dark fiber<br/>(leased or owned)"]
  F --> OB["DCI optics"]
  OB --> B["Building B<br/>GPU racks"]
  OA -.->|"DWDM: many wavelengths,<br/>one fiber pair"| DW["DWDM line system<br/>(beyond ~10 km)"]
  DW -.-> OB
  B --> BB["Backbone routing<br/>(TE, segment routing / SDN)"]
  BB --> C["Building C (another region)"]

Why use it

  • There is no bit path between buildings without it. Overlay & mesh networking secures and addresses traffic over a link that already exists; it does not create the link. WireGuard over the public internet is a real, cheaper substitute when a company does not own end-to-end fiber, but it inherits whatever DCI (or lack of it) the carriers between two points actually built.
  • Power and space force multi-building designs at gigawatt scale. Meta states its Prometheus cluster "will deliver 1-gigawatt of capacity," with infrastructure that will "span several data center buildings in a single larger region, interconnecting tens of thousands of GPUs."5 No single building's electrical service covers a gigawatt; DCI is what lets that capacity be operated as one cluster rather than several unconnected ones.
  • Cloud tenants can rent a slice of a provider's own DCI instead of leasing fiber. Azure's ExpressRoute Global Reach is explicit about this: "suppose you have two private data centers, one in California and one in Texas, each connected to an ExpressRoute circuit in their respective regions. You can use ExpressRoute Global Reach to link your data centers with these circuits, and your cross data-center traffic uses the Microsoft network."6 That is a customer's own sites linked through a cloud provider's backbone, the direct cloud-product answer to "how do I connect two of my own GPU clusters without owning fiber."

When to use it (and when not)

  • Use it when a single-region footprint has outgrown one building's power or space budget (the Prometheus case), or when two sites need deterministic, provider-graded bandwidth and latency rather than best-effort internet.
  • Metro (roughly under 100 km) is where DCI pays for tightly-coupled training. NVIDIA's own published case trained Nemotron-4 340B across Chicago (ORD) and Ashburn (IAD), Oracle Cloud sites "approximately 1,000 km" apart with a "measured 21 ms" round trip, and still reached "over 96% compared to single-site baseline" throughput.7 Read that result as evidence for what careful engineering buys at that specific distance and RTT, not as a general endorsement of any inter-DC distance; the physics below explains why the same approach degrades as distance grows.
  • Do not expect DCI to erase distance. Round-trip latency has a hard speed-of-light floor. The validated model below shows the ORD/IAD floor is 9.79 ms against NVIDIA's 21 ms measured figure, meaning routing, switching, and serialization already account for more than half the measured RTT at just 1,000 km; a transcontinental or intercontinental link makes the physical floor alone dominate. Past that point the answer is not more DCI bandwidth, it is a communication-efficient algorithm: DiLoCo and the geo-distributed rank-placement cost model.
  • Do not build DCI you do not need. If a link genuinely does not need to be owned end to end (bursting to a neocloud, a decentralized pool, multi-cloud), Overlay & mesh networking over commodity internet is cheaper and faster to stand up than leasing dark fiber.

How to use it: pick the optics tier by distance

# dci_reach_and_delay.py -- validated: which DCI optics tier fits a given inter-DC distance,
# and the physical speed-of-light floor on round-trip latency, cross-checked against NVIDIA's
# own published Chicago-Ashburn measurement (~1000 km, 21 ms RTT).
# Run: python3 dci_reach_and_delay.py

def optics_tier(distance_km: float) -> str:
    """Which pluggable-optics tier reaches this distance, using the verified reach figures:
    FR (~3 km, Meta 10x Backbone), LR (~10 km, Meta 10x Backbone), 400ZR (up to 120 km,
    OIF/Juniper), OpenZR+ (400G to 500 km, 300G to 900 km, 200G to 2000 km), MetroX-3 XC
    (InfiniBand-specific long-haul appliance, up to 40 km, DWDM-based)."""
    if distance_km <= 3:
        return "FR direct-attach pluggable"
    if distance_km <= 10:
        return "LR direct-attach pluggable (or MetroX for InfiniBand)"
    if distance_km <= 40:
        return "MetroX-3 XC (InfiniBand) or 400ZR (Ethernet), both over DWDM"
    if distance_km <= 120:
        return "400ZR over DWDM (InfiniBand needs an IB router hop, not MetroX)"
    if distance_km <= 500:
        return "OpenZR+ at 400G, or a leased/carrier DWDM line system"
    if distance_km <= 900:
        return "OpenZR+ at 300G (rate falls as reach grows)"
    if distance_km <= 2000:
        return "OpenZR+ at 200G, or a carrier long-haul transport service"
    return "beyond single-hop coherent optics reach; needs regeneration or a carrier backbone"


def one_way_delay_ms(distance_km: float, refractive_index: float = 1.468) -> float:
    """Speed-of-light floor for one-way propagation in single-mode fiber (SMF-28-class,
    refractive index ~1.468). This is a physical lower bound, not a prediction of real
    latency: real paths are never straight lines and add switching/serialization delay."""
    C_KM_S = 299_792.458
    v_km_s = C_KM_S / refractive_index
    return (distance_km / v_km_s) * 1000.0


if __name__ == "__main__":
    for km in (3, 10, 40, 120, 500, 900, 2000, 5000):
        print(f"{km:>5} km -> {optics_tier(km)}")

    # Cross-check against NVIDIA's own published measurement: Chicago (ORD) <-> Ashburn (IAD),
    # ~1000 km apart, MEASURED 21 ms round-trip.
    measured_rtt_ms = 21.0
    distance_km = 1000.0
    theoretical_one_way = one_way_delay_ms(distance_km)
    theoretical_rtt = theoretical_one_way * 2

    print(f"\nORD<->IAD: ~{distance_km:.0f} km great-circle, {measured_rtt_ms:.1f} ms measured RTT")
    print(f"Speed-of-light floor: {theoretical_one_way:.2f} ms one-way, {theoretical_rtt:.2f} ms RTT")
    overhead_ms = measured_rtt_ms - theoretical_rtt
    overhead_pct = (overhead_ms / measured_rtt_ms) * 100
    print(f"Non-physics overhead: {overhead_ms:.2f} ms ({overhead_pct:.1f}% of measured RTT) "
          f"-- real fiber routes, switching, and serialization, not a straight line")

    # Adversarial assertions: the physical floor must be a LOWER bound on any real measurement,
    # never the other way round -- if this failed, either the measured figure or the distance
    # assumption would be wrong.
    assert theoretical_rtt < measured_rtt_ms, (theoretical_rtt, measured_rtt_ms)
    assert 0 < overhead_pct < 100

    # Edge case: at 0 km, delay is exactly 0, not a divide-by-zero or negative artifact.
    assert one_way_delay_ms(0.0) == 0.0

    # Sanity bound: fiber propagation is always slower than vacuum light speed (index > 1).
    assert one_way_delay_ms(1000.0, refractive_index=1.0) < one_way_delay_ms(1000.0, refractive_index=1.468)

    print("\nALL ASSERTIONS PASSED")

Executed output:

    3 km -> FR direct-attach pluggable
   10 km -> LR direct-attach pluggable (or MetroX for InfiniBand)
   40 km -> MetroX-3 XC (InfiniBand) or 400ZR (Ethernet), both over DWDM
  120 km -> 400ZR over DWDM (InfiniBand needs an IB router hop, not MetroX)
  500 km -> OpenZR+ at 400G, or a leased/carrier DWDM line system
  900 km -> OpenZR+ at 300G (rate falls as reach grows)
 2000 km -> OpenZR+ at 200G, or a carrier long-haul transport service
 5000 km -> beyond single-hop coherent optics reach; needs regeneration or a carrier backbone

ORD<->IAD: ~1000 km great-circle, 21.0 ms measured RTT
Speed-of-light floor: 4.90 ms one-way, 9.79 ms RTT
Non-physics overhead: 11.21 ms (53.4% of measured RTT) -- real fiber routes, switching, and serialization, not a straight line

ALL ASSERTIONS PASSED

Read the 53.4% figure carefully: it is not a claim that routing is inefficient, only that the straight-line speed-of-light number is a floor, not an estimate. Real fiber follows road and right-of-way corridors, passes through regeneration and switching sites, and serializes frames onto the wire; all of that is additive to the physical floor, never subtractable from it.

How to develop with it: do you own the path end to end

Before comparing optics part numbers, answer one question: do you control both endpoints and everything between them?

  • Yes, and it is metro-scale (roughly under 40 km). Light your own or leased dark fiber with FR/LR pluggables or a DWDM line system; for InfiniBand specifically, this is the regime RDMA/InfiniBand fabric extension across metro distances covers.
  • Yes, but it is regional-to-continental. This is a carrier or hyperscaler-backbone problem, not a pluggable-optics problem: lease wavelengths or IP transit from a carrier, or, if operating at hyperscaler scale, build the traffic-engineered backbone Google's B4 and Meta's Express/10x Backbone exemplify.
  • No, at least one endpoint is a cloud region or a site you do not physically control. Use the cloud provider's own interconnect product: AWS Direct Connect (dedicated ports at "1 Gbps, 10 Gbps, 100 Gbps, and 400 Gbps"),8 Azure ExpressRoute (standard circuits at 50 Mbps to 10 Gbps, or ExpressRoute Direct for "dual 10-Gbps, 100-Gbps or 400-Gbps" ports, with Global Reach linking a customer's own sites through Microsoft's backbone as quoted above),6 or GCP Cloud Interconnect. These products terminate a customer's circuit into the provider's network; they are not a substitute for owning fiber between two of a customer's own non-cloud sites, except through a Global-Reach-style feature that explicitly re-exports the provider's backbone for that purpose.
  • No, and the link does not need to be owned or provider-graded at all. This is the Overlay & mesh networking case: run WireGuard over whatever internet path exists.

How to maintain it

DCI sits below the GPU cluster's own operational surface and is usually run by a network or optical-transport team, not the platform team that owns HPC networking fabric. Three things matter operationally regardless of who owns them:

  • Protection and diversity. A single fiber path is a single point of failure for an entire inter-DC link; production backbones typically run physically diverse paths with protection switching for exactly this reason. Meta's own metro fiber is pre-built as two rings rather than point-to-point runs, though Meta's own stated rationale for the ring design is standardized, scalable capacity, not explicitly fault protection; treat the protection benefit as this page's own inference from the topology, not a claim Meta itself makes.2
  • Optical-layer telemetry is a different signal set than a GPU fabric's. Bit-error rate, optical power budget, and DWDM channel health live in the transport layer, not in nccl-tests or ibdiagnet; a degraded but still "up" optical link can silently raise packet loss under an RDMA-lossless fabric long before a training job's own metrics point at the network.
  • Encryption at the optical or IP layer. NVIDIA's MetroX-3 XC advertises encryption "over long distances" as a built-in feature,9 and Meta secures its BAG inter-datacenter links with MACsec;5 a link leaving a building's physical security boundary should not be assumed trusted the way an in-rack cable is.

How to run it in production

The clearest published production shape is Meta's: a dedicated Ethernet layer purpose-built for inter-DC machine traffic (not shared with user-facing internet traffic, the original 2017 Express Backbone rationale),4 scaled for AI with distance-tiered optics and an aggregation layer, BAG (Backend Aggregation), described as "a centralized Ethernet-based super spine network layer that primarily functions to interconnect multiple spine layer fabrics across various data centers and regions," reaching "16-48 Pbps per region pair" of inter-BAG capacity, built on Jericho3 ASIC line cards ("up to 432x800G ports"), routed with eBGP and secured with MACsec, at roughly "4.5:1" L2-to-BAG oversubscription (4.98:1 for at least one named fabric).5 Read that oversubscription ratio as the load-bearing production fact: even a gigawatt-scale, purpose-built AI backbone is not built non-blocking end to end, so workload placement (which racks and which BAG a given job's ranks land on) still matters as much on the DCI layer as it does inside one building's fabric.

The commissioning handoff: artifacts to demand and what "accepted" means

This page does not commission a link (see the classification in Scope), but the platform team that will run GPU traffic over one should refuse the handoff until these artifacts exist. Each is a standard deliverable of an optical-transport build, not an invention of this page; where a number is stated, its source is cited, and everything else is deliberately left as "demand the measured value," because the right threshold is per-link engineering, not a constant.

  • Bill of materials with pinned part numbers and firmware. Transceiver model and firmware, line-system model, amplifier/mux inventory, fiber type and length per span. Without this, a later "same" replacement pluggable can silently change reach and FEC behavior.
  • Link budget and OSNR worksheet, per span and end to end. The launch/receive power per channel against the transceiver's specified windows, and measured OSNR against the value the optics vendor requires for the chosen mode. For the coherent-pluggable FEC itself there are hard published floors to check against: 400ZR's concatenated FEC (cFEC) corrects up to a pre-FEC BER of 1.22e-2 (net coding gain 10.8 dB, DP-16QAM), and OpenZR+'s oFEC corrects up to 2e-2 (net coding gain 11.1 dB QPSK / 11.6 dB 16QAM after three soft-decision iterations).10 Acceptance means the steady-state measured pre-FEC BER sits orders of magnitude below the correction floor (so there is degradation margin), not merely that post-FEC errors are zero on day one.
  • Channel plan. Which DWDM channels are lit, at what width and center frequency, and which are reserved for growth; for a 400ZR/OpenZR+ build this is what determines whether the "add capacity later" story is a pluggable insert or a line-system redesign.
  • Routing and encryption configuration as files, not screenshots. The BGP/IGP or TE configuration for the link, and the MACsec (or optical-layer encryption) configuration with key-rotation ownership named; Meta encrypts its inter-DC BAG links with MACsec,5 and a link leaving the building boundary without an equivalent needs an explicit risk sign-off.
  • A witnessed protection-switch test. If the design claims path diversity, acceptance includes deliberately failing the primary path and recording the switch time and the packet loss the GPU fabric actually observed; an untested protection path is a hope, not a design property.
  • Rollback and escalation. Who de-commissions the link back out of the routing domain if it misbehaves after traffic lands on it, with the specific drain procedure for the GPU fabric above it (for RoCE/InfiniBand traffic that procedure lives in RDMA fabric metro extension).

The GPU-side acceptance test after the optical handoff is the same one any new fabric segment gets: sustained RDMA-level throughput and loss measurement across the link, then a collective benchmark at the message sizes training actually uses, compared against the same test run inside one building. Optical-layer green does not certify a fabric; it only makes the fabric test worth running.

Failure modes

  • Treating DCI as a drop-in bandwidth upgrade with no latency cost. The floor is physical; see the 53.4% overhead figure above and budget accordingly for any step-synchronous collective crossing the link.
  • Conflating dark fiber, DWDM optics, and backbone routing. A dark-fiber lease alone carries nothing until it is lit; a lit wavelength alone carries nothing intelligently routed until a backbone (or a simple point-to-point config) exists on top of it.
  • Assuming a cloud interconnect product links a customer's own sites by default. AWS Direct Connect and standard Azure ExpressRoute circuits terminate into the provider's cloud, not into each other; only a specific feature (ExpressRoute Global Reach, or the analogous transit-gateway pattern on other clouds) re-exports the backbone for site-to-site traffic.
  • No protection path. A DCI link built as a single fiber run has no failure isolation; a fiber cut takes the inter-DC fabric down with it, exactly the failure a rail-aligned or ring topology is built to avoid at the metro layer.
  • Accepting a link on "zero post-FEC errors" alone. Post-FEC zero is the normal state right up until the FEC cliff; the acceptance artifact that predicts trouble is the measured pre-FEC BER margin against the correction floor (1.22e-2 for 400ZR cFEC, 2e-2 for OpenZR+ oFEC),10 trended over time, not sampled once at turn-up.
  • Ignoring oversubscription at the aggregation layer. Even hyperscaler AI backbones run oversubscribed by design at the DCI aggregation tier; a placement that assumes uniform bandwidth to every remote rack will see stragglers that a single-building fabric would not.

References

  • Meta Engineering, "10x Backbone: How Meta is scaling backbone connectivity for AI" (distance-tiered FR/LR/ZR optics, 64x800G per fiber pair, 80-90% power reduction, AI site-pair vs global backbone scale): https://engineering.fb.com/2025/10/16/data-center-engineering/10x-backbone-how-meta-is-scaling-backbone-connectivity-for-ai/
  • Meta Engineering, "Building Prometheus: How Backend Aggregation enables gigawatt-scale AI clusters" (1 GW / multi-building region, BAG architecture, 16-48 Pbps inter-BAG, Jericho3 432x800G, MACsec): https://engineering.fb.com/2026/02/09/data-center-engineering/building-prometheus-how-backend-aggregation-enables-gigawatt-scale-ai-clusters/
  • Meta Engineering, "Building Express Backbone: Facebook's new long-haul network" (dedicated machine-traffic backbone rationale, Open/R, MPLS segment routing): https://engineering.fb.com/2017/05/01/data-center-engineering/building-express-backbone-facebook-s-new-long-haul-network/
  • NVIDIA Developer Blog, "Turbocharge LLM Training Across Long-Haul Data Center Networks With NVIDIA NeMo Framework" (ORD/IAD ~1,000 km, 21 ms measured RTT, Nemotron-4 340B on 3,072 GPUs, >96% throughput, 49% vs 51% MFU): https://developer.nvidia.com/blog/turbocharge-llm-training-across-long-haul-data-center-networks-with-nvidia-nemo-framework/
  • Juniper Networks, "400ZR and 400G OpenZR+ Transceivers" (400ZR up to 120 km; OpenZR+ 400G/500 km, 300G/900 km, 200G/2000 km): https://www.juniper.net/documentation/us/en/hardware/coherent-optics-cables-guide/optics-coherent/topics/topic-map/400zr-400openzr-transceiver.html
  • NVIDIA Networking, InfiniBand long-haul systems (MetroX-3 XC, up to 40 km, DWDM, encryption): https://www.nvidia.com/en-us/networking/infiniband-long-haul-systems/
  • Jain, Kumar, Mandal, et al., "B4: Experience with a Globally-Deployed Software Defined WAN" (SIGCOMM 2013; SDN/OpenFlow centralized traffic engineering, cited here for architecture only): https://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf
  • AWS Direct Connect, dedicated connection port speeds (1/10/100/400 Gbps): https://docs.aws.amazon.com/directconnect/latest/UserGuide/dedicated_connection.html
  • Microsoft Learn, Azure ExpressRoute overview (bandwidth options, ExpressRoute Direct, ExpressRoute Global Reach site-to-site quote): https://learn.microsoft.com/en-us/azure/expressroute/expressroute-introduction
  • CableLabs, "Forward Error Correction (FEC): A Primer on the Essential Element for Optical Transmission Interoperability" (cFEC pre-FEC BER 1.22e-2 / NCG 10.8 dB; oFEC pre-FEC BER 2e-2 / NCG 11.1-11.6 dB): https://www.cablelabs.com/blog/forward-error-correction-fec-a-primer-on-the-essential-element-for-optical-transmission-interoperability

Related: HPC Networking Fabric · RDMA/InfiniBand Fabric Extension Across Metro Distances · Overlay & Mesh Networking · Geo-Distributed Training Placement · DiLoCo · GPU Communication Protocols: a Map from GPU-to-GPU to DC-to-DC · Cloud, Neoclouds & Cost · Glossary


  1. Juniper Networks, "400ZR and 400G OpenZR+ Transceivers": https://www.juniper.net/documentation/us/en/hardware/coherent-optics-cables-guide/optics-coherent/topics/topic-map/400zr-400openzr-transceiver.html 

  2. Meta Engineering, "10x Backbone: How Meta is scaling backbone connectivity for AI": https://engineering.fb.com/2025/10/16/data-center-engineering/10x-backbone-how-meta-is-scaling-backbone-connectivity-for-ai/ 

  3. Jain, Kumar, Mandal, et al., "B4: Experience with a Globally-Deployed Software Defined WAN," SIGCOMM 2013: https://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf 

  4. Meta Engineering, "Building Express Backbone: Facebook's new long-haul network": https://engineering.fb.com/2017/05/01/data-center-engineering/building-express-backbone-facebook-s-new-long-haul-network/ 

  5. Meta Engineering, "Building Prometheus: How Backend Aggregation enables gigawatt-scale AI clusters": https://engineering.fb.com/2026/02/09/data-center-engineering/building-prometheus-how-backend-aggregation-enables-gigawatt-scale-ai-clusters/ 

  6. Microsoft Learn, "Azure ExpressRoute Overview": https://learn.microsoft.com/en-us/azure/expressroute/expressroute-introduction 

  7. NVIDIA Developer Blog, "Turbocharge LLM Training Across Long-Haul Data Center Networks With NVIDIA NeMo Framework": https://developer.nvidia.com/blog/turbocharge-llm-training-across-long-haul-data-center-networks-with-nvidia-nemo-framework/ 

  8. AWS Direct Connect, dedicated connections: https://docs.aws.amazon.com/directconnect/latest/UserGuide/dedicated_connection.html 

  9. NVIDIA Networking, InfiniBand long-haul systems: https://www.nvidia.com/en-us/networking/infiniband-long-haul-systems/ 

  10. CableLabs, "Forward Error Correction (FEC): A Primer on the Essential Element for Optical Transmission Interoperability" (Steve Jia): https://www.cablelabs.com/blog/forward-error-correction-fec-a-primer-on-the-essential-element-for-optical-transmission-interoperability