Markdown

OOB network infrastructure¶

Scope: the physically separate management network that carries lights-out control traffic: dedicated 1G switches, the management VLAN/subnet, BMC addressing via DHCP reservations, isolation from the compute/storage fabrics, and the security posture (no internet route). The SuperPOD OOB pattern is the reference.

Reference templates, not hardware-tested. Switch SKUs, VLAN IDs, subnets and protocol versions below are illustrative; verify every command, port count and address against the vendor docs in References and your own design before applying. Treat all addresses and VLAN numbers as placeholders.

flowchart LR
  BMC["Node BMC / BlueField BMC"] --> OOBSW["OOB 1G switch - SN2201"]
  PDU["Rack PDU"] --> OOBSW
  SWMGMT["Switch mgmt ports"] --> OOBSW
  COME["NVSwitch COMe"] --> OOBSW
  OOBSW --> AGG["Aggregation / spine - VXLAN"]
  AGG --> JUMP["Mgmt / jump host"]
  AGG -. "no route" .-> NET["Internet"]

What it is¶

The OOB (out-of-band) network is a dedicated Ethernet plane, physically separate from the compute and storage fabrics, that carries management traffic to every device's controller: server BMCs, BlueField BMCs, NVSwitch COMe modules, switch management ports, rack PDUs and storage controllers. It exists so an operator (or a provisioning service) can reach the BMC (power, console, sensors, firmware) independent of the host OS, the data fabric, and even a wedged kernel.

"Out-of-band" is about the path, not the protocol. The protocols that ride this plane (IPMI and Redfish) and the BMC itself (OOB management / BMC) are documented separately. This page is the network that connects them: switches, VLANs, subnet, addressing and isolation.

In the NVIDIA DGX SuperPOD reference architecture the OOB network is one of two segments on a dedicated Ethernet fabric. A B300 SuperPOD "utilize[s] four network fabrics: Compute Fabric, Storage Fabric, Ethernet Fabric with two network segments (Inband Network, Out-of-band Network)."¹ The OOB segment "connects the management ports of all devices including DGX B300 compute trays including system BMCs and BlueField-3 BMCs, switches, and management servers, storage, networking gear, rack PDUs, and all other devices."¹

Why it's needed (and when)¶

The whole point of a management plane is that it works when nothing else does. If you carry BMC traffic over the data fabric (in-band), then a fabric outage, a misconfigured route, a kernel hang or a NIC firmware fault takes out your only path to recover the node, exactly when you need it. The OOB plane is the lights-out path that survives those failures, and the first thing a runbook checks (runbook: OOB unreachable).

You need a real, separate OOB network when any of the following holds, which is to say: always, at cluster scale.

Bare-metal provisioning at scale. Network boot and image rollout drive power and console through the BMC; the provisioning tools (MAAS / Warewulf / xCAT / Ironic) expect to reach every BMC before the node has an OS. PXE bring-up and image management both ride this plane.
Lights-out recovery. A hung node has no SSH. Power-cycle, serial-over-LAN console and sensor reads happen over OOB or not at all.
Security isolation. The BMC is root-equivalent and below the OS. It must not share a broadcast domain with tenant or data traffic, and must never be internet-reachable (security and multi-tenancy).
Blast-radius control. Management traffic (sensor polling, firmware pushes, telemetry) is kept off the paths that carry collectives, so it cannot perturb training bandwidth.

When not to overbuild: a single workstation or a 2-node lab can use the BMC's shared NIC port on the same switch as host traffic. That convenience does not survive contact with a multi-rack cluster. Separate the plane before it grows.

How it's set up & managed¶

Topology and switch tier¶

The OOB network is low-bandwidth and high-fan-out: one cheap 1G copper port per managed device, aggregated per rack, rolled up to a spine. It does not need the speed of the data fabric, only reach and isolation.

The SuperPOD reference uses NVIDIA Spectrum SN2201 switches for OOB.¹ The SN2201 is a 1U switch with 48x 1GbE RJ45 (1000BASE-T) host ports and 4x 100GbE QSFP28 uplinks, 448 Gb/s switching capacity. Its product line is literally named the "1G Management Switch."²³ That port mix is the shape of an OOB ToR: 48 copper drops for BMCs/PDUs/switch-mgmt in the rack, four high-speed uplinks to aggregation.

In the SuperPOD pattern the per-rack OOB switches do not form their own L2 mesh up to a separate OOB spine; instead "the OOB network is physically rolled up into the aggregation layer (spine layer) of each SU as a dedicated VXLAN."¹ The OOB segment shares the aggregation hardware with the in-band segment but stays logically isolated as its own overlay. The in-band segment, by contrast, uses SN5610 switches and carries cluster services, lower-speed NFS and end-user access to Slurm/Kubernetes.¹ Keep the two segments distinct in your head: in-band is for services the cluster offers; OOB is for controlling the hardware.

reference template, not hardware-tested

per rack:
  [ BMC ][ BMC ] ... [ PDU ][ switch-mgmt ][ NVSwitch COMe ]
        |  1GbE RJ45 (1000BASE-T) drops
        v
  [ SN2201 OOB ToR, 48x1G + 4x100G QSFP28 ]
        |  100GbE QSFP28 uplinks
        v
  [ aggregation / spine ]  <- OOB carried as a dedicated VXLAN, separate from in-band

VLAN and subnet¶

Put BMC ports on a dedicated access VLAN with its own subnet, distinct from any host, storage or tenant VLAN. In the SuperPOD reference the OOB segment is carried on the SN2201 switches as a dedicated VXLAN on the aggregation layer.¹ Pick a VLAN ID (101 below is an arbitrary example) and a private subnet (RFC 1918) sized for the device count, and never let it route to the data planes or the internet.

# reference template, not hardware-tested
# Cumulus Linux (NVUE) on the SN2201 OOB ToR: define the BMC access VLAN,
# tag uplinks, leave host-facing swp ports as access in that VLAN.
# Verify syntax against the Cumulus Linux user guide for your release.

nv set bridge domain br_default vlan 101
nv set interface swp1-48 bridge domain br_default access 101    # BMC / PDU drops
nv set interface swp49-52 bridge domain br_default vlan 101      # 100G uplinks (trunk)
nv config apply

Subnet plan worth writing down before any cabling:

One VLAN/subnet for the OOB plane (e.g. 10.10.0.0/16), large enough for every BMC, PDU, switch-mgmt and storage controller, with headroom.
A small static block for infrastructure: the gateway/router interface, the provisioning/DHCP host, the jump host.
A DHCP pool for dynamic discovery, with the reserved addresses excluded from the pool (see below).
No default route to the internet on this subnet. Egress, if any, is via an explicit proxy or a jump host only.

BMC addressing (DHCP reservations)¶

BMCs should come up at a deterministic, known IP so provisioning and runbooks can target them by name. The clean pattern is DHCP with a per-MAC reservation: the BMC boots, requests an address, and the server hands back the same fixed IP every time, keyed on the BMC MAC. This beats hand-keying static IPs on hundreds of BMC web UIs.

# reference template, not hardware-tested
# dnsmasq on the OOB provisioning host: reserve a fixed IP per BMC MAC.
# dhcp-host=<MAC>,<IP> binds the lease; keep reservations OUT of the pool range.
# Verify against dnsmasq(8) for your version.

interface=oob0
dhcp-range=10.10.200.10,10.10.200.250,12h          # discovery pool only
dhcp-host=0c:42:a1:00:00:01,10.10.10.11,node01-bmc # reserved, outside the pool
dhcp-host=0c:42:a1:00:00:02,10.10.10.12,node02-bmc
dhcp-host=0c:42:a1:00:00:03,10.10.10.13,node03-bmc

The reserved addresses are intentionally below the dhcp-range so a dynamic lease can never collide with a reserved one.⁵ At scale, generate dhcp-host lines from inventory and load them with --dhcp-hostsfile / --dhcp-hostsdir so the server picks up new nodes without a restart.⁵ Provisioning stacks (MAAS, Ironic and friends) manage this reservation table for you and enlist BMCs by their MAC.

If a BMC must be seeded with a static address before it can reach DHCP (no PXE/DHCP yet, or an isolated bench), set it over the host with ipmitool. The LAN channel number is BMC-specific (often 1, sometimes 8); discover it before writing.

# reference template, not hardware-tested
# Configure the BMC LAN interface from the host OS via the KCS channel.
# Channel id (here 1) varies by platform; confirm with `lan print`.

ipmitool lan print 1                                   # read current LAN config

# Option A: hand the BMC over to OOB DHCP (the scaled default)
ipmitool lan set 1 ipsrc dhcp

# Option B: seed a static address on an isolated bench
ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr 10.10.10.11
ipmitool lan set 1 netmask 255.255.0.0
ipmitool lan set 1 defgw ipaddr 10.10.0.1

These lan set / lan print subcommands and their ipsrc, ipaddr, netmask, defgw ipaddr arguments are standard ipmitool LAN configuration.⁴ After switching to DHCP, wait for the BMC to acquire a lease before querying it.⁴ Misconfiguring the LAN interface can lock you out of the BMC over the network. Change it from the local host, not from the very session you would lose.⁴

Reaching it: the modern path¶

New work should target Redfish over HTTPS rather than legacy IPMI-over-LAN. Redfish exposes a single, fixed service root at /redfish/v1/, from which every managed resource (Systems, Chassis, Managers) is discoverable.⁶ Reach it only from inside the OOB plane (jump host or provisioning service); the BMC is never published to the internet.

# reference template, not hardware-tested
# Discover a BMC's Redfish service root from a host ON the OOB network.
# Real BMCs require auth (session or basic) and a valid/managed TLS cert.

curl -sk https://10.10.10.11/redfish/v1/            # service root, lists collections
curl -sk -u "$BMC_USER:$BMC_PASS" \
  https://10.10.10.11/redfish/v1/Systems            # systems collection

Isolation and security posture¶

The OOB plane is a separate network precisely so its trust boundary is hard. The non-negotiables:

No internet route. The OOB subnet has no default route off-site; BMCs are never directly reachable from outside. Egress for firmware/updates is via an explicit proxy or jump host only (security and multi-tenancy).
Separate broadcast domain. OOB is its own VLAN/subnet, not co-mingled with host, storage or tenant traffic. In SuperPOD it is a dedicated VXLAN on the aggregation layer.¹
Kill defaults. Change every BMC default credential at enlist time; default BMC creds on a reachable plane are a full out-of-band node compromise (security and multi-tenancy).
One way in. Operator and automation access funnels through a hardened jump/bastion host on the OOB subnet, not from arbitrary corporate endpoints.
Prefer Redfish+TLS over legacy IPMI where the BMC supports it; IPMI-over-LAN's cipher and auth weaknesses are well known.

This is the same posture the security page mandates: "Put the OOB network on a separate VLAN/physical segment, kill default credentials, prefer Redfish over TLS to legacy IPMI, patch BMC firmware, and never expose it to the internet." See security and multi-tenancy for the firmware/attestation surface (SPDM, signed VBIOS/GSP) that sits behind the BMC, and datacentre physical readiness for where the OOB drops land in the rack elevation.

Validated usage & tests¶

Reference template, not hardware-tested. The descriptions below are the expected shape of healthy output; no numbers here were measured on hardware. Run these from a host on the OOB subnet.

Reachability: is the BMC plane alive end to end?

# reference template, not hardware-tested
ping -c 3 10.10.10.11                  # BMC answers ICMP on the OOB subnet
ipmitool -I lanplus -H 10.10.10.11 -U "$BMC_USER" -P "$BMC_PASS" chassis status

Expected shape: ICMP replies from the reserved BMC address (not a pool address, confirming the reservation took), and a chassis status block reporting system power state and last power event. A failure here is the entry condition for runbook: OOB unreachable.

Addressing: did the reservation bind the intended IP?

# reference template, not hardware-tested
ipmitool -I lanplus -H 10.10.10.11 -U "$BMC_USER" -P "$BMC_PASS" lan print 1

Expected shape: IP Address Source reads DHCP Address (or Static Address if seeded), IP Address matches the reservation, and the subnet mask / default gateway match the OOB subnet plan. A BMC that came up on a pool address instead of its reserved one means the MAC-to-IP reservation did not match. Check the dhcp-host MAC.

Redfish service root: is the modern path answering?

# reference template, not hardware-tested
curl -sk https://10.10.10.11/redfish/v1/ | head

Expected shape: a JSON document with @odata.id of /redfish/v1/ and links to the Systems, Chassis and Managers collections.⁶ An empty body or a TLS error points at BMC firmware/cert state, not the network.

Isolation: confirm the plane does not route off-site.

# reference template, not hardware-tested
ip route get 8.8.8.8 2>&1            # expect: no route / unreachable from OOB

Expected shape: no default route toward the internet from the OOB subnet. If this resolves to a gateway, the isolation invariant is broken. Fix before the plane carries any BMC.

Failure modes¶

OOB unreachable. BMC does not answer on the management plane (cabling, VLAN/switch-port misconfig, BMC LAN channel wrong, lost lease). Lights-out recovery is impossible until restored. See runbook: OOB unreachable.
Reservation drift. A BMC lands on a dynamic pool address instead of its reserved IP (MAC typo, pool overlapping reservations), so inventory and provisioning target the wrong host. Exclude reservations from the pool; key on verified MACs.
Flat plane. OOB shares a VLAN/subnet with host or tenant traffic, collapsing the isolation boundary and exposing a root-equivalent surface (security and multi-tenancy).
Internet-reachable BMC. A default route or NAT exposes the BMC off-site; combined with default creds this is full out-of-band compromise, invisible to the OS (security and multi-tenancy).
Management traffic on the data fabric. Carrying BMC/telemetry in-band means a fabric outage removes the only recovery path exactly when it is needed.

For the broader provisioning and scheduling failure surface this plane underpins, see operational runbooks and provisioning and scheduling.

References¶

NVIDIA DGX SuperPOD B300 (XDR) — Network Fabrics (OOB on SN2201, in-band on SN5610, four fabrics, OOB rolled up as a dedicated VXLAN): https://docs.nvidia.com/dgx-superpod/reference-architecture/scalable-infrastructure-b300-xdr/latest/network-fabrics.html
NVIDIA Spectrum SN2201 — 1G Management Switch Systems User Manual: https://docs.nvidia.com/networking/display/sn2201switchesum
NVIDIA Spectrum SN2201 datasheet (48x 1GbE RJ45 + 4x QSFP28, 1U, 448 Gb/s): https://www.delltechnologies.com/asset/en-us/products/networking/technical-support/nvidia-spectrum-sn2201-datasheet.pdf
DMTF Redfish Specification DSP0266 (service root /redfish/v1/): https://www.dmtf.org/standards/redfish
ipmitool LAN configuration (lan print / lan set ipsrc|ipaddr|netmask|defgw): https://www.thomas-krenn.com/en/wiki/Configuring_IPMI_under_Linux_using_ipmitool
dnsmasq(8) dhcp-host MAC-to-IP reservations (same-subnet rule) and --dhcp-hostsfile/--dhcp-hostsdir: https://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html

NVIDIA DGX SuperPOD with DGX B300 (Quantum-X800) Reference Architecture, Network Fabrics: four fabrics (Compute, Storage, Ethernet with Inband + Out-of-band segments); "The OOB management network uses SN2201 (AC or DC) switches"; connects "system BMCs and BlueField-3 BMCs, switches, and management servers, storage, networking gear, rack PDUs, and all other devices"; "The OOB network is physically rolled up into the aggregation layer (spine layer) of each SU as a dedicated VXLAN"; in-band uses SN5610. https://docs.nvidia.com/dgx-superpod/reference-architecture/scalable-infrastructure-b300-xdr/latest/network-fabrics.html ↩↩↩↩↩↩↩
NVIDIA Spectrum SN2201 / SN2201_M 1G Management Switch Systems User Manual (product role: 1G management switch). https://docs.nvidia.com/networking/display/sn2201switchesum ↩
NVIDIA Spectrum SN2201 datasheet: 1U, 48x 1GbE RJ45 (1000BASE-T) + 4x 100GbE QSFP28, up to 448 Gb/s switching, ONIE/SONiC/Cumulus. https://www.delltechnologies.com/asset/en-us/products/networking/technical-support/nvidia-spectrum-sn2201-datasheet.pdf ↩
ipmitool LAN configuration: lan print <channel>; lan set <channel> ipsrc dhcp|static; lan set <channel> ipaddr|netmask; lan set <channel> defgw ipaddr; misconfiguring the interface can lose BMC network access; wait for DHCP to bind before querying. https://www.thomas-krenn.com/en/wiki/Configuring_IPMI_under_Linux_using_ipmitool ↩↩↩
dnsmasq(8): dhcp-host=<MAC>,<IP> reserves a fixed address per MAC; such addresses "are not constrained to be in the range given by the --dhcp-range option, but they must be in the same subnet as some valid dhcp-range"; --dhcp-hostsfile reads reservations from a file (re-read on SIGHUP) and --dhcp-hostsdir reads a directory whose "Changed or new files ... are read automatically, without the need to send SIGHUP". https://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html ↩↩
DMTF Redfish Specification (DSP0266): the service root is always at /redfish/v1/, the entry point to top-level resource collections. https://www.dmtf.org/standards/redfish ↩↩