Skip to content
Markdown

NVIDIA DGX & HGX systems

Scope: NVIDIA's turnkey AI systems (DGX) and the OEM baseboards they share their silicon with (HGX). What makes a DGX operationally different from a self-built GPU server: the preconfigured OS, the validated firmware/driver stack, integrated NVSwitch with Fabric Manager, ConnectX/BlueField networking, and the Base Command / Mission Control management plane. Datasheet figures shift; re-check the cited NVIDIA pages before relying on any single number.

Figures verified against NVIDIA product pages and DGX user guides as of June 2026. Vendor-quoted FLOPS vary by dense-vs-sparse and peak-vs-sustained framing; confirm per system on the linked page.

What it is

A DGX system is a fully integrated NVIDIA appliance: GPUs, NVSwitch fabric, CPUs, NICs, storage, and a validated software image shipped as one supported unit. HGX is the building block underneath, an 8-GPU SXM baseboard with NVLink and NVSwitch that OEMs and "Taiwan's top server makers" build their own servers around. DGX is what NVIDIA builds on HGX and supports end to end; HGX is what Supermicro, Dell, HPE, and the ODMs ship with their own chassis, BMC, and firmware.

The practical split is who owns the integration and the support contract. With DGX, NVIDIA owns the firmware/driver matrix, the NVSwitch wiring, and the break/fix path. With HGX, the OEM owns it. With build-your-own (PCIe GPUs in a generic server), the operator owns all of it.

flowchart TB
  HGX["HGX baseboard: 8x SXM + NVLink + NVSwitch"] --> DGX["NVIDIA DGX: turnkey appliance + DGX OS + support"]
  HGX --> OEM["OEM server: Supermicro / Dell / HPE chassis + firmware"]
  PCIE["PCIe GPUs in a generic server"] --> BYO["Build-your-own: operator owns drivers, fabric, support"]
  DGX --> POD["DGX BasePOD / SuperPOD reference architecture"]
  OEM --> POD

Lineup & specifications

All node figures below are per NVIDIA DGX user guides and product pages (cited in References). NVSwitch generation tracks the GPU generation.

System GPUs NVSwitch NVLink (GPU-to-GPU) Total GPU memory Networking CPU
DGX H100 8x H100 SXM 4x (4th-gen) 900 GB/s 640 GB HBM3 4x OSFP → 8x ConnectX-7 (400 Gb/s) 2x Intel Xeon 8480C
DGX H200 8x H200 SXM 4x (4th-gen) 900 GB/s 1,128 GB HBM3e 4x OSFP → 8x ConnectX-7 (400 Gb/s) 2x Intel Xeon 8480C
DGX B200 8x Blackwell SXM 2x 14.4 TB/s aggregate 1,440 GB HBM3e (64 TB/s) 8x ConnectX-7 (400 Gb/s) + 2x BlueField-3 DPU 2x Intel Xeon Platinum 8570
DGX GB300 72x Blackwell Ultra (NVL72 rack) 9x L1 NVLink switches 1.8 TB/s per GPU (5th-gen) 20 TB HBM3e (576 TB/s) 72x ConnectX-8 (800 Gb/s) + 18x BlueField-3 36x Grace (2,592 Neoverse V2 cores)
DGX Station 1x Blackwell Ultra (GB300 desktop) n/a (single GPU) NVLink-C2C (Grace↔GPU) 748 GB coherent (252 HBM3e + 496 LPDDR5X) ConnectX-8 SuperNIC (up to 800 Gb/s) 1x Grace 72-core Neoverse V2

Performance, as quoted by NVIDIA: DGX H200 ~32 PFLOPS FP8; DGX B200 144/72 PFLOPS FP4 (sparse/dense) and 72 PFLOPS FP8; DGX GB300 1,440/1,080 PFLOPS FP4 (sparse/dense); DGX Station up to ~20 PFLOPS, models up to 1 trillion parameters. The desktop-class DGX Spark (GB10) is covered separately; see DGX Spark.

Operational differences

What separates a DGX from a rack of self-integrated GPUs is the stack NVIDIA validates and supports as one unit.

  • DGX OS (preconfigured). DGX systems ship with DGX OS 7, a customized Ubuntu 24.04 ("system-specific optimizations and configurations, additional drivers, and diagnostic and monitoring tools") on Linux kernel 6.8, with access to all NVIDIA GPU driver branches, CUDA toolkit versions, and NVIDIA DOCA OFED preintegrated. The DGX Software Stack is also downloadable to layer onto a stock Ubuntu 24.04 install. This is the same driver/CUDA/NCCL surface as the GPU software stack, but pinned and validated by NVIDIA rather than assembled by the operator.
  • Integrated NVSwitch + Fabric Manager. The on-baseboard NVSwitches form a single NVLink memory fabric across all 8 GPUs (or 72 in NVL72). Fabric Manager is the system service that "configures the NVSwitch memory fabrics to form one memory fabric among all participating GPUs and monitors the NVLinks that support the fabric." It must run for the NVLink domain to come up; it is mandatory on DGX/HGX and irrelevant on single-GPU or PCIe-only nodes. See provisioning & scheduling for where it fits in bring-up.
  • Validated firmware/driver matrix. NVIDIA qualifies a specific BMC/BIOS/VBIOS/NIC-firmware/driver combination per DGX model and ships it as a coherent release. On HGX the OEM owns this matrix; on build-your-own the operator owns it and carries the drift risk (see reliability, RAS & failure modes).
  • ConnectX / BlueField networking. DGX nodes integrate ConnectX SuperNICs (CX-7 on H100/H200/B200, CX-8 on GB300/Station) for GPUDirect RDMA east-west traffic, plus BlueField-3 DPUs on B200/GB300 for north-south/storage offload and isolation. These feed the cluster fabric described in networking fabric.
  • Base Command / Mission Control (management plane). Base Command Manager streamlines "cluster provisioning, workload management, and infrastructure monitoring" across bare metal, edge, and cloud. Mission Control is the full-stack operations layer for the newest racks (GB300 NVL72, DGX Station). This is the managed alternative to assembling Slurm/Kubernetes/observability yourself.
  • Support model. DGX carries NVIDIA Enterprise Support and NVIDIA AI Enterprise entitlement as a single SKU; the entire box (hardware + firmware + DGX OS + libraries) is one supported unit. HGX support routes through the OEM; build-your-own support is fragmented across vendors.

Feature support (MIG, vGPU, ECC, Confidential Computing) follows the GPU silicon, not the DGX wrapper. An H100 in a DGX has the same capabilities as an H100 in an HGX server. See GPU generations and Blackwell platform for the per-GPU matrix.

Install & setup

DGX systems arrive imaged; the operator re-images for fleet uniformity and verifies the fabric. The commands below are reference templates (unexecuted, not hardware-tested; pin versions to your DGX OS release).

DGX OS vs self-managed HGX

On a DGX, NVIDIA owns the validated image: DGX OS 7 (customized Ubuntu 24.04, Linux kernel 6.8) ships with the GPU driver branches, CUDA toolkit, NVIDIA DOCA-OFED, Fabric Manager, DCGM, nvidia-persistenced, and nvsm preintegrated. Re-imaging or in-place updates flow through the DGX OS / BasePOD/SuperPOD update path rather than ad-hoc apt installs.

On HGX or build-your-own, the operator assembles the equivalent stack on stock Ubuntu 24.04. The DGX Software Stack can be layered on to approximate the validated baseline; the networking layer is DOCA-OFED (NVIDIA's successor to MLNX_OFED, with the same kernel drivers, user libraries, and management tools; ConnectX-7 is the last ConnectX generation MLNX_OFED supported, so new installs use DOCA-OFED). RDMA diagnostics (ibstat via infiniband-diags) are installed alongside. See the GPU software stack and Ansible bring-up.

Firmware qualification (the part operators skip and regret)

DGX qualifies one BMC/BIOS/EROT/PCIe-Retimer/PCIe-Switch/VBIOS/NVSwitch/NIC-firmware combination per model and ships it as a coherent bundle, applied with nvfwupd. On HGX the OEM owns this matrix; on build-your-own the operator owns it and carries the drift risk.

# Confirm DGX OS / driver / CUDA baseline
cat /etc/dgx-release
nvidia-smi                       # all 8 GPUs present, expected VBIOS
nvidia-smi topo -m               # NV# links between every GPU pair (NVSwitch fabric)

# Firmware: show installed versions and the qualified bundle (DGX uses nvfwupd)
sudo nvfwupd show_version        # compare against the model's qualified firmware bundle

# Fabric Manager must be active before the NVLink domain is usable
systemctl status nvidia-fabricmanager
nvidia-smi nvlink --status       # all links up, no errors

# Persistence (prefer the daemon; legacy nvidia-smi -pm flag is slated for deprecation)
sudo systemctl enable --now nvidia-persistenced

# Health / acceptance before scheduling
sudo nvidia-smi -q | grep -i ecc           # ECC enabled (datacenter GPUs)
dcgmi diag -r 3                             # DCGM diagnostics, level 3 (extended)

Firmware updates stop the GPU services first (nvidia-fabricmanager, nvidia-dcgm, dcgm-exporter, nvidia-persistenced, nvsm); follow the model's firmware-update guide rather than improvising. Management plane: for fleet provisioning, workload management, and monitoring use Base Command Manager (BCM); the newest racks (GB300 NVL72, DGX Station) add Mission Control as the full-stack operations layer. Note the version-coupling gotcha: BasePOD/SuperPOD estates on BCM 10.x stay on DGX OS 6 (Ubuntu 22.04) until BCM 11 / DGX OS 7 (Ubuntu 24.04) is supported for the upgrade. Fleet imaging and OOB provisioning: provisioning & scheduling and Ansible bring-up.

When to use it

  • DGX when the priority is a single supported stack, predictable firmware/driver matrix, and a vendor break/fix path, and the price premium is acceptable. The default for SuperPOD/BasePOD reference deployments.
  • HGX (OEM-integrated) when an OEM relationship, chassis flexibility, or procurement terms matter more than NVIDIA owning the full stack. Firmware qualification and support then move to the OEM.
  • Build-your-own (PCIe GPUs) for cost-sensitive, smaller, or inference-tilted estates where NVLink/NVSwitch is not required and the team can own drivers, fabric, and support. Note: PCIe nodes have no NVSwitch domain and no Fabric Manager; multi-GPU collectives fall back to PCIe peer-to-peer.

Networking

DGX nodes are built for rail-aligned east-west fabrics: one ConnectX SuperNIC per GPU gives each GPU a direct RDMA path. DGX H100/H200/B200 use ConnectX-7 (up to 400 Gb/s NDR per port); DGX GB300 and DGX Station use ConnectX-8 (up to 800 Gb/s XDR). B200 and GB300 add BlueField-3 DPUs for storage and north-south offload. These NICs connect to Quantum-X800 InfiniBand or Spectrum-X Ethernet exactly as described in networking fabric; the DGX integration guarantees the NIC-to-GPU PCIe topology that GPUDirect RDMA depends on. Out-of-band/BMC management runs on a separate network (provisioning & scheduling).

There are two distinct fabrics to bring up and prove: the intra-node NVSwitch NVLink domain (Fabric Manager up, all NVLinks healthy in nvidia-smi nvlink --status) and the inter-node ConnectX/BlueField RDMA fabric (links up in ibstat, GPUDirect RDMA engaged). Do not re-document the validation command set here; follow the shared keystone Fabric bring-up, validation & benchmarking for the IB/RoCE link checks and the nccl-tests all_reduce_perf / all_gather_perf runs read against line rate. NCCL tuning pointer: on these rail-aligned fabrics, confirm GPUDirect RDMA is in use (NCCL_DEBUG=INFO should log the GDRDMA/[GPU Direct RDMA] transport, not host bounce buffers), pin NCCL_IB_HCA to the per-rail HCAs, and let SHARP in-network reduction engage on Quantum switches where present (networking fabric). Confirm bus bandwidth lands near the topology expectation before trusting any training benchmark.

Gotchas & failure modes

  • Fabric Manager down ⇒ no NVLink domain. If the nvidia-fabricmanager service fails to start (often a driver/FM version mismatch after a partial upgrade), GPUs come up but the NVSwitch fabric does not; NCCL silently degrades to PCIe. Always confirm FM is active post-upgrade (see rolling driver / CUDA upgrade).
  • DGX OS vs stock Ubuntu drift. Installing the DGX Software Stack on stock Ubuntu approximates but does not equal a DGX image; BMC/BIOS/NIC firmware are not covered and must be qualified separately.
  • NVSwitch count is not a capability dial. DGX B200 ships 2 NVSwitch ASICs vs 4 on DGX H100; this reflects switch radix per generation, not a downgrade. Compare aggregate NVLink bandwidth, not switch count.
  • DGX B200/GB300 use Intel Xeon or Grace, not interchangeably. The DGX B200 node is Intel Xeon; Grace appears in the GB300 NVL72 rack and DGX Station, not in the 8-GPU B200 node. Do not assume Grace coherent memory on a DGX B200.
  • HGX feature parity is silicon-level, support is not. An HGX H100 has identical GPU features to a DGX H100 but a different firmware owner and support path; do not assume the DGX validation transfers.
  • Confidential Computing / MIG availability is per GPU. Enable per the GPU generation (Hopper+ for CC; see security & multi-tenancy), not per the DGX label.

References

  • NVIDIA DGX platform overview: https://www.nvidia.com/en-us/data-center/dgx-platform/
  • NVIDIA DGX H100/H200 user guide (8x GPU, 4x NVSwitch, 900 GB/s): https://docs.nvidia.com/dgx/dgxh100-user-guide/introduction-to-dgxh100.html
  • NVIDIA DGX H200 product page: https://www.nvidia.com/en-us/data-center/dgx-h200/
  • NVIDIA DGX B200 product page (8x Blackwell, 2x NVSwitch, BlueField-3): https://www.nvidia.com/en-us/data-center/dgx-b200/
  • NVIDIA DGX GB300 product page (NVL72, 72x Blackwell Ultra, Mission Control): https://www.nvidia.com/en-us/data-center/dgx-gb300/
  • NVIDIA DGX Station product page (GB300 desktop, 748 GB coherent, ConnectX-8): https://www.nvidia.com/en-us/products/workstations/dgx-station/
  • NVIDIA HGX platform (8-GPU baseboard, NVLink/NVSwitch, OEM building block): https://www.nvidia.com/en-us/data-center/hgx/
  • NVIDIA DGX OS 7 user guide (Ubuntu 24.04, kernel 6.8, driver/CUDA/DOCA OFED): https://docs.nvidia.com/dgx/dgx-os-7-user-guide/index.html
  • NVIDIA Base Command Manager docs (provisioning, workload mgmt, monitoring): https://docs.nvidia.com/base-command-manager/index.html
  • NVIDIA Fabric Manager user guide (configures NVSwitch fabric on DGX/HGX): https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html
  • NVIDIA DGX H100/H200 firmware update guide (nvfwupd, qualified bundle, service stop list): https://docs.nvidia.com/dgx/dgxh100-fw-update-guide/
  • NVIDIA nvfwupd tool reference: https://docs.nvidia.com/dgx/dgxh100-fw-update-guide/nvfwupd-reference.html
  • NVIDIA MLNX_OFED to DOCA-OFED transition guide (ConnectX-7 last MLNX_OFED gen; DOCA-OFED successor): https://docs.nvidia.com/doca/sdk/mlnx_ofed-to-doca-ofed-transition-guide/index.html
  • NVIDIA DGX SuperPOD update guide — DGX OS (BCM 10.x/DGX OS 6 vs BCM 11/DGX OS 7 coupling): https://docs.nvidia.com/dgx-superpod/update-guide/latest/dgx-os.html
  • NVIDIA DCGM (data center GPU manager) diagnostics: https://docs.nvidia.com/datacenter/dcgm/latest/index.html
  • NVIDIA Mission Control (full-stack AI factory operations layer): https://www.nvidia.com/en-us/data-center/mission-control/
  • NVIDIA Mission Control documentation: https://docs.nvidia.com/mission-control/index.html

Related: DGX Spark · Blackwell platform · GPU generations · Networking fabric · Provisioning & scheduling · GPU software stack · Glossary