Markdown

Runbook: fabric qualification gate before a platform change¶

Scope: running the fabric ladder as a gate rather than as an investigation. What one qualification run has to record, how a baseline is keyed so two topologies are never compared, the parsing traps that let a run report success having measured nothing, and the decision rule that separates a node fault from a fleet-wide change without flapping.

Run this before a driver, CUDA, NCCL, NIC firmware, switch software or cabling change reaches production, when commissioning new nodes, and on a rotating sample of the fleet. If you are already in an incident, you want the regression ladder instead.

Commands are reference templates on real APIs and were not run against a fabric during authoring. nccl-tests flag availability was verified by walking the upstream release tags; DCGM suite membership and option parsing were read from NVIDIA's published reference and the DCGM source; perftest units were read from perftest_parameters.h. The Python block was executed; its output is pasted verbatim.

The ladder in the regression runbook answers "which layer diverged". A gate answers a narrower question under time pressure: may this change proceed. That difference is why a gate needs things an investigation does not, namely a machine-readable record, a baseline keyed to a topology rather than a number, and a rule that is stable enough to run unattended. The rung procedures themselves are not repeated here; neither are the DCGM run levels, the standing per-pair matrix, the node-level scheduler plumbing, or the acceptance framing.

The failure this runbook is built around is not a slow fabric. It is a gate that says pass because it never measured anything.

Trigger¶

A driver, CUDA, NCCL, GPU firmware, NIC firmware or switch software change is staged for the fleet.
New nodes are being commissioned, or capacity is being added (capacity add).
Cabling or topology changed, including a rail moved between leaves.
A scheduled re-qualification of a rotating fleet sample is due.
A distributed workload regressed and the fleet needs re-baselining after the fix.

Pre-checks¶

The nodes under test are drained. The level-3 DCGM plugins are explicit that they need idle GPUs: the diagnostic plugin "should run on GPUs that are not serving production workloads", and targeted_stress and targeted_power "should run on idle GPUs".¹ Level 4 escalates that to an idle system. This is the opposite of the pre-job dispatch gate, which stays at level 1 precisely because the node is busy (GPU health gating).
A baseline exists for this exact topology class. If it does not, this run is establishing one, not gating against one. Say which it is.
The tool versions are pinned and recorded, because two of the gate's inputs are version-dependent in ways that fail silently. See the parsing traps below.
DCGM_NCCL_TESTS_BIN_PATH is set, in the host engine's environment, if you intend the DCGM nccl_tests plugin to run. NVIDIA requires it to be set before nv-hostengine or the nvidia-dcgm service starts, because the plugin reads it in the host-engine process; exporting it in the operator's shell before dcgmi diag has no effect. The two failure shapes differ and only one of them is visible: with the variable unset the plugin registers no test at all, so nccl_tests is simply absent from the report, while a variable pointing at a missing or unusable path produces a genuine skip result.⁴

Flow¶

flowchart TB
    A["Change staged"] --> B["Drain sample nodes"]
    B --> C["Run suite: DCGM, nccl-tests, perftest"]
    C --> D["Parse into records"]
    D --> E{"Did every step<br/>actually measure?"}
    E -->|"no"| F["no-measurement: fix the harness"]
    E -->|"yes"| G{"Same topology class<br/>as the baseline?"}
    G -->|"no"| H["incomparable: get the right baseline"]
    G -->|"yes"| I{"Cohort moved too?"}
    I -->|"yes"| J["fleet-regression: blame the change"]
    I -->|"no"| K["pass / warn / block on this node"]

Procedure¶

1. Pin the profile¶

Record the same profile the ladder's rung 0 records, and treat it as the baseline key rather than as metadata. A baseline is a sentence, not a number: "8 x H100 SXM, two nodes, one rail" and "8 x H100 SXM, eight nodes, four rails" are different baselines and never versions of each other.

2. Run the suite on drained nodes¶

dcgmi diag -r 3 -j --fail-early > diag.json

-j is a boolean switch, not a path. --fail-early is long-only and has no short form, and --check-interval is rejected unless --fail-early is also set. The level aliases that parse are quick, short, medium, long and xlong; extended appears in legacy help text but is not accepted by the parser, so -r extended is read as a test name and fails.²

Then the collective and RDMA rungs, using the same invocations that produced the baseline (regression ladder, rungs 2 to 5) and the error-counter procedure around them.

3. Parse into records, and treat every parse failure as a failure¶

Four traps turn a green gate into a meaningless one. Each is silent.

An unknown nccl-tests flag exits 0 without benchmarking. The per-iteration timing flag -I first appears in release v2.19.2, and -K in v2.19.6; both are absent from every earlier tag. Given an option it does not recognise, getopt_long returns '?', so glibc writes invalid option -- 'I' to stderr, the binary then prints invalid option '?' to stdout, and it executes return 0 after the usage block.³ A harness keyed on exit status records a successful qualification run containing no measurement. Gate on a parsed number, never on an exit code, and probe the binary first:

./build/all_reduce_perf -h | grep -q per_iter_timing || echo "no -I on this build"

#wrong reads N/A rather than 0 when checking is off. Correctness reporting follows the check-iteration count, so -c 0 prints N/A in that column. Parsing N/A as zero claims correctness was verified when it was never checked.

A skipped DCGM test is not a passed one. Read per-test status out of the JSON rather than the overall verdict, which does not distinguish them.

perftest reports MiB/sec by default, not MB/sec. On current releases the header reads BW peak[MiB/sec] and BW average[MiB/sec], and --report_gbits switches it to BW peak[Gb/sec]; builds at or before the 24.04 release printed MB/sec for the identical mebibyte value, so read the header rather than assuming it. perftest's own help gives the conversion as Factor = 10^9/(2^20*8) = 119.2.⁶ Treating a MiB/sec figure as MB/sec and multiplying by 8 divides by 125 instead of 119.2, which reads about 4.6% low: 12,000 MiB/s is 100.7 Gb/s and the mistake reports 96.0. That is inside the range a gate is trying to resolve, and it errs toward failing healthy hardware.

For DCGM specifically, the exit code carries information worth keeping: 226 means the diagnostic ran and reported an error, 205 means it reported a condition that requires isolation, 217 means another diagnostic was already running, and 204 means the NVVS binary was not found.⁵ 226 and 205 are node verdicts and 205 is the stronger of the two, since DCGM is asking for the GPU to be isolated; 217 and 204 are harness problems and must not be recorded as node failures. If dcgmi itself is killed by a signal the shell reports 128 plus the signal number, which describes the process rather than any diagnostic result.

4. Decide¶

The rule is in the executed block below. It is deliberately conservative in one direction: anything it cannot interpret blocks rather than passes.

5. Land the verdict¶

A verdict that does not change scheduling is a report, not a gate. Every verdict the rule can return needs a landing place:

Verdict	Where it lands
`corrupt`	Stop the rollout. Wrong answers outrank every performance question.
`block`	Cordon the node and route to the regression ladder.
`fleet-regression`	Escalate to the change owner, not the node's owner. Do not cordon the fleet.
`warn`	Record and re-measure. Repeated warnings escalate rather than accumulate silently.
`no-measurement`, `diag-incomplete`, `no-baseline`, `unstable`, `implausible`	Hold in inventory and fix the harness. These are not clean nodes; they are absent measurements.
`correctness-unverified`	Re-run with checking on. Never release on it.
`incomparable`	Get the right baseline, or establish one and say so.

The decision rule¶

# gate.py -- validated: decide a verdict for one qualification run against a baseline,
# and refuse to pass anything the run did not actually measure.
# numpy only.
import numpy as np

WARN, BLOCK = 0.05, 0.10          # matches the fabric-regression runbook's policy table
# Chosen conventions, not derived quantities. Both bound "the run measured something
# other than what the baseline measured" rather than "the fabric is slow".
FAST, SLOW = 1.5, 0.25
# The gate asserts on a NAMED SET, because a test that never registered contributes no
# status at all and an absent status is not a failed one.
REQUIRED_DIAG = ("pcie", "memory", "nccl_tests")


def _drop(samples, baseline_median):
    """Fractional shortfall of this run's median against the baseline median."""
    return 1.0 - float(np.median(samples)) / baseline_median


def _noise_bar(cv, n):
    """Two standard errors of the baseline's own run-to-run spread.

    cv/sqrt(n) is the standard error of the MEAN, used here for the median's, which is
    wider by about 25% on normal data. The substitution makes the bar mildly permissive.
    It is the requirement to repeat, not the constant, that carries the argument.
    """
    return 2.0 * cv / np.sqrt(n)


def _significant(drop, bar, cv, n):
    """A shortfall counts only if it clears both the policy bar and the measurement noise."""
    return drop > bar and drop > _noise_bar(cv, n)


def _unmeasured(record, baseline):
    """Every way a run can fail to be a measurement. Checked before any arithmetic."""
    if not record.get("samples"):
        return "no-measurement"                       # exit status 0 is not a measurement
    if not baseline.get("median"):
        return "no-baseline"                          # absent or zero: nothing to divide by
    if record.get("wrong") is None:
        return "correctness-unverified"               # the #wrong column read N/A
    if record["wrong"] > 0:
        return "corrupt"                              # baseline-independent, so checked early
    diag = record.get("diag") or {}
    if any(t not in diag for t in REQUIRED_DIAG) or any(s != "pass" for s in diag.values()):
        return "diag-incomplete"
    if record.get("class") != baseline.get("class"):
        return "incomparable"                         # a baseline is a topology, not a number
    spread = float(np.std(record["samples"]) / np.mean(record["samples"]))
    if len(record["samples"]) > 1 and spread > 3.0 * baseline["cv"]:
        return "unstable"                             # the run disagrees with itself
    return None


def gate(record, baseline, cohort_drops=()):
    """Verdict for one candidate measurement. Anything not clearly measured blocks."""
    unmeasured = _unmeasured(record, baseline)
    if unmeasured:
        return unmeasured

    ratio = float(np.median(record["samples"])) / baseline["median"]
    if ratio > FAST or ratio < SLOW:
        return "implausible"

    drop, n = _drop(record["samples"], baseline["median"]), len(record["samples"])
    if not _significant(drop, WARN, baseline["cv"], n):
        return "pass"
    # A fleet event excuses a node only if the node is INDISTINGUISHABLE from the fleet.
    # Comparing against the cohort median alone excuses an arbitrarily bad node.
    if len(cohort_drops) >= 5 and np.mean(np.asarray(cohort_drops) > WARN) >= 0.6:
        if drop - float(np.median(cohort_drops)) <= _noise_bar(baseline["cv"], n):
            return "fleet-regression"
    if _significant(drop, BLOCK, baseline["cv"], n):
        return "block"
    return "warn"


def with_hysteresis(verdict, history, clear_after=2, escalate_after=3):
    """Damp both edges. One good run does not clear a block; warnings accumulate."""
    recent = list(history)
    if recent and recent[-1] == "block":
        clean = verdict == "pass" and all(v == "pass" for v in recent[-clear_after + 1:])
        return "pass" if clean and clear_after > 1 and len(recent) >= clear_after - 1 else "block"
    if verdict == "warn" and recent[-escalate_after + 1:].count("warn") >= escalate_after - 1:
        return "block"
    return verdict


BASE = {"class": "8xH100-SXM/2node/1rail", "median": 240.0, "cv": 0.04}


def rec(**kw):
    r = {"class": BASE["class"], "samples": [240.0] * 9, "wrong": 0,
         "diag": {t: "pass" for t in REQUIRED_DIAG}}
    r.update(kw)
    return r


# (1) A run at baseline passes.
assert gate(rec(), BASE) == "pass"

# (2) Adversarial, and the reason this gate exists: nccl-tests answers an unknown option
# by printing usage and returning 0. A gate keyed on exit status records a pass having
# measured nothing. No samples is never a pass.
assert gate(rec(samples=[]), BASE) == "no-measurement"

# (3) Adversarial: an ABSENT diagnostic block must not read as a pass. DCGM does not
# report a skip when DCGM_NCCL_TESTS_BIN_PATH is unset; the plugin registers no test at
# all, so its status is missing rather than failed, and `any()` over an empty dict is
# False. The gate therefore asserts on a named set, not on the statuses it happens to find.
bare = rec()
del bare["diag"]
assert gate(bare, BASE) == "diag-incomplete"
assert gate(rec(diag={"pcie": "pass", "memory": "pass"}), BASE) == "diag-incomplete"

# (4) A test that ran and was skipped, which is what a WRONG path produces, is also not a pass.
assert gate(rec(diag={"pcie": "pass", "memory": "pass", "nccl_tests": "skip"}), BASE) == "diag-incomplete"

# (5) With -c 0 the #wrong column prints N/A. Reading absent as zero claims correctness
# was verified when it was never checked.
assert gate(rec(wrong=None), BASE) == "correctness-unverified"

# (6) Adversarial ordering: data corruption needs no baseline to be meaningful, so it must
# not be masked by a mismatched topology. Checking the class first would report this run as
# merely incomparable and discard the corruption signal.
assert gate(rec(wrong=3, **{"class": "8xH100-SXM/8node/4rail"}), BASE) == "corrupt"

# (7) A baseline is a topology, not a number. An 8-node result against a 2-node baseline
# compares two systems, and the arithmetic works perfectly on both.
assert gate(rec(**{"class": "8xH100-SXM/8node/4rail"}), BASE) == "incomparable"

# (8) A missing or zero baseline divides by nothing. Fail closed rather than raise.
assert gate(rec(), {"class": BASE["class"], "median": 0.0, "cv": 0.04}) == "no-baseline"

# (9) Adversarial: a run that disagrees with ITSELF has not measured a stable quantity,
# even though its median sits exactly on baseline and the sample count looks reassuring.
assert gate(rec(samples=[100., 100., 240., 380., 380.]), BASE) == "unstable"

# (10) Too good is a defect, not a win, and so is absurdly bad: both mean the run measured
# something other than what the baseline measured. 1.5 and 0.25 are chosen conventions.
assert gate(rec(samples=[380.0] * 9), BASE) == "implausible"
assert gate(rec(samples=[1.0] * 9), BASE) == "implausible"     # a parser fault, not a fabric fault

# (11) Adversarial statistics: a 7% shortfall from ONE sample, on a fleet whose own
# run-to-run cv is 4%, is inside the noise: one sample admits nothing under 8%. The policy
# bar alone would raise a warning and send someone to investigate a fabric that is fine.
one = rec(samples=[223.2])                       # 7% under 240
assert _drop(one["samples"], BASE["median"]) > WARN     # the bar says warn
assert gate(one, BASE) == "pass"                        # the noise says nothing happened

# (12) The same 7% measured nine times is a finding: nine samples narrow the noise bar from
# 8% to 2.7%. Repetition, not a lower threshold, turns a suspicion into a verdict.
assert gate(rec(samples=[223.2] * 9), BASE) == "warn"

# (12b) The noise bar is a strict inequality, so a shortfall landing exactly on two
# standard errors reads as pass. Do not build a demonstration on that boundary: a value
# nominally equal to 2 SE can compute either side of it depending on how it was produced.
assert _significant(0.08000001, WARN, 0.04, 1) and not _significant(0.08, WARN, 0.04, 1)

# (13) Adversarial attribution: a 12% drop the whole cohort shares is a fleet event, a
# driver or NCCL rollout, not this node. Blaming the node sends a healthy card to RMA.
fleet = (0.12, 0.115, 0.125, 0.118, 0.122)
assert gate(rec(samples=[211.2] * 9), BASE, cohort_drops=fleet) == "fleet-regression"

# (14) Adversarial: the fleet event must not become an alibi for an arbitrarily bad node.
# Comparing only against the cohort MEDIAN excuses a node at half of baseline because the
# fleet moved 6%. The node has to be indistinguishable from the fleet, not merely joined to it.
mild = (0.06, 0.055, 0.051, 0.06, 0.058)
assert gate(rec(samples=[120.0] * 9), BASE, cohort_drops=mild) == "block"

# (15) Adversarial: two faulty nodes in a three-member cohort are a majority of its median,
# so a small cohort lets the second faulty node alibi the first. Requiring five members and
# a clear majority past the bar is what stops that, not a count of three.
assert gate(rec(samples=[211.2] * 9), BASE, cohort_drops=(0.12, 0.11, 0.0)) == "block"

# (16) The identical measurement against a healthy cohort blocks the node.
assert gate(rec(samples=[211.2] * 9), BASE, cohort_drops=(0.0, 0.01, 0.0, 0.005, 0.0)) == "block"

# (17) Hysteresis damps both edges. One good run does not clear a block, and warnings
# accumulate into one. Without the first, a node near the bar rejoins the fleet every
# other run; without the second, a node warns forever and is never acted on.
assert with_hysteresis("pass", ["block"]) == "block"
assert with_hysteresis("pass", ["block", "pass"]) == "pass"
assert with_hysteresis("warn", ["warn", "warn"]) == "block"

ROWS = [
    ("at baseline",                 rec(),                                      ()),
    ("exit 0, nothing parsed",      rec(samples=[]),                            ()),
    ("diag block absent",           bare,                                       ()),
    ("nccl_tests skipped",          rec(diag={"pcie": "pass", "memory": "pass",
                                              "nccl_tests": "skip"}),           ()),
    ("#wrong read N/A",             rec(wrong=None),                            ()),
    ("corrupt + wrong baseline",    rec(wrong=3, **{"class": "other"}),          ()),
    ("run disagrees with itself",   rec(samples=[100., 100., 240., 380., 380.]), ()),
    ("0.4% of baseline",            rec(samples=[1.0] * 9),                     ()),
    ("7% down, 1 sample",           rec(samples=[223.2]),                       ()),
    ("7% down, 9 samples",          rec(samples=[223.2] * 9),                   ()),
    ("12% down, cohort also down",  rec(samples=[211.2] * 9),                 fleet),
    ("50% down, cohort mildly down", rec(samples=[120.0] * 9),                 mild),
    ("12% down, cohort healthy",    rec(samples=[211.2] * 9), (0.0, 0.01, 0.0, 0.005, 0.0)),
]
print(f"{'case':32s} {'n':>2s}  verdict")
for label, r, coh in ROWS:
    print(f"{label:32s} {len(r.get('samples', [])):>2d}  {gate(r, BASE, coh)}")
print()
print("all assertions passed")

Executed output:

case                              n  verdict
at baseline                       9  pass
exit 0, nothing parsed            0  no-measurement
diag block absent                 9  diag-incomplete
nccl_tests skipped                9  diag-incomplete
#wrong read N/A                   9  correctness-unverified
corrupt + wrong baseline          9  corrupt
run disagrees with itself         5  unstable
0.4% of baseline                  9  implausible
7% down, 1 sample                 1  pass
7% down, 9 samples                9  warn
12% down, cohort also down        9  fleet-regression
50% down, cohort mildly down      9  block
12% down, cohort healthy          9  block

all assertions passed

Six of those rows are why the rule is code rather than a threshold in a wiki page.

exit 0, nothing parsed is the failure mode that motivates the whole gate. Every arithmetic check downstream is correct, and the run measured nothing. This is not hypothetical: it is what an older nccl-tests binary does when handed -I 1.

#wrong read N/A, diag block absent and nccl_tests skipped are the same defect in three shapes. In each the absence of a result is representable as something that looks like a pass, and the gate has to distinguish "checked and fine" from "not checked". The diag block absent row is the sharpest: any() over an empty collection is False, so a diagnostic section that never ran passes every status check written as a filter over the statuses present. That is why the rule asserts on a named set of required tests instead.

The two 7% down rows are the same shortfall measured once and measured nine times, and they get opposite verdicts. On a fleet whose own run-to-run coefficient of variation is 4%, one sample carries a 4% standard error, so the noise bar sits at 8% and a single run admits nothing below it. Nine samples narrow that bar to 2.7% and the same 7% becomes a finding. The policy bar alone would have warned on the single sample and sent someone to investigate a healthy fabric, which is why the gate specifies a sample count rather than only a percentage. Two honest limits on this: the noise bar is a strict inequality, so a shortfall landing exactly on two standard errors reads as pass, and a value nominally equal to 2 SE can compute either side of that point depending on how it was derived, which is why the assertions avoid demonstrating on it; and cv/sqrt(n) is the standard error of the mean standing in for the median's, which is about 25% wider on normal data, so the bar is mildly permissive.

The 12% down rows are identical measurements on the node, differing only in what the rest of the cohort did. When the whole cohort moved together the node is not the story and the change is, and blaming the node sends a healthy card to RMA while leaving the actual regression deployed. Two guards keep that from becoming an excuse. The node must be indistinguishable from the cohort, not merely accompanied by it: the 50% down row is joined to a fleet that moved 6%, and grading it against the cohort median alone would excuse a node running at half of baseline. And the cohort must be large enough that a majority is not two bad nodes: with three members, two faults are the median, so the second faulty node alibis the first, which is why the rule wants five members with a clear majority past the bar.

Verification¶

The gate reproduces its own verdict on a re-run of the same profile against the same baseline.
Every record contains a parsed number, and the count of records equals the count of runs attempted. A missing record is a harness failure, not a silent pass.
A deliberately broken run is caught. Point the harness at a binary without -I support, or unset DCGM_NCCL_TESTS_BIN_PATH, and confirm the gate returns no-measurement or diag-incomplete rather than pass. A gate that has never failed has not been tested.
The baseline used is the one for this topology class, confirmed by reading the key rather than trusting the lookup.
block actually cordons. Confirm the scheduler state changed, not just that a line was logged.

Rollback¶

The gate blocks changes; rolling it back means letting a change through, so treat an override as a decision with a name attached.

An override needs a recorded reason and an expiry. A permanently overridden gate is a gate that has been deleted slowly.
If the gate itself is wrong, fix the parser or the baseline and re-run, rather than lowering the bar. A threshold lowered to make a fleet pass tells you nothing on the next change.
If a baseline turns out to be stale, re-establish it deliberately on known-good hardware and record when and on what it was taken. Re-baselining on top of an undiagnosed regression bakes the regression into every future comparison.
Cordoned nodes are released only after the rung that failed is back at baseline (regression ladder).

NCCL / fabric performance regression: the ladder this gate automates, and where you go when the gate fails.
Fabric link errors and congestion counters: the counter procedure that runs alongside each rung.
Add GPU capacity: per-node admission for new hardware.
NCCL socket fallback: the transport check a gate should include.
operational runbooks: the runbook index.

References¶

NVIDIA nccl-tests, argument parsing and the unknown-option path: https://github.com/NVIDIA/nccl-tests/blob/master/src/common.cu
NVIDIA nccl-tests releases (the tags at which -I and -K first appear): https://github.com/NVIDIA/nccl-tests/tags
NVIDIA DCGM diagnostics, suite levels and plugin membership: https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/dcgm-diagnostics.html
NVIDIA dcgmi diag command reference (level aliases, -j, --fail-early, --check-interval, -p, exit codes): https://docs.nvidia.com/datacenter/dcgm/latest/reference/command-line-reference/dcgmi/dcgmi-diag.html
NVIDIA DCGM source, suite-to-plugin mapping: https://github.com/NVIDIA/DCGM
linux-rdma perftest, result format and unit handling: https://github.com/linux-rdma/perftest/blob/master/src/perftest_parameters.h
NVIDIA NCCL troubleshooting, performance and tuning: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/performance_and_tuning.html

NVIDIA DCGM plugin references, per-plugin prerequisites. Diagnostics: "The plugin requires CUDA support and should run on GPUs that are not serving production workloads because it applies sustained compute, memory, and power load" (https://docs.nvidia.com/datacenter/dcgm/latest/reference/diagnostics/plugins/diagnostic.html). Targeted Stress: "should run on idle GPUs so that competing compute work does not distort the achieved-performance result" (https://docs.nvidia.com/datacenter/dcgm/latest/reference/diagnostics/plugins/targeted-stress.html). Targeted Power: "Run it on idle GPUs: competing workloads and an enforced power limit below the configured target can prevent a meaningful target-power result" (https://docs.nvidia.com/datacenter/dcgm/latest/reference/diagnostics/plugins/targeted-power.html). DCGM states no level-based drain rule; the requirement is per plugin, and the level-3 set is where these plugins first appear. ↩
NVIDIA dcgmi diag reference: "The names quick and short, medium, long, and xlong are aliases for levels 1 through 4, respectively." Level 1 runs the built-in software deployment checks; level 2 adds memory and pcie; level 3 adds diagnostic, memory_bandwidth, targeted_stress, targeted_power, nvbandwidth and nccl_tests, plus the EUD plugins when DCGM runs as root; level 4 adds memtest and pulse_test. NVIDIA's user-guide table and its dcgmi diag reference disagree over whether memory_bandwidth runs at level 2 or level 3; the reference and the DCGM source both place it at level 3 and above. https://docs.nvidia.com/datacenter/dcgm/latest/reference/command-line-reference/dcgmi/dcgmi-diag.html ↩
Verified by walking the upstream tags: the getopt_long option string in src/common.cu first contains I: at v2.19.2 and K: at v2.19.6, and neither appears at v2.19.1 or earlier. At v2.19.1 the parser's fallthrough reads case 'h': default: if (c != 'h') printf("invalid option '%c'\n", c); followed by the usage block and return 0, so an unrecognised option terminates the process with status 0 and no benchmark. Note c there is getopt_long's return value for an unknown option, which is '?' rather than the offending letter, and opterr is never cleared, so glibc emits its own invalid option -- 'I' on stderr first. Reproduced locally by compiling the same optstring and default arm. https://github.com/NVIDIA/nccl-tests/blob/master/src/common.cu ↩
DCGM source, nvvs/plugin_src/nccl_tests/NcclTestsWrapper.cpp: info->numTests = 0; if (SupportNcclTestsPlugin()) { ... }, where SupportNcclTestsPlugin() returns whether the DCGM_NCCL_TESTS_BIN_PATH environment variable exists. With the variable unset no test is registered, so the suite omits nccl_tests entirely rather than reporting it skipped; the SetResult(testName, NVVS_RESULT_SKIP) paths in NcclTestsPlugin.cpp are reached when the variable is set but the path is missing or unusable. https://github.com/NVIDIA/DCGM ↩
NVIDIA dcgmi diag reference, exit codes: 226 DCGM_ST_NVVS_ERROR ("The diagnostic ran but reported an error"), 205 DCGM_ST_NVVS_ISOLATE_ERROR ("The diagnostic reported a condition that requires isolation"), 217 DCGM_ST_DIAG_ALREADY_RUNNING, 204 DCGM_ST_NVVS_BINARY_NOT_FOUND, and 203 DCGM_ST_NVVS_KILLED for a diagnostic process terminated by a signal (https://docs.nvidia.com/datacenter/dcgm/latest/reference/command-line-reference/dcgmi/dcgmi-diag.html). The 128-plus-signal convention concerns dcgmi itself and is documented separately: "If dcgmi is terminated by a signal, POSIX-style shells conventionally report 128 plus the signal number, such as 130 for SIGINT and 143 for SIGTERM. Those values describe process termination rather than a dcgmReturn_t result." https://docs.nvidia.com/datacenter/dcgm/latest/reference/command-line-reference/dcgmi/index.html ↩
linux-rdma perftest, src/perftest_parameters.h: #define RESULT_FMT " #bytes #iterations BW peak[MiB/sec] BW average[MiB/sec] MsgRate[Mpps]" and #define RESULT_FMT_G " #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]". The --report_gbits help text states: "Note: MiB=2^20 byte, while Gb=10^9 bits. Use these formulas for conversion: Factor=10^9/(2^20*8)=119.2". Builds at or before the 24.04 release labelled the same value MB/sec; the value was always mebibytes and only the label was corrected. https://github.com/linux-rdma/perftest/blob/master/src/perftest_parameters.h ↩