Markdown

Runbook: fabric link errors and congestion counters¶

Scope: turning InfiniBand and RoCE error, discard and congestion counters into one defensible verdict. Which counter to read, why the reading must be a delta measured across a controlled test rather than a value read once, how counter saturation makes a failing link report clean, and how to separate a physical-layer fault from a lossless-fabric failure from ordinary congestion.

Run this when a fabric is suspected but not yet localised: retries or discards are climbing, a link has flapped, a rail is slower than its peers, or a RoCE estate has lost throughput with every link still up. Read counters here; measure bandwidth elsewhere.

Commands are reference templates on real APIs and were not run against a fabric during authoring. Counter names and widths were read from source: libibmad/fields.c and infiniband-diags/perfquery.c in rdma-core, and drivers/infiniband/hw/mlx5/counters.c plus drivers/net/ethernet/mellanox/mlx5/core/en_stats.c in the Linux kernel. The Python block was executed; its output is pasted verbatim.

This runbook is rung 4b of the fabric performance regression ladder expanded into its own procedure. That page decides which layer has diverged from baseline; this one is what you run once the answer is "the fabric" and you need to know which part of it. Bandwidth and latency benchmarking belongs to fabric bring-up and benchmarking, the PFC, ECN and DCQCN configuration that produces these counters belongs to RDMA and RoCE tuning, and transport selection belongs to the socket-fallback runbook. None of those are re-derived here.

Two ideas carry the whole procedure. A counter is evidence only as a delta measured across a controlled test, because a lifetime value records every event since the port last reset and says nothing about this incident. And counters have a precedence order set by causation, not by magnitude: a marginal optic manufactures the transport retries that follow from it, so the largest number in the output is usually the symptom.

One deliberate choice to be explicit about: error counters are graded here at zero tolerance, which is a conservative convention for a fleet gate rather than a property of healthy hardware. NVIDIA's own ibdiagnet grades the physical layer against a bit-error-rate threshold, not against zero, so a link with a handful of corrected symbol errors can sit inside the vendor's pass bar and still trip this rule.⁴ Zero tolerance is the right default when the question is whether a change may ship; it is the wrong default for deciding whether to replace hardware, which is what the physical-layer step exists for.

Trigger¶

A performance-regression ladder run localised the fault to the fabric, at rung 3, 4 or 5.
ibstat shows a port down or polling, or a Rate: below the fabric's negotiated rate. It carries no flap history: a port that bounced and recovered looks identical to one that never did, which is what LinkDownedCounter is for, subject to the saturation caveat below.
A RoCE estate lost throughput with every link reporting up, which is the signature of congestion control or lossless configuration rather than hardware.
#wrong non-zero on an nccl-tests run, which is data corruption and outranks every performance question.
One rail or one node pair is worse than its peers in a per-pair sweep, and you need to decide between the endpoint, the cable and the switch port.

Pre-checks¶

Know which side you are reading. RDMA counters and Ethernet counters are different namespaces on the same card. The CNP counters live on the RDMA device under /sys/class/infiniband/<dev>/ports/<n>/hw_counters/; PFC pause, physical discards and the per-priority ECN-marking counters live on the netdev under ethtool -S <nic>. Reading one and reporting the other is a common and confident error, and an ethtool -S grep for cnp returns nothing on an upstream mlx5 driver.
Have a controlled test to measure across. A delta needs traffic behind it and a known byte count. Use the same ib_write_bw or nccl-tests invocation that produced your baseline, not ambient production traffic, whose volume you cannot state.
Do not reset counters on a fabric carrying production traffic without agreement. perfquery -R clears state that another team may be mid-investigation on.
ibdiagnet is usually the network operator's tool, not the workload owner's. It needs subnet-manager access. Ask rather than assume you may run it.

Flow¶

flowchart TB
    A["Fabric suspected"] --> B["Snapshot counters"]
    B --> C["Run the controlled test"]
    C --> D["Snapshot again, diff"]
    D --> E{"Any counter pegged<br/>or reset mid-test?"}
    E -->|"yes"| F["Unusable: reset, re-measure"]
    E -->|"no"| G{"Lowest layer that moved"}
    G -->|"symbol / link down"| H["Physical: optic, cable, port"]
    G -->|"discards"| I["Lossless broken: PFC or buffers"]
    G -->|"seq err / RNR"| J["Transport retry: look below first"]
    G -->|"pause storm events"| S["Pause storm: stuck receiver"]
    G -->|"pause / CNP only"| K["Congestion: DCQCN, not hardware"]
    G -->|"nothing"| L["Fabric is not the fault"]

Procedure¶

HCA=mlx5_0                 # the RDMA device actually used by the workload, not the default
NIC=ens1f0np0              # the netdev that HCA is bound to, for RoCE only
PORT=1
LID=$(ibstat "$HCA" "$PORT" | awk '/Base lid/ {print $NF}')   # InfiniBand only

1. Snapshot before¶

Take all three namespaces at once, so the diff later is a single artifact:

rdma statistic show link > before-rdma.txt
# grep -H keeps the filename, which is the counter name. A bare cat writes an unlabelled
# column of integers that nothing downstream can parse.
grep -H . /sys/class/infiniband/$HCA/ports/$PORT/hw_counters/* > before-hw.txt
ethtool -S "$NIC" > before-eth.txt        # RoCE only; InfiniBand has no netdev counters

On InfiniBand also capture the switch-side port counters, which are the ones with saturating widths:

sudo perfquery "$LID" "$PORT"             # classic PortCounters: the error set
sudo perfquery -x "$LID" "$PORT"          # PortCountersExtended: 64-bit data counters

perfquery with no flag reads classic PortCounters, which is where SymbolErrorCounter, LinkErrorRecoveryCounter, LinkDownedCounter, PortRcvErrors and PortXmitDiscards live. -x reads PortCountersExtended, whose headline content is the 64-bit data and packet counters; 64-bit copies of the error counters are printed only where the device advertises the IB_PM_IS_ADDL_PORT_CTRS_EXT_SUP bit in its PerfMgt ClassPortInfo capability mask.¹ Do not assume -x alone gives you the error set, and always pass the port: with the port omitted the query goes to port 0.

2. Run the controlled test and record the byte count¶

Use the invocation that produced the baseline, and write down how many bytes it moved. The byte count is not bookkeeping: step 4 divides by it.

3. Snapshot after and diff¶

rdma statistic show link > after-rdma.txt
grep -H . /sys/class/infiniband/$HCA/ports/$PORT/hw_counters/* > after-hw.txt
ethtool -S "$NIC" > after-eth.txt
diff -u before-rdma.txt after-rdma.txt
diff -u before-hw.txt   after-hw.txt
diff -u before-eth.txt  after-eth.txt

4. Classify the deltas¶

The counters worth reading, by the layer they implicate:

Layer	Counters	What it means
Physical	`SymbolErrorCounter`, `LinkErrorRecoveryCounter`, `LinkDownedCounter`, `PortRcvErrors`	Optic, cable, connector or port. Everything above is downstream of this.
Pause storm	`tx_pause_storm_warning_events`, `tx_pause_storm_error_events`	A receiver stalled long enough to hold the link off. Presents through the pause counters but is a hardware fault, not congestion.
Lossless broken	`rx_discards_phy`, `rx_prio<n>_buf_discard`, `rx_prio<n>_cong_discard`, `PortXmitDiscards`	Packets dropped. On a fabric that promises losslessness this is a configuration or buffer fault.
Transport retry	`packet_seq_err`, `rnr_nak_retry_err`, `local_ack_timeout_err`, `implied_nak_seq_err`, `out_of_sequence`	Retransmission. Usually a symptom of one of the two layers above.
Congestion	`rx_prio<n>_pause`, `tx_prio<n>_pause`, `rx_prio<n>_pause_duration`, `np_cnp_sent`, `rp_cnp_handled`, `np_ecn_marked_roce_packets`	The lossless fabric working. Not a fault by itself, unless the stall counters above moved with it.

The mlx5 RDMA counter names are the kernel's q-counter and congestion-counter sets.² The pause, discard and ECN names are the kernel's ethtool stats, where %s expands to global or prio<n>.³

Two absences that are not zeros. The rx_prio<n>_pause family is emitted only for priorities on which PFC is actually enabled, because the driver fills those strings by iterating the enabled-priority bitmask. A grep prio3_pause that returns nothing means PFC is off on priority 3, not that no pause frames were seen. The per-priority discard and ECN-marking counters behave differently: they are emitted for all eight priorities and gated instead on a device capability. The RDMA congestion counters are gated on capability too, so a card that does not advertise congestion-statistics support legitimately exposes none of them.

5. Where the counters cannot answer¶

Three cases end the procedure without a verdict, and each is silent if you do not test for it:

Saturation. The classic InfiniBand error counters are narrow. SymbolErrorCounter is 16 bits; LinkErrorRecoveryCounter and LinkDownedCounter are 8 bits,⁶ and they saturate at their maximum rather than wrapping to zero.⁵ A port that has bounced 255 times reads 255 in both snapshots, so the delta is zero and the link grades clean at its worst. When a counter sits at its cap, reset and re-measure rather than reading the delta.
Reset mid-test. A driver reload, or another operator running perfquery -r, produces a negative delta. It is unrecoverable, not zero.
An incomplete snapshot. A counter absent from either side has no delta at all, and a missing key contributes nothing to a sum without complaining.

Because the classic counters saturate, the reliable InfiniBand method is reset, measure, read, rather than read, measure, read:

sudo perfquery -R "$LID" "$PORT"          # reset only, then run the controlled test
sudo perfquery "$LID" "$PORT"             # read the counts for that test alone

6. Physical layer, when the verdict is physical¶

sudo mst start && mst status              # resolve the /dev/mst/... device first
sudo mlxlink -d /dev/mst/mt4123_pciconf0 -m -c -e
sudo ibdiagnet                            # fabric-wide, network operator's tool

mlxlink reports module temperature and vendor, FEC mode, raw and effective bit error rate, and eye opening. A marginal optic shows a rising corrected-error rate long before it starts dropping links, which is what makes this the check that catches a fault before it becomes an outage. It takes an mst device path rather than an RDMA device name, which is why mst start comes first.

Note the packaging: ibstat, ibstatus and perfquery come from infiniband-diags, but none of the three commands above does. ibdiagnet ships in ibutils2, distributed with DOCA-Host, MLNX_OFED and UFM. mst and mlxlink come from NVIDIA MFT; the open-source mstflint builds the same link tool under the name mstlink, which takes a PCI address directly and needs no mst start. On a node with only infiniband-diags installed, none of them is present.

Reading the deltas¶

The rule below is what the diff from step 3 gets fed into. Its job is to be right about the cases where reading the diff by eye is wrong, and to refuse to answer where the counters cannot support an answer.

# counter_delta.py -- validated: turn two counter snapshots taken around one controlled
# test into a single verdict, and refuse to answer where the counters cannot support one.
# Names and widths are read from source, not from memory: SymbolErrorCounter 16-bit and
# LinkErrorRecoveryCounter / LinkDownedCounter 8-bit from rdma-core libibmad/fields.c;
# packet_seq_err, rnr_nak_retry_err, local_ack_timeout_err, rp_cnp_handled from the
# kernel's mlx5 q-counter and congestion-counter sets; rx_discards_phy, rx_prio3_pause
# and tx_pause_storm_error_events from the kernel's mlx5 ethtool stats.
# Standard library only.
PHYS, STALL, LOSS, RETRY, CONG = 0, 1, 2, 3, 4
LAYER = {PHYS: "physical", STALL: "pause-storm", LOSS: "lossless-broken",
         RETRY: "transport-retry", CONG: "congestion"}

# A fabric exposes ONE of these namespaces, not both: the InfiniBand PMA names come from
# perfquery, the RoCE names from ethtool and the netdev. Grading a snapshot against the
# union guarantees "incomplete" on every real fleet. CAP is None where the hardware width
# was not verified; a cap must never be guessed, because a wrong cap invents saturation.
PROFILES = {
    "ib": {
        "SymbolErrorCounter":       (PHYS,  2**16 - 1),
        "LinkErrorRecoveryCounter": (PHYS,  2**8 - 1),
        "LinkDownedCounter":        (PHYS,  2**8 - 1),
        "PortRcvErrors":            (PHYS,  2**16 - 1),
        "PortXmitDiscards":         (LOSS,  2**16 - 1),
        "packet_seq_err":           (RETRY, None),
        "rnr_nak_retry_err":        (RETRY, None),
        "local_ack_timeout_err":    (RETRY, None),
    },
    "roce": {
        "tx_pause_storm_error_events": (STALL, None),
        "rx_discards_phy":             (LOSS,  None),
        "rx_prio3_buf_discard":        (LOSS,  None),
        "packet_seq_err":              (RETRY, None),
        "rnr_nak_retry_err":           (RETRY, None),
        "local_ack_timeout_err":       (RETRY, None),
        "rx_prio3_pause":              (CONG,  None),
        "rp_cnp_handled":              (CONG,  None),
    },
}


def classify(before, after, bytes_moved, fabric, cong_per_gb=1_000.0):
    """One verdict from two snapshots taken around a controlled test.

    Error counters (physical, stall, loss, retry) are graded at zero tolerance. That is a
    deliberately conservative convention for a fleet gate, not a property of the hardware:
    NVIDIA's own ibdiagnet grades the physical layer against a bit-error-rate threshold
    rather than against zero. Congestion counters are graded as a rate per GB, because
    pause and CNP are the lossless fabric working and rise with offered load. The 1000/GB
    default is this page's convention and not a vendor threshold.
    """
    assert bytes_moved > 0, "a counter delta with no traffic behind it is not a measurement"
    ctrs = PROFILES[fabric]
    missing = sorted(set(ctrs) - (set(before) & set(after)))
    if missing:
        return {"verdict": "incomplete", "detail": missing, "live": []}

    gb = bytes_moved / 1e9
    unusable, live = [], []
    for name, (layer, cap) in ctrs.items():
        b, a = before[name], after[name]
        if a < b or (cap is not None and a >= cap):
            unusable.append(name)      # reset mid-test, or pegged at a saturating width
            continue
        if a == b:
            continue
        if layer != CONG or (a - b) / gb > cong_per_gb:
            live.append(name)
    live.sort()
    # An unreadable counter does not erase the readable part of the same measurement.
    if unusable:
        return {"verdict": "unusable", "detail": sorted(unusable), "live": live}
    if not live:
        return {"verdict": "clean", "detail": [], "live": []}
    worst = min(ctrs[n][0] for n in live)
    return {"verdict": LAYER[worst], "detail": [n for n in live if ctrs[n][0] == worst],
            "live": live}


IB = {k: 0 for k in PROFILES["ib"]}
RO = {k: 0 for k in PROFILES["roce"]}
TB = 1_000e9

# (1) Baseline: a terabyte moved and nothing counted.
assert classify(IB, dict(IB), TB, "ib")["verdict"] == "clean"

# (2) The discipline the rung exists for: a large LIFETIME value with a zero delta says
# nothing about this incident. Reading a counter once and reacting to its size is the error.
old = dict(IB, SymbolErrorCounter=3_000, packet_seq_err=250_000)
assert classify(old, dict(old), TB, "ib")["verdict"] == "clean"

# (3) Adversarial: LinkDownedCounter is 8 bits and saturates rather than wrapping. A port
# that has bounced 255 times reads 255 in both snapshots, so the delta is 0 and the naive
# rule grades a flapping link "clean" exactly when it is worst.
peg = dict(IB, LinkDownedCounter=255)
assert classify(peg, dict(peg), TB, "ib")["verdict"] == "unusable"

# (4) Adversarial: an unreadable counter must not erase the readable part of the same
# measurement. The verdict is withheld, but 500 discards still reach the operator.
r = classify(peg, dict(peg, PortXmitDiscards=500), TB, "ib")
assert r["verdict"] == "unusable" and r["live"] == ["PortXmitDiscards"], r

# (5) One count below the cap is a real, readable measurement.
assert classify(dict(IB, LinkDownedCounter=253), dict(IB, LinkDownedCounter=254), TB, "ib")["verdict"] == "physical"

# (6) A reset between snapshots (driver reload, perfquery -r/-R) gives a negative delta.
# Clamping it to zero reports clean; it is unrecoverable, not zero.
assert classify(dict(IB, packet_seq_err=900), dict(IB, packet_seq_err=12), TB, "ib")["verdict"] == "unusable"

# (7) Adversarial precedence: a marginal optic drives symbol errors AND the retransmissions
# that follow from them. Ranking by magnitude blames the symptom, 12,000 against 90, and
# sends you to chase retransmissions instead of replacing a cable.
phys = dict(IB, SymbolErrorCounter=90, packet_seq_err=12_000, local_ack_timeout_err=800)
res = classify(IB, phys, TB, "ib")
assert res["verdict"] == "physical" and res["detail"] == ["SymbolErrorCounter"], res
assert max(phys, key=phys.get) == "packet_seq_err"       # what magnitude ranking picks
assert PROFILES["ib"]["packet_seq_err"][0] == RETRY       # and it is a retry, not congestion

# (8) Adversarial precedence: pause frames outnumber discards by four orders of magnitude,
# but pause IS the lossless fabric working and a discard is it failing.
loss = dict(RO, rx_prio3_pause=8_000_000, rp_cnp_handled=90_000, rx_discards_phy=310)
assert classify(RO, loss, TB, "roce")["detail"] == ["rx_discards_phy"]

# (9) The same shape minus the discards is not a hardware fault at all: congestion
# signalling under load is the design working, so this is a DCQCN question.
assert classify(RO, dict(RO, rx_prio3_pause=8_000_000, rp_cnp_handled=90_000), TB, "roce")["verdict"] == "congestion"

# (10) Adversarial: a stalled receiver pausing the fabric is a HARDWARE fault that presents
# through the same pause counter. Without its own layer it reads as congestion, and the
# operator is sent to tune DCQCN for a card that needs replacing.
storm = dict(RO, tx_pause_storm_error_events=4, rx_prio3_pause=8_000_000)
assert classify(RO, storm, TB, "roce")["verdict"] == "pause-storm"

# (11) Adversarial normalisation: an identical congestion delta means different things at
# different traffic volumes, so congestion is graded per GB moved.
paused = dict(RO, rx_prio3_pause=900_000)
assert classify(RO, paused, TB, "roce")["verdict"] == "clean"
assert classify(RO, paused, 10e9, "roce")["verdict"] == "congestion"

# (12) Error counters get no such allowance, and that asymmetry is the convention: one
# sequence error across a terabyte is a finding where 900k pause frames are not.
assert classify(IB, dict(IB, packet_seq_err=1), TB, "ib")["verdict"] == "transport-retry"

# (13) Adversarial: grading a snapshot against the union of both namespaces returns
# "incomplete" for every real fabric, because a fabric exposes one or the other.
assert classify(IB, dict(IB), TB, "roce")["verdict"] == "incomplete"
assert classify(RO, dict(RO), TB, "ib")["verdict"] == "incomplete"

# (14) A counter absent from EITHER side has no delta, and a missing key contributes
# nothing to a sum without complaining.
gone = {k: v for k, v in RO.items() if k != "rx_discards_phy"}
assert classify(RO, gone, TB, "roce") == {"verdict": "incomplete",
                                          "detail": ["rx_discards_phy"], "live": []}

# (15) Boundary: the congestion rate bar is exclusive, checked either side of it.
assert classify(RO, dict(RO, rx_prio3_pause=1_000_000), TB, "roce")["verdict"] == "clean"
assert classify(RO, dict(RO, rx_prio3_pause=1_000_001), TB, "roce")["verdict"] == "congestion"

# (16) A delta with no traffic behind it is not a measurement.
try:
    classify(IB, dict(IB, SymbolErrorCounter=5), 0, "ib")
    raise SystemExit("guard did not fire")
except AssertionError:
    pass

CASES = [
    ("ib: nothing moved, 1 TB",      IB, dict(IB),                            TB,    "ib"),
    ("ib: large lifetime, no delta", old, dict(old),                          TB,    "ib"),
    ("ib: LinkDowned pegged at 255", peg, dict(peg, PortXmitDiscards=500),    TB,    "ib"),
    ("ib: reset mid-test",           dict(IB, packet_seq_err=900), dict(IB, packet_seq_err=12), TB, "ib"),
    ("ib: 90 symbol, 12000 seq",     IB, phys,                                TB,    "ib"),
    ("ib: one seq err over 1 TB",    IB, dict(IB, packet_seq_err=1),          TB,    "ib"),
    ("roce: pause 8M + 310 discard", RO, loss,                                TB,    "roce"),
    ("roce: pause 8M, no discard",   RO, dict(RO, rx_prio3_pause=8_000_000),  TB,    "roce"),
    ("roce: pause 8M + stall event", RO, storm,                               TB,    "roce"),
    ("roce: 900k pause over 1 TB",   RO, paused,                              TB,    "roce"),
    ("roce: 900k pause over 10 GB",  RO, paused,                              10e9,  "roce"),
    ("roce: counter missing",        RO, gone,                                TB,    "roce"),
]
print(f"{'case':32s} {'verdict':17s} blamed")
for label, b, a, n, fab in CASES:
    v = classify(b, a, n, fab)
    print(f"{label:32s} {v['verdict']:17s} {','.join(v['detail']) or '-'}")
print()
print("all assertions passed")

Executed output:

case                             verdict           blamed
ib: nothing moved, 1 TB          clean             -
ib: large lifetime, no delta     clean             -
ib: LinkDowned pegged at 255     unusable          LinkDownedCounter
ib: reset mid-test               unusable          packet_seq_err
ib: 90 symbol, 12000 seq         physical          SymbolErrorCounter
ib: one seq err over 1 TB        transport-retry   packet_seq_err
roce: pause 8M + 310 discard     lossless-broken   rx_discards_phy
roce: pause 8M, no discard       congestion        rx_prio3_pause
roce: pause 8M + stall event     pause-storm       tx_pause_storm_error_events
roce: 900k pause over 1 TB       clean             -
roce: 900k pause over 10 GB      congestion        rx_prio3_pause
roce: counter missing            incomplete        rx_discards_phy

all assertions passed

Five groups of rows are what make the rule worth having rather than eyeballing the diff.

The optic case is the precedence argument in numbers. packet_seq_err moved by 12,000 and SymbolErrorCounter by 90, so the largest delta in the output is 133 times the one that matters, and ranking by magnitude lands on the retry counter and sends you to chase retransmissions while a cable degrades. Retries are what a bad optic produces, so the physical layer is read first and the retry counters are read as confirmation.

The three pause rows are the same fabric under three readings. Pause frames are the lossless mechanism doing its job, so the row with only pause is a tuning question, the row with 310 discards is the mechanism failing, and the row with four stall events is a receiver stuck holding the fabric off. All three are dominated by the same 8,000,000 pause delta, and only the smaller counter beside it says which of the three you are in. Without a stall layer the third reads as congestion, and the operator tunes DCQCN for a card that needs replacing.

The two 900k pause rows are identical counter deltas with opposite meanings, separated only by the byte count of the test behind them. This is why step 2 records how many bytes moved: without it the same number is both a fabric pausing rarely per byte moved and one pausing eighty times more often for the same work, and nothing in the counter output distinguishes them. Note the model grades pause frequency per byte, not paused time: a pause frame carries a quanta value, so the frame count alone cannot give a duty cycle. rx_prio<n>_pause_duration is the counter that measures time, and reading it is the honest way to make a duty-cycle claim.

The pegged, reset and missing rows are the three ways this procedure returns no verdict. All three grade clean under a naive delta, and for the same structural reason: a zero delta, a negative delta clamped to zero, and an absent measurement are indistinguishable from a healthy link unless each is checked for separately. An 8-bit LinkDownedCounter at 255 is the worst link in the fleet reporting the same delta as the best. Note the pegged row still reports the 500 discards it could read: withholding a verdict is not a reason to discard the readable half of a measurement.

Verification¶

The counter that produced the verdict is flat across a repeat of the same controlled test, with the same byte count, after the repair.
No counter is pegged or reset in the verifying run, so the flat reading is a measurement rather than an absence of one.
The layer below the one you fixed is still flat. Fixing a retry symptom while a physical fault continues gives a clean retry counter for one run.
The bandwidth measurement recovered, in ib_write_bw or nccl-tests. Counters going quiet is not the same as throughput coming back.
The verdict was reproduced at least once before hardware is replaced on the strength of it.

Rollback¶

Counter reading changes nothing, so there is nothing to roll back from the diagnosis itself. What needs reverting is any action taken on it:

A counter reset cannot be undone. If you reset a port another team was investigating, say so rather than letting them read your test as their incident.
A PFC or DCQCN change made in response to a congestion verdict should be reverted singly and re-measured (RDMA and RoCE tuning). Congestion tuning that helps one traffic pattern regularly hurts another.
A cable or optic swap that did not move the counter should be recorded as not having moved it. An unrecorded negative result gets repeated.
Do not leave a port administratively down after a physical verdict without draining the node that depends on it (GPU health gating).

NCCL / fabric performance regression: the ladder that decides whether the fabric is the layer at fault. This runbook is its rung 4b in full.
NCCL socket fallback: throughput is down because the transport is wrong, not because the link is.
NCCL hang / collective stall: progress stopped rather than slowed.
PCIe / P2P bandwidth regression: the equivalent inside a node.
Fabric qualification gate: running these checks before a change reaches production rather than after.
operational runbooks: the runbook index.

References¶

rdma-core libibmad/fields.c (counter names and bit widths for PortCounters and PortCountersExtended): https://github.com/linux-rdma/rdma-core/blob/master/libibmad/fields.c
rdma-core infiniband-diags/perfquery.c (option table: -x extended, -r reset after read, -R reset only, -a, -l): https://github.com/linux-rdma/rdma-core/blob/master/infiniband-diags/perfquery.c
Linux kernel mlx5 InfiniBand counters (packet_seq_err, rnr_nak_retry_err, local_ack_timeout_err, implied_nak_seq_err, out_of_buffer, out_of_sequence, np_cnp_sent, rp_cnp_handled, np_ecn_marked_roce_packets): https://github.com/torvalds/linux/blob/master/drivers/infiniband/hw/mlx5/counters.c
Linux kernel mlx5 ethtool statistics (rx_discards_phy, rx_%s_pause, rx_%s_pause_duration, rx_prio%d_buf_discard, rx_prio%d_cong_discard, rx_prio%d_marked): https://github.com/torvalds/linux/blob/master/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
iproute2 rdma utility (rdma statistic show link): https://github.com/iproute2/iproute2
NVIDIA NCCL troubleshooting, networking (fabric diagnostics and the GPUDirect comparison method): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html
linux-rdma perftest (ib_write_bw, ib_write_lat): https://github.com/linux-rdma/perftest

rdma-core infiniband-diags/perfquery.c option table: {"extended", 'x', 0, NULL, "show extended port counters"}, {"reset_after_read", 'r', ...}, {"Reset_only", 'R', ...}. The 64-bit error-counter copies appear in libibmad/fields.c under the comment "More PortCountersExtended fields", gated by CounterSelect2, which is why they are device-dependent rather than guaranteed. ↩
Linux kernel drivers/infiniband/hw/mlx5/counters.c, INIT_Q_COUNTER(packet_seq_err), INIT_Q_COUNTER(rnr_nak_retry_err), INIT_Q_COUNTER(local_ack_timeout_err), INIT_Q_COUNTER(implied_nak_seq_err), and INIT_CONG_COUNTER(np_cnp_sent), INIT_CONG_COUNTER(rp_cnp_handled), INIT_CONG_COUNTER(np_ecn_marked_roce_packets). ↩
Linux kernel drivers/net/ethernet/mellanox/mlx5/core/en_stats.c, pport_per_prio_pfc_stats_desc with the source comment /* %s is "global" or "prio{i}" */ over rx_%s_pause, rx_%s_pause_duration, tx_%s_pause, tx_%s_pause_duration, rx_%s_pause_transition, and { "rx_discards_phy", PPORT_2863_OFF(if_in_discards) }. ↩
NVIDIA ibdiagnet bit-error-rate testing documents a non-zero threshold for physical-layer grading, with --ber_test carrying a default threshold and the fabric-health report checking symbol BER against it, rather than requiring zero errors. https://networking-docs.nvidia.com/ibdiagnetutilityum/2.26.0/bit-error-rate-ber ↩
The saturating behaviour, as distinct from the widths, is an InfiniBand architecture requirement: counters stick at their maximum value rather than wrapping. The Linux kernel records the same rule in the commit "IB/mlx4: Saturate RoCE port PMA counters in case of overflow": "In case of overflow, according to the IB spec, we have to saturate a counter to its max value, do that." https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e7d5b95ddb951db1a5b53c0c807bcac3ff64a7a2 ↩
rdma-core libibmad/fields.c, classic PortCounters: {BITSOFFS(32, 16), "SymbolErrorCounter", ...}, {BITSOFFS(48, 8), "LinkErrorRecoveryCounter", ...}, {BITSOFFS(56, 8), "LinkDownedCounter", ...}, {BITSOFFS(64, 16), "PortRcvErrors", ...}, {BITSOFFS(112, 16), "PortXmitDiscards", ...}. ↩