Markdown

LLM-as-a-judge reliability and validity¶

Scope: what it means to validate an LLM judge before trusting its scores, drawn from "Reliability without Validity" (arXiv 2606.19544), the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, run under three protocols over 118 runs and roughly 541,000 judgments. The paper's thesis is that the field's default validation metric, exact-match agreement, is reliable-looking but not valid: it does not correct for chance, so it systematically overstates a judge's discriminative ability, and a judge can be perfectly reproducible while being systematically biased. This page is the measurement deep dive: how each metric is defined, what the cohort actually scored, the minimum viable validation protocol the authors propose, and a validated stdlib model of the statistics. It is the judge-side companion to the LLM evaluation harness and eval gate, extends the bias cautions in reward-model training and reward design, sits under the same integrity discipline as evaluation integrity and anti-gaming, and is a sibling of NatureBench.

The paper is a March-April 2026 snapshot of hosted-model behavior; provider endpoints drift without versioning, so the per-judge numbers below should be read as a fixed baseline, not a permanent ranking. The metric model at the end is executed and self-asserting (pure stdlib), with every asserted number anchored to the paper's Table 2 or Table 3. Where the paper's own sections disagree with each other, the discrepancy is flagged rather than smoothed over.

flowchart TB
  STUDY["21 judges, 9 providers, 3 benchmarks<br/>541k judgments, 118 runs, temperature 0"] --> RAW["Exact-match agreement<br/>(no chance correction)"]
  RAW -->|"subtract chance agreement pe"| KAPPA["Cohen's kappa<br/>(po - pe) / (1 - pe)"]
  KAPPA -->|"gap is the kappa deflation"| DEFL["33.8 to 41.3 pp on MT-Bench"]
  RAW --> VALID{"Reliable, but is it valid?"}
  VALID --> CONS["Consistency<br/>test-retest above 0.95"]
  VALID --> POS["Position bias<br/>abs(P(first) - 0.5) above 0.10"]
  VALID --> VERB["Verbosity bias<br/>Pearson(len-diff, verdict) below 0.011"]
  CONS -.->|"stable AND biased"| PARA["Consistency-bias paradox<br/>Qwen 3 8B, Gemini 2.5 Flash"]
  POS -.-> PARA

What it is¶

LLM-as-a-Judge is the now-dominant pattern for reference-free scoring: prompt a capable model with a rubric and two candidate responses, and read its verdict as the quality signal. It is deployed across hundreds of documented production evaluation pipelines and underpins the open-ended half of most evaluation harnesses. The problem the paper attacks is meta-evaluation: how a team decides whether a given judge is good enough to trust. In practice that decision rests on exact-match agreement, the proportion of items on which the judge's verdict matches a human label, reported tie-excluded for pairwise comparisons.¹

The paper's organizing distinction is reliability versus validity. Reliability is whether a judge is consistent: does it give the same verdict when asked again, and is it stable across runs. Validity is whether the verdict is correct: does it track the human label, and is it free of order and length artifacts. A judge can be highly reliable and still invalid, and the paper shows this is not a corner case but a measured, repeatable failure across the cohort. To separate the two, every judge is measured under three protocols.² The agreement protocol produces one verdict per item and reports Cohen's kappa, Krippendorff's alpha, and tie-excluded exact match against the human label. The consistency protocol runs three to five independent evaluations per item with response caching disabled, presenting each pair in both AB and BA orderings, and reports test-retest reliability, self-consistency, and position flip rate. The bias-audit protocol presents AB and BA orderings together with a response-length analysis and reports position bias and verbosity bias.

The three benchmarks are chosen to span different latent constructs.³ MT-Bench contributes 2,391 pairwise comparisons with expert human judgments and a balanced A/B/Tie distribution; it measures preference alignment and is the most widely used judge benchmark. JudgeBench contributes 350 items across mathematics, coding, creative writing, and analysis, labeled for objective correctness rather than aesthetic preference. RewardBench contributes 2,981 chosen-versus-rejected pairs, presented under per-item position randomization so the chosen response is equally likely to sit in either slot. The cohort spans 8B to over-100B judges across three capability tiers, including the April 2026 frontier.⁴

Why it matters¶

Exact-match agreement rewards chance agreement as if it were genuine discrimination. Cohen's kappa corrects for this: kappa = (po - pe) / (1 - pe), where po is observed agreement and pe is the agreement expected by chance under the two raters' marginal label distributions.¹ When the marginals are skewed, which real judge-versus-human data almost always is, pe climbs and the same raw agreement corresponds to a much smaller kappa. The paper names the gap kappa deflation, dk(j,b) = EM(j,b) - kappa(j,b), and reports that it is universal: on MT-Bench every one of the 21 judges shows a deflation between 33.8 and 41.3 percentage points, with a cohort mean of 38.6 pp.⁵ The practical translation, in the paper's words, is that a judge reporting "85% agreement" on MT-Bench has a kappa near 0.48, which is moderate, not the near-perfect discrimination the percentage implies.

The deflation is not a model-quality artifact; it is how an uncorrected metric interacts with the benchmark's label marginals. It shrinks with the label distribution: the cohort-mean deflation contracts from 38.6 pp on MT-Bench (balanced ternary) to 23.7 pp on JudgeBench and to about 10.2 pp on RewardBench (imbalanced binary), exactly as Cohen's correction predicts.⁵ Because the size of the overstatement depends on the benchmark rather than the judge, an exact-match figure used to justify a deployment can overstate real discriminative ability by tens of points in a way the team cannot see without computing the chance-corrected number.

The second reason it matters is that a single benchmark does not generalize. Judge rankings are non-transferable: models shift by up to 14 positions across benchmarks, and only two judges (Claude Opus 4.6 and Gemini 3.1 Pro) hold a top-three position on all three.⁶ The instability is amplified by how differently the benchmarks separate judges: MT-Bench compresses all 21 into a 13.5 pp kappa band, while JudgeBench spreads the same judges across 60.4 pp, a factor of 4.5 wider. Where the underlying scale is compressed, tiny kappa differences produce large rank changes, so a leaderboard position on one benchmark is a poor predictor of position on another. Every downstream system that gates on an LLM judge, from an eval gate to a generative reward model to agent evaluation, inherits both problems: an inflated headline number and a ranking that may not survive a change of benchmark.

When to trust an LLM judge (and when not)¶

Trust the chance-corrected number, not the raw rate. Treat Cohen's kappa or Krippendorff's alpha as the headline reliability metric and read exact match only as a companion. A judge at kappa 0.5 on a balanced benchmark is a moderate rater regardless of how high its exact-match percentage looks.
Trust a ranking only across at least two benchmarks of contrasting label structure. One that spans the preference-versus-correctness axis, because a judge that looks strong under one construct can collapse under another (Llama 3.3 70B moved from near the top of MT-Bench to near the bottom of JudgeBench).
Do not treat high test-retest as evidence of a good judge. Stability is necessary but not sufficient; a deterministic judge can be perfectly reproducible and systematically wrong. Measure consistency and bias jointly, never consistency alone.
Do not assume the endpoint you validated is the endpoint you serve. All runs were a five-week snapshot; hosted models drift without versioning, so re-validate on a schedule rather than once.
Do not over-read the verbosity result. The cohort's verbosity bias is small (below 0.011) under one pairwise rubric and one length operationalization; the paper explicitly declines to claim verbosity bias is solved under arbitrary rubrics, and neither should a deploying team.
Do not extrapolate past the tested setting. The study is English-only and text-only, with reasoning traces suppressed and calibration deferred; multilingual, multimodal, and confidence-calibrated judging are out of scope and need their own measurement.

The metrics that matter¶

The paper implements 14 metrics; six carry the argument. Each is defined precisely so a team can reproduce it.¹⁵

Exact-match (raw) agreement. Proportion of items where the judge's verdict matches the human label, tie-excluded for pairwise items. Does not correct for chance, so it is sensitive to the benchmark's label distribution. This is the metric the paper argues should never stand alone.
Cohen's kappa. (po - pe) / (1 - pe), the chance-corrected agreement between judge and human. The paper's primary agreement metric. Krippendorff's alpha coincides with kappa in the two-rater nominal case and extends to the multi-run consistency setting.
Kappa deflation. dk = EM - kappa, the number of percentage points by which raw agreement overstates chance-corrected discrimination. Universal on MT-Bench at 33.8 to 41.3 pp; the diagnostic the paper introduces.
Test-retest reliability. A judge's agreement with itself across independent re-evaluations of the same items at temperature 0 with caching disabled. Cohort mean 0.943 on MT-Bench, falling to 0.911 on JudgeBench. Self-consistency (majority-verdict agreement across N runs) and position flip rate (fraction of items whose verdict changes when the order is swapped) are the item-level companions.
Position bias. abs(P(A wins) - 0.5) over paired AB and BA evaluations, where A is the first-presented slot. A fair judge scores 0; the cohort ranges from 0.002 (Gemini 2.5 Pro) to 0.192 (Qwen 3 8B), nearly two orders of magnitude. Frontier and reasoning judges tend lower but do not reach zero; only Gemini 3.1 Pro (0.038) met the paper's a-priori sub-0.05 prediction for thinking models.
Verbosity bias. Pearson correlation between the length differential abs(len(A)) - abs(len(B)) and the judge's verdict. All 21 judges scored below 0.011 (largest GPT-4o-mini at about 0.010), an order of magnitude smaller than 2023-era reports, under this single rubric.

The finding that ties the metrics together is the consistency-bias paradox: high test-retest reliability can coexist with severe position bias in the same judge, because test-retest measures the stability of outputs, not the correctness of the decision process. Position bias and within-judge agreement are mathematically orthogonal: a judge that deterministically prefers the first slot reproduces itself perfectly across runs while exhibiting maximal position bias.⁷ The paper set this as a pre-registered hypothesis (at least one judge with test-retest above 0.95 and position bias above 0.10) and found two production-deployed judges that instantiate it: Qwen 3 8B (test-retest 0.992, position bias 0.192, and the third-lowest JudgeBench kappa at 0.289) and Gemini 2.5 Flash (test-retest 0.988, position bias 0.125). The most reproducible judges were among the least valid, so reporting test-retest alone misleads precisely in the deployment cases that matter most.

The minimum viable validation protocol¶

The paper distills its findings into a Minimum Viable Validation Protocol (MVVP): five steps to run before deploying an LLM judge.⁸

Chance-correct. Report Cohen's kappa (or Krippendorff's alpha) alongside any exact-match figure, and treat the chance-corrected metric as the headline reliability number.
Swap positions. Measure position bias with paired AB and BA evaluations and report abs(P(A wins) - 0.5).
Replicate. Measure test-retest reliability over at least three independent runs at temperature 0 with response caching disabled.
Cross-validate. Evaluate on at least two benchmarks spanning preference-style and correctness-style label distributions.
Audit the paradox. When test-retest exceeds 0.95, verify that position bias is below 0.10 before claiming reliability. High stability with high bias is a failure mode, not a strength.

The protocol maps onto existing KB practice. Step 1 is the reporting discipline the eval gate already applies to benchmark scores. Step 2 is the same both-orders correction the harness page validates for judge win-rates and that reward design recommends against position bias. Steps 3 through 5 are the additions that turn a one-shot judge score into a validated one, and step 5 is the guard that no amount of reproducibility can substitute for.

How to run the validation¶

The metrics are small enough to compute exactly, and doing so makes the paper's thesis concrete. The model below is pure stdlib (fractions, statistics). It reproduces kappa deflation from skewed marginals, the position-bias metric, the consistency-bias paradox in one judge, verbosity bias as a contrast, and the paper's thesis in a single adversarial object: a judge that always picks the first slot has perfect test-retest, kappa 0, and maximal position bias, with the undefined-kappa edge case handled explicitly.

One honesty note on the mechanism. The paper's discussion says the MT-Bench chance baseline is pe near 1/3, but with a truly uniform marginal pe is exactly 1/3 and 85% agreement would give kappa near 0.78, not the observed 0.48. The real deflation comes from the judge and human marginals being skewed (concentrated on a clear winner with few ties), which lifts pe to roughly 0.7; solving the actual Table 2 rows (for example Gemini 3.1 Pro at EM 0.849, kappa 0.511) confirms pe near 0.69. The model below therefore reproduces the deflation the correct way, with skewed marginals, and hits the paper's own "85% agreement gives kappa near 0.48" anchor.

# llm_judge_reliability.py -- validated: the statistics behind "Reliability without
# Validity" (arXiv 2606.19544). Every number in an assert is anchored to the paper's
# Table 2 / Table 3 values. Pure stdlib (fractions, statistics).
#
# Reproduced:
#   1. Kappa deflation: HIGH raw (exact-match) agreement but much LOWER Cohen's kappa,
#      because skewed marginals inflate chance agreement (target 33-41 pp, MT-Bench).
#   2. Position bias = abs(P(first-position chosen) - 0.5) over AB+BA pairs.
#   3. The consistency-bias paradox: test-retest > 0.95 AND position bias > 0.10 in the
#      SAME judge (mirrors Qwen 3 8B: test-retest 0.992, position bias 0.192).
#   4. Verbosity bias small (< 0.011) as a contrast: Pearson(len-diff, verdict).
#   5. The thesis in one object: an "always pick position A" judge has perfect
#      test-retest, kappa = 0 (no better than chance), maximal position bias; plus the
#      pe = 1 edge case (kappa undefined) handled explicitly, not crashed.

from fractions import Fraction
from statistics import correlation


def raw_agreement(confusion):
    """Exact-match / raw agreement: diagonal mass over total. No chance correction."""
    n = sum(sum(row) for row in confusion)
    k = len(confusion)
    return Fraction(sum(confusion[i][i] for i in range(k)), n)


def cohen_kappa(confusion):
    """Cohen's kappa = (po - pe)/(1 - pe) from a square confusion (rows=human, cols=judge).
    Returns None when pe == 1 (expected agreement total -> kappa is 0/0, undefined)."""
    n = sum(sum(row) for row in confusion)
    assert n > 0
    k = len(confusion)
    po = Fraction(sum(confusion[i][i] for i in range(k)), n)
    row = [Fraction(sum(confusion[i]), n) for i in range(k)]
    col = [Fraction(sum(confusion[i][j] for i in range(k)), n) for j in range(k)]
    pe = sum(row[i] * col[i] for i in range(k))
    if pe == 1:
        return None
    return (po - pe) / (1 - pe)


# ---------------- 1. Kappa deflation (MT-Bench band 33.8-41.3 pp) ----------------
# Verdicts over {A, B, Tie}. Both human and judge concentrate on A with few ties, so
# the marginals are skewed and chance agreement pe is high; raw agreement stays at 85%
# while chance-corrected kappa collapses. Rows = human label, cols = judge verdict.
MT_BENCH = [
    # A    B   Tie   <- judge
    [770, 55, 15],   # human A   (row 840)
    [45,  65, 10],   # human B   (row 120)
    [18,   7, 15],   # human Tie (row 40)
]
raw = raw_agreement(MT_BENCH)          # 850/1000
kap = cohen_kappa(MT_BENCH)
defl = raw - kap
assert raw == Fraction(850, 1000)                              # 85% raw agreement
assert kap < raw                                              # kappa is deflated
assert Fraction(338, 1000) <= defl <= Fraction(413, 1000)     # 33.8-41.3 pp, MT-Bench band
# paper's own anchor: "a judge reporting 85% agreement on MT-Bench has kappa ~ 0.48"
assert abs(kap - Fraction(48, 100)) < Fraction(2, 100)


# ---------------- 2. Position bias = abs(P(first chosen) - 0.5) ----------------
def position_bias(picked_first):
    """picked_first: 1/0 per AB-or-BA judgment, 1 = judge chose the first-presented
    response. Position bias = abs(P(first chosen) - 0.5); 0 for a fair judge."""
    p_first = Fraction(sum(picked_first), len(picked_first))
    return abs(p_first - Fraction(1, 2)), p_first


# Fair judge: picks the better response regardless of slot. Over the AB+BA pair for one
# item, the better response is first in one order and second in the other, so the
# first-position indicator is exactly (1, 0) per item -> P(first) = 0.5, bias = 0.
fair = [1, 0] * 1000
fair_bias, _ = position_bias(fair)
assert fair_bias == 0

# Qwen 3 8B-like: first-position chosen 69.2% of judgments -> bias 0.192, the paper's
# most position-biased judge. 2000 judgments, 1384 chose the first slot.
biased = [1] * 1384 + [0] * 616
biased_bias, p_first = position_bias(biased)
assert p_first == Fraction(692, 1000)
assert biased_bias == Fraction(192, 1000)                     # Qwen 3 8B pb = 0.192
assert biased_bias > Fraction(10, 100)                        # breaches the 0.10 ceiling


# ---------------- 3. The consistency-bias paradox ----------------
def test_retest(runs):
    """runs: per-run verdict lists aligned by item. Test-retest = fraction of items whose
    verdict is identical across all runs (a judge's agreement with itself)."""
    n_items = len(runs[0])
    assert all(len(r) == n_items for r in runs)
    stable = sum(1 for i in range(n_items) if len({r[i] for r in runs}) == 1)
    return Fraction(stable, n_items)

# Near-deterministic judge (temperature 0): identical verdict on 992 of 1000 items across
# 3 replicate runs, flips on 8 -> test-retest 0.992 (Qwen 3 8B's reported value).
run_a = [0] * 1000
run_b = [0] * 1000
run_c = [0] * 1000
for i in range(992, 1000):             # 8 unstable items
    run_b[i] = 1
tr = test_retest([run_a, run_b, run_c])
assert tr == Fraction(992, 1000)                              # test-retest 0.992
assert tr > Fraction(95, 100)                                # > 0.95
# THE PARADOX: the same judge is severely position-biased. High reproducibility and low
# validity hold simultaneously -> consistency does not imply validity.
assert tr > Fraction(95, 100) and biased_bias > Fraction(10, 100)


# ---------------- 4. Verbosity bias small (< 0.011), with a detecting contrast --------
# Fair judge: verdict decided by quality, uncorrelated with the length differential. For
# every length differential d there is one win and one loss, so verdict carries no length
# signal -> Pearson ~ 0.
len_diff, verdict = [], []
for d in range(-200, 201, 10):
    len_diff += [d, d]
    verdict += [1, 0]
r_fair = correlation(len_diff, verdict)
assert abs(r_fair) < 0.011                                   # cohort: all judges < 0.011

# Contrast: a length-biased judge that always prefers the longer response. The SAME metric
# flags it (the 2023-era failure the 2026 cohort no longer shows).
len_diff2 = list(range(-200, 201, 10))
verdict2 = [1 if d > 0 else 0 for d in len_diff2]
r_biased = correlation(len_diff2, verdict2)
assert abs(r_fair) < 0.011 < r_biased


# ---------------- 5. The thesis in one object + pe=1 edge case ----------------
# A judge that ALWAYS picks the first-presented response.
always_first = [1] * 2000
af_bias, af_p = position_bias(always_first)
assert af_p == 1 and af_bias == Fraction(1, 2)               # maximal position bias 0.5
# Deterministic -> identical across any number of runs -> perfect test-retest.
det = [1] * 1000
assert test_retest([det, det, det]) == 1
# Yet kappa = 0: the verdict tracks PRESENTATION ORDER, not response identity, so against a
# fixed human label it is right only by position luck (~50%) with 50/50 marginals -> pe=0.5,
# kappa=0. Confusion of judge response-identity vs human preference (order randomized):
ALWAYS_FIRST_VS_HUMAN = [
    # judge=X  judge=Y
    [250, 250],   # human prefers X
    [250, 250],   # human prefers Y
]
k_af = cohen_kappa(ALWAYS_FIRST_VS_HUMAN)
assert k_af == 0                                             # no better than chance
assert raw_agreement(ALWAYS_FIRST_VS_HUMAN) == Fraction(1, 2)  # 50% raw, but kappa 0
# One object: perfect consistency, kappa 0, maximal position bias -> the paper's thesis.

# pe = 1 edge case: judge emits one label on every item and human labels are all that same
# label -> observed and expected agreement both total -> kappa is 0/0, undefined.
DEGENERATE = [
    [1000, 0],
    [0,    0],
]
assert cohen_kappa(DEGENERATE) is None                       # handled, not a ZeroDivision

print(
    f"1. kappa deflation: raw={float(raw):.3f}, kappa={float(kap):.3f}, "
    f"deflation={float(defl) * 100:.1f}pp (MT-Bench band 33.8-41.3pp)\n"
    f"2. position bias: fair={float(fair_bias):.3f}, biased={float(biased_bias):.3f} "
    f"(P(first)={float(p_first):.3f}, > 0.10 ceiling)\n"
    f"3. paradox: test-retest={float(tr):.3f} (>0.95) WITH position bias="
    f"{float(biased_bias):.3f} (>0.10) in one judge\n"
    f"4. verbosity: fair r={r_fair:.4f} (<0.011) vs length-biased r={r_biased:.3f}\n"
    f"5. always-first judge: test-retest=1.0, kappa={float(k_af):.1f}, raw="
    f"{float(raw_agreement(ALWAYS_FIRST_VS_HUMAN)):.2f}, position bias={float(af_bias):.1f}; "
    f"degenerate pe=1 -> kappa={cohen_kappa(DEGENERATE)}"
)

Run output:

1. kappa deflation: raw=0.850, kappa=0.471, deflation=37.9pp (MT-Bench band 33.8-41.3pp)
2. position bias: fair=0.000, biased=0.192 (P(first)=0.692, > 0.10 ceiling)
3. paradox: test-retest=0.992 (>0.95) WITH position bias=0.192 (>0.10) in one judge
4. verbosity: fair r=0.0000 (<0.011) vs length-biased r=0.866
5. always-first judge: test-retest=1.0, kappa=0.0, raw=0.50, position bias=0.5; degenerate pe=1 -> kappa=None

The always-first object is the whole paper compressed: it is perfectly reliable (test-retest 1.0), yet its kappa is exactly 0 and its position bias is maximal. Reliability without validity. The kappa-deflation block shows the quieter version of the same lesson: a judge can post 85% raw agreement and still be a kappa-0.47 rater once chance is removed.

Failure modes¶

Reporting raw agreement as the reliability number. Exact match overstates chance-corrected discrimination by tens of points, and the size of the overstatement depends on the benchmark's label marginals, not the judge. Report kappa or alpha as the headline (MVVP step 1).
Validating on one benchmark. Rankings shift by up to 14 positions across benchmarks and only two judges hold a top-three slot on all three; a single leaderboard is a poor predictor of another. Cross-validate on at least two contrasting label structures (MVVP step 4).
Claiming reliability from test-retest alone. High test-retest can mask a determinism that maximizes position bias; Qwen 3 8B and Gemini 2.5 Flash are the measured examples. Always pair consistency with a position-bias audit (MVVP steps 3 and 5).
Trusting a validation snapshot indefinitely. Hosted endpoints drift without versioning, and the study is a five-week window; a judge validated once can silently change. Re-validate on a schedule.
Over-generalizing the verbosity result. The below-0.011 cohort figure holds under one pairwise rubric and one length operationalization; rubric design interacts with bias profiles, so it is not a universal claim that verbosity bias is solved.
Extrapolating past the tested regime. English-only, text-only, reasoning traces suppressed, no calibration. Multilingual, multimodal, and confidence-calibrated judging need their own measurement before any of these numbers transfer.
Reading a single number where the paper itself wobbles. The kappa-deflation ceiling appears as 41.2 pp in Section 2.1, its Figure 1 caption, and an appendix hypothesis table, but 41.3 pp in the results and conclusion (the abstract itself only rounds to a bare "41 pp"); the largest rank shift is stated as "up to 14 positions" in the abstract while Section 4.3 describes Llama 3.3 70B moving from rank 5 to rank 20 (a 15-position shift) and Section 5.2 gives the same model as rank 5 to rank 19 (14 positions); the RewardBench mean deflation is 10.2 pp in Table 2 and 10.4 pp in one prose line (a second appendix table gives 10.4 pp again, alongside a JudgeBench mean of 24.0 pp against Table 2's 23.7 pp). The paper also states "nine providers" four times, but its own Table 1 and an appendix table list ten distinct provider names when counted directly. Cite the range and note the wobble rather than a single decimal.

References¶

Norman, Rivera, Hughes, "Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias" (arXiv 2606.19544): https://arxiv.org/abs/2606.19544
Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (MT-Bench; the exact-match / tie-excluded convention this paper follows): https://arxiv.org/abs/2306.05685
Tan et al., "JudgeBench: A Benchmark for Evaluating LLM-based Judges" (correctness-labeled items): https://arxiv.org/abs/2410.12784
Lambert et al., "RewardBench: Evaluating Reward Models for Language Modeling" (chosen-vs-rejected pairs): https://arxiv.org/abs/2403.13787

arXiv 2606.19544, Section 2.1 (Metrics and Definitions). Exact match: proportion of items on which the judge verdict matches the human label, tie-excluded for pairwise (denominator restricted to non-tie human labels, following Zheng et al. 2023). Cohen's kappa: (po - pe)/(1 - pe) with po observed and pe chance agreement under marginal label distributions (Cohen, 1960); it is the paper's primary agreement metric. Krippendorff's alpha accommodates more than two raters and coincides with kappa in the two-rater nominal case, extending to multi-run consistency. Test-retest reliability: agreement of a judge with itself across independent re-evaluations. Self-consistency: proportion of items where the majority verdict across N runs agrees with each run. Position flip rate: fraction of items whose verdict changes when the response order is swapped. Position bias: abs(P(A wins) - 0.5) over paired AB+BA evaluations. Verbosity bias: Pearson correlation between the length differential abs(len(A)) - abs(len(B)) and the judge's verdict (following Dubois et al.). ↩↩↩
arXiv 2606.19544, Section 3 (Methodology, Evaluation Protocols). Agreement protocol: one judgment per item vs the human label, reporting Cohen's kappa, Krippendorff's alpha, and tie-excluded exact match. Consistency protocol: N in [3,5] independent evaluations per item with response caching disabled, each pair presented in both AB and BA orderings, reporting test-retest, self-consistency, and position flip rate. Bias-audit protocol: AB+BA orderings with response-length analysis, reporting position bias and verbosity bias. All evaluations at temperature 0; position-swap debiasing applied in the consistency and bias-audit protocols; 118 runs and ~541,000 judgments over a five-week March-April 2026 window. ↩
arXiv 2606.19544, Section 3 (Benchmarks). MT-Bench: 2,391 pairwise comparisons with expert human judgments, balanced A/B/Tie, most widely used judge benchmark (Zheng et al. 2023); measures preference alignment. JudgeBench: 350 items across mathematics, coding, creative writing, and analysis, labeled for objective correctness (Tan et al. 2025); measures correctness discrimination. RewardBench: 2,981 chosen-vs-rejected pairs under per-item position randomization (Lambert et al. 2025); measures chosen-vs-rejected discrimination. ↩
arXiv 2606.19544, Section 3 and Table 1. 21 general-purpose judges from what the paper's own text states four times as nine providers, though Table 1 and an appendix table (Table 8) both list ten distinct provider names when counted directly (OpenAI, Google, Anthropic, Meta, Alibaba, Mistral, MiniMax, DeepSeek, Moonshot, Zhipu): a further instance of the paper's internal count wobbling, not a KB transcription error. Grouped into three tiers: Tier 1 production judges, Tier 2 cost-conscious, Tier 3 April-2026 frontier and open-source. Parameter counts span 8B to over 100B. For models with a built-in reasoning trace (GPT-5.4, GPT-5.4-mini, Gemini 3.1 Pro, DeepSeek V3.2, Kimi K2.5, GLM-5, Minimax M2.7) the reasoning channel was suppressed. ↩
arXiv 2606.19544, Section 2.1 and Section 4.1, Table 2. Kappa deflation dk = EM - kappa. On MT-Bench, dk in [33.8, 41.3] pp across all 21 judges, cohort mean 38.6 pp (the abstract and Figure 1 caption state the upper bound as 41.2 pp; Section 4.1 and the Conclusion state 41.3 pp). JudgeBench mean 23.7 pp (range 8.1 to 38.5); RewardBench cohort mean 10.2 pp in Table 2 (one prose line in Section 4.1 says 10.4 pp), range 5.9 to 21.3. Anchor: "a judge reporting 85% agreement on MT-Bench has kappa near 0.48"; Section 5.1 states exact-match in [0.80, 0.85] corresponds to kappa near 0.48. Deflation tracks the label distribution: balanced ternary (MT-Bench) yields the largest gap, imbalanced binary (RewardBench) the smallest. Example Table 2 row: Gemini 3.1 Pro EM 0.849, kappa 0.511, dk 33.8 pp. ↩↩↩
arXiv 2606.19544, Sections 4.3 and 5.2. Judge rankings shift by up to 14 positions across benchmarks; more than half (11 of 21) shift by four or more positions; only Claude Opus 4.6 and Gemini 3.1 Pro hold a top-three position on all three benchmarks. Benchmark discriminability differs sharply: MT-Bench kappa spread 13.5 pp (0.376 to 0.511), JudgeBench 60.4 pp (0.271 to 0.875), a factor of 4.5 wider, RewardBench 28.1 pp. Internal discrepancy on the largest single shift: Section 4.3 gives Llama 3.3 70B moving from rank 5 (MT-Bench) to rank 20 (JudgeBench), a 15-position shift, while Section 5.2 gives the same model as rank 5 to rank 19, consistent with the headline "up to 14". ↩
arXiv 2606.19544, Sections 2.1, 4.7, and 5.3. Consistency-bias paradox: high test-retest (alpha > 0.95) coexists with severe position bias in the same judge, because test-retest measures output stability, not decision-process correctness; position bias and within-judge agreement are orthogonal, so a judge deterministically favoring position A achieves near-perfect test-retest while exhibiting maximal position bias. Hypothesis H7 (at least one judge with test-retest > 0.95 and position bias > 0.10) confirmed by two production judges: Qwen 3 8B (test-retest 0.992, position bias 0.192, JudgeBench kappa 0.289, the third-lowest) and Gemini 2.5 Flash (test-retest 0.988, position bias 0.125). Cohort-mean test-retest 0.943 on MT-Bench, 0.911 on JudgeBench. Position bias range across the cohort: 0.002 (Gemini 2.5 Pro) to 0.192 (Qwen 3 8B). Verbosity bias: all 21 judges below 0.011 (largest GPT-4o-mini about 0.010), 17 of 21 below 0.005, under a single pairwise rubric. ↩
arXiv 2606.19544, Section 5.3 (Minimum Viable Validation Protocol) and Section 6. Five steps: (1) chance-correct, report Cohen's kappa or Krippendorff's alpha alongside exact match and treat the corrected metric as headline; (2) swap positions, measure position bias via paired AB+BA and report abs(P(A wins) - 0.5); (3) replicate, measure test-retest over at least three independent runs at temperature 0 with caching disabled; (4) cross-validate, evaluate on at least two benchmarks spanning preference-style and correctness-style labels; (5) audit the paradox, when test-retest exceeds 0.95 verify position bias is below 0.10 before claiming reliability. Limitations (paper's own): English-only and text-only benchmarks, a five-week temporal snapshot with no drift measurement, a single pairwise rubric, calibration deferred (no logprob-based ECE or Brier), and list-price cost figures. ↩