Markdown

Rubric-based rewards for RL¶

Scope: deriving a usable RL reward for open-ended tasks that no deterministic verifier can score, using instance-specific rubrics evaluated by an LLM judge. This page covers how a rubric is decomposed and aggregated, the aggregation shape that stops the policy learning to decorate instead of to answer, when averaging over criteria actually reduces judge noise and when it does not, the effect graded rewards have on degenerate GRPO groups, and the operational discipline a judge in the training loop needs. It extends RLVR past math and code, and depends on the reliability work in LLM judge reliability.

The numpy block is executed and asserted here. It models judge behaviour statistically to isolate the aggregation properties; it is not a measurement of any judge model.

What it is¶

RLVR works because a verifier returns a correct answer cheaply and cannot be talked into a wrong one. Most valuable work is not like that. Clinical advice, research writing, code review, and long-form analysis are judged on several criteria at once, and no exact-match function scores them.

A rubric reward decomposes that judgment into an explicit, prompt-specific checklist. For each prompt, a rubric of N criteria is authored or generated in advance, ideally grounded in a reference answer or expert guidance. At training time an LLM judge evaluates each criterion separately against the completion, and the per-criterion verdicts are aggregated into one scalar reward. Rubrics as Rewards (RaR) reports that the best rubric-aggregation variant achieved relative improvements of up to 31% on HealthBench and 7% on GPQA-Diamond over LLM-as-judge baselines using direct Likert-scale rewards.

Three properties distinguish this from asking a judge for a score out of ten:

The criteria are explicit, so the reward is auditable. When the policy games it, the specific criterion it gamed is visible.
The judgment is decomposed, so each call is a narrow binary question rather than a holistic aesthetic one, which smaller judges answer more reliably.
The aggregation is code, not model output. Weights, gates, and normalization are engineering decisions with the same status as anything else in reward design.

Why use it¶

It unlocks the domains RL has been stuck outside. Everything RLVR cannot score is a candidate, which is most knowledge work.
It produces a graded reward instead of a binary one. That is not a side benefit. A binary verifier at a 90% pass rate wastes 43% of its groups on zero-advantage batches, measured below; a five-criterion rubric wastes under 2%.
It reduces judge variance when the criteria are genuinely independent. Averaging N independent verdicts at error rate e gives a score variance of e(1-e)/N, validated below.
It is inspectable at the point of failure. A holistic judge score that drifts tells you nothing. A per-criterion breakdown tells you which criterion the policy learned to satisfy without doing the work.
It works better with smaller judges. RaR reports that structured rubric signals give better alignment for smaller judges and reduce performance variance across judge scales, which matters because the judge runs on every rollout.

When to use it (and when not)¶

Use it when the task is open-ended, quality is multi-dimensional, and a per-prompt reference answer or expert guidance exists to ground the rubric.
Prefer RLVR whenever a deterministic verifier exists. A rubric adds a model to the reward path, and that model can be gamed in ways a symbolic checker cannot.
Compose them. A verifier gate on correctness plus a rubric for everything above the gate is stronger than either, and is the structure validated in check (1).
Do not use it without holding out a rubric-free eval. If both training and evaluation go through the same rubric, reward hacking and genuine improvement are indistinguishable (evaluation integrity).
Do not use generic rubrics. The reported gains come from rubrics that are prompt-specific, comprehensive, and grounded in a reference. A single global checklist reduces to a verbose Likert score.
Do not use it if the judge cannot be afforded on every rollout. The judge runs N times per completion times G completions per prompt. That cost is a serving deployment, not a rounding error.

Architecture¶

flowchart TB
  P["Prompt + reference answer"] --> RB["Rubric: N prompt-specific criteria,<br/>weights, hard/soft flags"]
  RB --> CACHE["Rubric cache, versioned per prompt"]
  ROLL["G rollouts from the policy"] --> J["LLM judge: one call per criterion"]
  CACHE --> J
  J --> AGG["Aggregate in code:<br/>gate on hard criteria, then weighted sum"]
  AGG --> R["Scalar reward"]
  R --> ADV["Group-relative advantage"]
  ADV --> UPD["Policy update"]
  VER["Deterministic verifier, where one exists"] -.->|"hard gate"| AGG
  HOLD["Rubric-free held-out eval"] -.->|"detects hacking"| R

How to use it¶

The aggregation code is where a rubric reward is won or lost. The block below asserts the four properties that decide it.

# rubric_reward.py -- the four properties that decide whether a rubric reward trains a good
# model or a well-formatted bad one: aggregation shape, judge-error averaging, group
# degeneracy, and cross-prompt weight normalization. numpy only.
import numpy as np

# A rubric for one prompt: criteria, weights, and which are HARD requirements.
CRITERIA = ["answer is correct", "cites a source", "states uncertainty",
            "uses the requested format", "is concise"]
WEIGHTS = np.array([0.40, 0.15, 0.15, 0.15, 0.15])
HARD = np.array([True, False, False, False, False])       # correctness gates the rest

def weighted_sum(sat):
    return float(WEIGHTS @ sat)

def gated(sat):
    """Hard criteria gate the score; soft criteria only differentiate above the gate."""
    if not sat[HARD].all():
        return 0.0
    return float(WEIGHTS @ sat)

# (1) ADVERSARIAL: a completion that is WRONG but ticks every cosmetic box outranks a
#     correct, plain one under a plain weighted sum, 0.60 against 0.40. Gating on the hard
#     criterion is what stops the policy learning to decorate instead of to answer. Any
#     rubric whose soft weights can outvote its hard one has this hole.
polished_wrong = np.array([0, 1, 1, 1, 1], dtype=bool)
correct_plain  = np.array([1, 0, 0, 0, 0], dtype=bool)
assert weighted_sum(polished_wrong) > weighted_sum(correct_plain)
assert gated(polished_wrong) == 0.0 and gated(correct_plain) > 0.0
assert WEIGHTS[~HARD].sum() > WEIGHTS[HARD].sum()          # the exact condition for the hole

# (2) JUDGE ERROR AVERAGES DOWN ONLY IF ERRORS ARE INDEPENDENT. With N criteria judged
#     independently at error rate e, the score variance is e(1-e)/N. One shared error
#     decision across all criteria leaves it at e(1-e), no matter how many criteria.
rng = np.random.default_rng(11)
M, e = 200_000, 0.15
def judged_score(N, correlated):
    truth = np.ones((M, N), dtype=bool)                   # a perfect completion
    flip = (rng.random((M, 1)) < e) if correlated else (rng.random((M, N)) < e)
    return (truth ^ flip).mean(axis=1)
var_iid = {N: float(judged_score(N, False).var()) for N in (1, 4, 16)}
var_cor = {N: float(judged_score(N, True).var()) for N in (1, 4, 16)}
for N in (1, 4, 16):
    assert abs(var_iid[N] - e * (1 - e) / N) < 0.02 * e * (1 - e) / N   # matches e(1-e)/N
    assert abs(var_cor[N] - e * (1 - e)) < 0.02 * e * (1 - e)           # no reduction at all
assert var_iid[16] < var_cor[16] / 10

# (3) A GRADED RUBRIC RESCUES GROUPS THAT A BINARY VERIFIER WASTES. A group whose members
#     all score the same has zero group-relative advantage and teaches nothing. For a binary
#     reward the chance of that is p^G + (1-p)^G, which is severe once the policy is good.
G = 8
def degenerate_binary(p):
    return p ** G + (1 - p) ** G
def degenerate_rubric(p, N=5, trials=200_000):
    r = (rng.random((trials, G, N)) < p).mean(axis=2)     # N independent criteria
    return float((r.std(axis=1) == 0).mean())
tbl = [(p, round(degenerate_binary(p), 4), round(degenerate_rubric(p), 4))
       for p in (0.5, 0.8, 0.9)]
for p, b, r in tbl:
    assert r < b                                          # graded is never worse
assert tbl[-1][1] > 0.4 and tbl[-1][2] < 0.02             # at p=0.9: 43% wasted vs under 2%

# (4) WEIGHTS MUST NORMALIZE PER PROMPT. Two prompts with different numbers of criteria and
#     unnormalized weights give the longer rubric a larger reward range, so it dominates the
#     batch gradient purely by having more boxes to tick.
short = np.ones(3); long_ = np.ones(9)                    # unit weight per criterion
assert long_.sum() / short.sum() == 3.0                   # 3x the reward range
assert np.isclose((short / short.sum()).sum(), (long_ / long_.sum()).sum())

# (5) BOUNDARY: an empty rubric, and a rubric whose weights are all zero, must not divide
#     by zero or silently score 1.0.
for w in (np.array([]), np.zeros(4)):
    total = w.sum()
    assert total == 0.0                                   # callers must reject, not divide

print("weighted sum: polished-but-wrong", weighted_sum(polished_wrong),
      "| correct-but-plain", weighted_sum(correct_plain))
print("gated       : polished-but-wrong", gated(polished_wrong),
      "| correct-but-plain", round(gated(correct_plain), 2))
print("score variance, independent judge errors:", {k: round(v, 5) for k, v in var_iid.items()})
print("score variance, correlated judge errors :", {k: round(v, 5) for k, v in var_cor.items()})
print("degenerate-group rate (pass rate, binary, 5-criterion rubric):")
for row in tbl:
    print("   ", row)

Executed output:

weighted sum: polished-but-wrong 0.6 | correct-but-plain 0.4
gated       : polished-but-wrong 0.0 | correct-but-plain 0.4
score variance, independent judge errors: {1: 0.12644, 4: 0.03195, 16: 0.00797}
score variance, correlated judge errors : {1: 0.12767, 4: 0.12795, 16: 0.12772}
degenerate-group rate (pass rate, binary, 5-criterion rubric):
    (0.5, 0.0078, 0.0002)
    (0.8, 0.1678, 0.001)
    (0.9, 0.4305, 0.015)

Check (1) is the defect to look for first. With correctness weighted at 0.40 and four cosmetic criteria at 0.15 each, a wrong-but-well-decorated completion scores 0.60 against a correct-but-plain one at 0.40, so the policy is being taught, precisely and measurably, to decorate rather than to answer. The condition is mechanical and worth asserting in the reward function itself: if the soft weights can outvote the hard ones, the hole exists. Gating removes it.

Check (2) sets the limit on how much noise decomposition removes. With independent per-criterion errors, score variance falls as e(1-e)/N, from 0.126 at one criterion to 0.008 at sixteen. With a single shared error decision it stays at roughly 0.128 regardless of N. Real judge errors sit between these, and the correlated component is what a shared judge model, a shared prompt template, and a shared misreading of the completion produce. Adding criteria that all fail together buys nothing.

Check (3) is the argument for rubrics that has nothing to do with non-verifiable domains. At a 90% pass rate a binary reward makes 43.05% of G = 8 groups degenerate, all-correct or all-wrong, contributing exactly zero gradient. The same policy scored on five graded criteria degenerates on 1.5% of groups. A graded reward is a cheaper fix for the zero-variance-group problem than DAPO's dynamic sampling, which solves it by generating more rollouts.

How to develop with it¶

# Reference template (needs TRL and a judge client). A rubric reward function in the shape
# TRL's GRPOTrainer / RLOOTrainer expect: reward_funcs return one float per completion.
# The gate-then-sum aggregation is the arithmetic validated above.
from trl import GRPOConfig, GRPOTrainer

def rubric_reward(prompts, completions, rubrics, **kwargs) -> list[float]:
    out = []
    for prompt, completion, rubric in zip(prompts, completions, rubrics):
        verdicts = judge_each_criterion(prompt, completion, rubric)   # one call per criterion
        weights = normalize(rubric.weights)                           # per prompt, sums to 1
        if not all(v for v, c in zip(verdicts, rubric.criteria) if c.hard):
            out.append(0.0)                                           # hard gate
            continue
        out.append(float(sum(w * v for w, v in zip(weights, verdicts))))
    return out

cfg = GRPOConfig(num_generations=8, beta=0.0)
trainer = GRPOTrainer(model=..., reward_funcs=rubric_reward, args=cfg, train_dataset=...)

Authoring and operational rules that follow:

Normalize weights per prompt. Check (4) is the cross-prompt version of the loss-aggregation length bias in GRPO variants: a nine-criterion rubric with unit weights has three times the reward range of a three-criterion one and dominates the batch gradient for no reason connected to quality.
Judge one criterion per call. Asking for all N verdicts in one response reintroduces exactly the correlated error that check (2) shows destroys the averaging benefit.
Version and cache the rubric with the prompt. A rubric regenerated mid-run silently changes the objective. Treat it as training data, under the same governance as anything else in governance registries.
Pin the judge model and its decoding parameters. A judge upgrade is a reward change. Re-baseline before, not after.
Keep a deterministic gate wherever one exists. A unit test, a schema check, or a symbolic answer match is strictly better than asking a model the same question.

How to maintain it¶

Track per-criterion satisfaction rates over training. The signature of reward hacking is one criterion saturating while the others stay flat and held-out quality does not move. A holistic reward number cannot show this.
Hold out a rubric-free eval. Human preference, a different benchmark, or a stronger judge with a different prompt. Training reward rising while this stalls is the standard reward-hacking divergence (reward design).
Audit judge agreement periodically. Sample completions, have a stronger judge or a human re-score them, and measure per-criterion agreement rather than aggregate correlation. Aggregate agreement hides a single criterion the judge scores at chance.
Watch the degenerate-group rate anyway. A rubric reduces it but does not eliminate it, and a rubric that has effectively collapsed to binary will show up here first.
Watch judge cost per step. At N criteria times G rollouts, the judge can quietly become the largest component of the step, at which point rubric size is a capacity decision.

How to run it in production¶

Size the judge as a serving deployment. It handles N * G requests per prompt per step, running concurrently with rollout generation and competing for the same GPUs unless it is placed deliberately (async and disaggregated RL, rollout fleet sizing).
Use a smaller judge and shorten the criteria before cutting the rubric. RaR reports rubrics improve alignment for smaller judges specifically; that is the intended lever.
Keep the KL penalty available. Unlike a deterministic verifier, a judge is a learned proxy and is hackable, which is the classic case for keeping beta above zero (KL regularization).
Batch and cache judge calls per criterion. The rubric text is constant per prompt across all G completions, so it is a prefix worth caching (prompt caching).
Fail closed. A judge timeout or a malformed verdict must not score as satisfied. Score it as unsatisfied or drop the completion; never let an infrastructure failure read as quality.

Failure modes¶

Soft criteria outvoting the hard one. Teaches decoration over correctness, as in check (1). Gate, do not sum.
All criteria judged in one call. Correlated errors, no variance reduction, and one bad parse takes out the whole rubric.
Generic rubrics. Collapse to a verbose Likert score and lose the reported advantage over one.
Unnormalized weights across prompts. Long rubrics dominate the batch gradient.
Judge failures scoring as passes. A timeout that returns a default of satisfied is a reward-hacking channel that requires no cleverness from the policy at all.
Evaluating with the training rubric. Makes hacking invisible by construction.
Rubric drift mid-run. Regenerating rubrics changes the objective without changing the config.
Ignoring judge cost. N * G judge calls per prompt per step is a capacity plan, and discovering that mid-run stalls the training fleet.

References¶

Gunjal et al., Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains: https://arxiv.org/abs/2507.17746
Arora et al., HealthBench: Evaluating Large Language Models Towards Improved Human Health (physician-written rubric criteria): https://arxiv.org/abs/2505.08775
Shao et al., DeepSeekMath (GRPO and group-relative advantage): https://arxiv.org/abs/2402.03300
Yu et al., DAPO (dynamic sampling for zero-advantage groups): https://arxiv.org/abs/2503.14476
TRL GRPO Trainer documentation (custom reward functions): https://huggingface.co/docs/trl/grpo_trainer
Cameron R. Wolfe, Rubric-Based Rewards for RL: https://cameronrwolfe.substack.com/p/rubric-rl
Cameron R. Wolfe, Reinforcement Learning for LLMs: The Complete Guide: https://cameronrwolfe.substack.com/p/llm-rl