Markdown

Reasoning-effort control¶

Scope: the training-time and serving-time machinery behind a reasoning_effort: low | high | max switch. This page covers the four reward shapes that produce effort levels (an absolute length penalty, a group-relative length penalty, a per-problem reward override, and a phase-alternating gate), the two ways those levels get packaged into one shipped checkpoint (separate specialists plus distillation, or a single conditioned policy), and how to pick an operating point from request feasibility and fleet efficiency. The sibling pages are inference-time scaling, which spends compute without touching weights, and RLVR, which supplies the correctness half of every reward here. Consolidating the trained specialists is multi-teacher on-policy distillation.

The numpy block below is executed and asserted in this page (Python 3.11, numpy 2.x, no other dependencies). It implements the published reward equations on synthetic rollouts, so the reward shapes and their edge cases are verifiable without a training run. It does not reproduce anyone's benchmark numbers; those are quoted from the cited reports and are not independently measured here.

What it is¶

Reasoning-effort control is the ability to ask a reasoning model for a specific amount of thinking and get it. It exists at two layers that are easy to conflate:

The interface layer is a knob. It can be a system-prompt line (Reasoning effort: low, the GPT-OSS style), a structured option message (Kimi K3 exposes effort as a thinking-effort global option in its XTML chat template, placed before all input messages because it governs the whole session), a mode token pair (/think and /no_think in Qwen3's thinking-mode fusion), or a number of tokens written into the prompt (L1's Think for 512 tokens).
The behaviour layer is what makes the knob mean anything. A knob with no training behind it is decoration. The behaviour comes from a length-aware reward during RL, which is the actual subject of this page.

Four reward shapes cover essentially all of the published work:

Shape	Signal	Source
Absolute deviation from a prompted target	`I(correct) - alpha * abs(n_target - n_out)`	L1 / LCPO-Exact
Soft one-sided budget	`I(correct) * clip(alpha*(n_target - n_out) + delta, 0, 1)`	L1 / LCPO-Max
Group-relative rank within a rollout group	shortest correct gets `+0.5`, longest gets `-0.5`	Kimi k1.5
Per-problem reward cliff	reward becomes `-1` past `tau * b0(x)`	Kimi K3

On top of those sits Toggle (Kimi K2.5), which is not a fifth shape but a schedule: it alternates budget-constrained iterations with unconstrained ones to stop the model from forgetting how to use a large budget.

Why use it¶

Thinking tokens are a major cost line on a reasoning deployment. A model at max effort can emit several times the tokens of the same model at low effort for the same prompt. Kimi K2.5 reports that Toggle cuts output tokens by 25 to 30% on average with negligible reported performance impact. That is a reduction in generated-token work, not necessarily in total request cost: prefill, input tokens, batching effects, and fixed overhead remain.
Overthinking is a real training pathology, not a user preference. Kimi k1.5 names it explicitly: response length grows steadily during RL, "although this leads to better performance, an excessively lengthy reasoning process is costly during training and inference." The length reward exists to stop the growth, not to please anyone's taste.
Effort is a low-friction quality knob, when the model was trained to honour it. Changing effort is a request parameter; changing models is a deployment. That makes it a cheap thing to try for a traffic tier that needs a few more points of accuracy. It is not automatically a cheap thing that works: on some models the knob barely moves token count at all (see the ceiling caveat below), so treat it as a hypothesis to measure rather than a lever you can assume.
Length control transfers out of distribution. L1 trains only on math and finds the length-following behaviour generalises to logical reasoning and to MMLU, so a budget knob trained in one domain is worth something in another.
A short-CoT model distilled from a long-CoT model beats a native short-CoT model. This is the k1.5 long2short result: the short variant reaches 60.8 on AIME and 94.6 on MATH500, well past the short-CoT models it is compared against. Effort control is therefore also a capability technique, not only a cost technique.

When to use it (and when not)¶

Use it when:

You serve a mixed workload where a large fraction of requests are easy. Routing those to low effort is close to free accuracy-neutral savings.
You have a tight per-request latency SLO and want the model to finish before the serving cap more often. Length-aware training reduces truncation risk; only the engine's decode limit provides a hard bound.
You are already running RLVR and can afford one more reward term. The marginal cost of adding a length term to an existing verifiable reward is small.

Do not use it when:

You have no verifier. Every shape here multiplies or adds to a correctness signal. With a noisy or absent correctness term, the length term dominates and you train a model that is confidently brief and wrong.
You need a guaranteed hard cap. These are soft trained behaviours. L1 reports low but nonzero mean deviation from the requested length. If the cap is a contractual limit, enforce it in the serving layer as well.
Your traffic is uniformly hard. If nearly every request needs max effort, the machinery buys you nothing and the budget-constrained phases cost you accuracy.
You are training a single narrow-budget model and expect it to scale up later. This is the length-overfitting failure K2.5 documents: "models trained under rigid budget constraints often fail to generalize to higher compute scales", defaulting to truncated reasoning even when given room. Use an alternating schedule instead.

Architecture¶

flowchart TB
  subgraph TRAIN["Post-training"]
    SFT["SFT cold start"]
    RL["RLVR loop"]
    LEN["Length-aware reward term<br/>(one of the four shapes)"]
    EXP["Effort specialists<br/>low / high / max"]
    SFT --> RL
    LEN --> RL
    RL --> EXP
  end
  subgraph SHIP["Packaging"]
    MERGE["Consolidate:<br/>multi-teacher on-policy distillation<br/>or weight merge"]
    ONE["Single conditioned checkpoint"]
    EXP --> MERGE --> ONE
  end
  subgraph SERVE["Serving"]
    KNOB["effort knob in the request<br/>(option message / system line / token target)"]
    ROUTE["Router: difficulty to effort tier"]
    CAP["Hard token cap in the engine<br/>(the only hard decode bound)"]
    ONE --> KNOB
    ROUTE --> KNOB
    KNOB --> CAP
  end

The load-bearing point of the diagram is that the knob at serving time and the reward at training time have to agree on what low means. If RL trained low against a budget of roughly 3k thinking tokens and the serving tier caps at 1k, the model is being asked for a mode it was never trained on and will truncate rather than compress.

How it works: the reward shapes, validated¶

import numpy as np


# --- L1 / LCPO-Exact, Eq. 1 of arXiv:2503.04697 -------------------------------
def lcpo_exact(correct: np.ndarray, n_out: np.ndarray, n_target: int,
               alpha: float) -> np.ndarray:
    return correct.astype(float) - alpha * np.abs(n_target - n_out)


# --- L1 / LCPO-Max, Eq. 2 -----------------------------------------------------
def lcpo_max(correct: np.ndarray, n_out: np.ndarray, n_target: int,
             alpha: float, delta: float = 0.5) -> np.ndarray:
    soft = np.clip(alpha * (n_target - n_out) + delta, 0.0, 1.0)
    return correct.astype(float) * soft


# --- Kimi k1.5 group length reward -------------------------------------------
def k15_length_reward(correct: np.ndarray, lengths: np.ndarray) -> np.ndarray:
    lo, hi = lengths.min(), lengths.max()
    if hi == lo:
        return np.zeros(len(lengths))
    lam = 0.5 - (lengths - lo) / (hi - lo)
    return np.where(correct, lam, np.minimum(0.0, lam))


# --- Kimi K3 per-problem budget override -------------------------------------
def k3_budget_override(task_reward: np.ndarray, tokens: np.ndarray,
                       b0: float, tau: float) -> np.ndarray:
    return np.where(tokens > tau * b0, -1.0, task_reward)


# --- Kimi K2.5 Toggle gate ----------------------------------------------------
def toggle_reward(rewards: np.ndarray, lengths: np.ndarray,
                  budget: float, lam: float) -> np.ndarray:
    group_pass = rewards.mean()
    keep = (group_pass < lam) | (lengths <= budget)
    return rewards * keep


def toggle_phase(t: int, m: int) -> int:
    """0 = budget-limited Phase0, 1 = free-scaling Phase1."""
    return (t // m) % 2


def toggle_step(t: int, m: int, rewards: np.ndarray, lengths: np.ndarray,
                budget: float, lam: float) -> np.ndarray:
    if toggle_phase(t, m) == 1:
        return rewards.copy()
    return toggle_reward(rewards, lengths, budget, lam)


def toggle_budget(rewards: np.ndarray, lengths: np.ndarray, rho: float) -> float:
    """Percentile rho of the lengths of the CORRECT rollouts only."""
    correct = lengths[rewards > 0]
    assert correct.size, "no correct rollout: budget is undefined for this prompt"
    return float(np.percentile(correct, rho))


ALPHA_EXACT = 0.0003
N_TARGET = 1000

# 1. LCPO-Exact peaks exactly at the target and is symmetric around it.
lens = np.arange(200, 1801, 50)
r = lcpo_exact(np.ones_like(lens, dtype=bool), lens, N_TARGET, ALPHA_EXACT)
assert lens[r.argmax()] == N_TARGET
assert np.isclose(lcpo_exact(np.array([True]), np.array([800]), N_TARGET, ALPHA_EXACT),
                  lcpo_exact(np.array([True]), np.array([1200]), N_TARGET, ALPHA_EXACT))

# 2. The Eq. 1 crossover: a correct answer far enough off-target scores BELOW a
#    wrong answer that lands on the target. Solve alpha*|d| > 1  =>  |d| > 1/alpha.
d_cross = 1.0 / ALPHA_EXACT
wrong_on_target = lcpo_exact(np.array([False]), np.array([N_TARGET]), N_TARGET, ALPHA_EXACT)[0]
assert wrong_on_target == 0.0
just_inside = lcpo_exact(np.array([True]), np.array([N_TARGET + int(d_cross) - 100]),
                         N_TARGET, ALPHA_EXACT)[0]
just_outside = lcpo_exact(np.array([True]), np.array([N_TARGET + int(d_cross) + 100]),
                          N_TARGET, ALPHA_EXACT)[0]
assert just_inside > wrong_on_target > just_outside, (just_inside, just_outside)

# 3. LCPO-Max: zero for every incorrect answer, exactly delta at the budget, and
#    a correct answer overshooting by more than delta/alpha also collapses to 0.
ALPHA_MAX = 0.0003
assert lcpo_max(np.array([False]), np.array([100]), N_TARGET, ALPHA_MAX)[0] == 0.0
assert lcpo_max(np.array([True]), np.array([N_TARGET]), N_TARGET, ALPHA_MAX)[0] == 0.5
overshoot = int(0.5 / ALPHA_MAX) + 100
assert lcpo_max(np.array([True]), np.array([N_TARGET + overshoot]), N_TARGET, ALPHA_MAX)[0] == 0.0

# 3b. The ceiling is reached only by undershooting (1 - delta)/alpha tokens, which
#     a 1k budget cannot even express: being 900 tokens short still scores 0.77.
assert np.isclose(lcpo_max(np.array([True]), np.array([100]), N_TARGET, ALPHA_MAX)[0], 0.77)
saturating_target = 4096
undershoot_needed = (1.0 - 0.5) / ALPHA_MAX
assert undershoot_needed > N_TARGET  # unreachable at a 1k budget
assert lcpo_max(np.array([True]), np.array([saturating_target - int(undershoot_needed) - 1]),
                saturating_target, ALPHA_MAX)[0] == 1.0

# 4. The vanishing-gradient trap: a GRPO group where every rollout is either
#    wrong or badly over budget has identical rewards, so the advantage is zero.
grp_correct = np.array([False, False, True, True])
grp_len = np.array([1400, 1600, N_TARGET + overshoot, N_TARGET + overshoot + 500])
grp_r = lcpo_max(grp_correct, grp_len, N_TARGET, ALPHA_MAX)
assert grp_r.std() == 0.0 and grp_r.sum() == 0.0

# 5. Kimi k1.5: shortest response gets +0.5, longest -0.5, and a wrong answer
#    can never earn a positive length reward.
c = np.array([True, True, True, False])
L = np.array([400, 700, 1200, 400])
lr = k15_length_reward(c, L)
assert np.isclose(lr[0], 0.5) and np.isclose(lr[2], -0.5)
assert lr[3] <= 0.0
assert np.all(k15_length_reward(c, np.full(4, 800)) == 0.0)

# 6. Kimi K3: the reward override is a cliff, not a slope. One token past
#    tau*b0 flips a perfect trajectory's training reward to -1.
b0, tau = 4000.0, 1.5
tok = np.array([5999, 6000, 6001])
kr = k3_budget_override(np.ones(3), tok, b0, tau)
assert list(kr) == [1.0, 1.0, -1.0]

# 7. Kimi K2.5 Toggle: when the group is solving the problem (pass rate >= lam),
#    only within-budget rollouts keep their reward; when the group is failing,
#    the budget gate is lifted so learning signal survives.
rw = np.array([1.0, 1.0, 1.0, 0.0])
ln = np.array([500, 900, 2500, 600])
solved = toggle_reward(rw, ln, budget=1000, lam=0.5)
assert list(solved) == [1.0, 1.0, 0.0, 0.0]
hard = toggle_reward(np.array([1.0, 0.0, 0.0, 0.0]), ln, budget=1000, lam=0.5)
assert list(hard) == [1.0, 0.0, 0.0, 0.0]
assert toggle_reward(np.array([0.0, 0.0, 0.0, 1.0]), np.array([2500, 2500, 2500, 2500]),
                     budget=1000, lam=0.5).sum() == 1.0

# 7b. Phase alternation every m iterations, and the budget percentile is taken
#     over correct rollouts only (a wrong short answer must not set the budget).
assert [toggle_phase(t, m=4) for t in range(12)] == [0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0]
assert list(toggle_step(0, 4, rw, ln, 1000, 0.5)) == [1.0, 1.0, 0.0, 0.0]
assert list(toggle_step(5, 4, rw, ln, 1000, 0.5)) == list(rw)  # Phase1 lifts the gate
b_est = toggle_budget(np.array([1.0, 1.0, 1.0, 0.0]), np.array([500, 900, 2500, 120]), rho=50)
assert b_est == 900.0, b_est

# 8. Budget adherence: mean absolute deviation from the requested length.
def mean_deviation(requested: np.ndarray, produced: np.ndarray) -> float:
    return float(np.mean(np.abs(requested - produced)))

req = np.array([512, 1024, 2048, 3600])
got = np.array([548, 1001, 2110, 3475])
assert abs(mean_deviation(req, got) - 61.5) < 1e-9, mean_deviation(req, got)

# 9. Per-request feasibility and fleet efficiency are different questions.
def request_feasibility(tokens: dict, request_cap: float) -> dict:
    return {k: tokens[k] <= request_cap for k in tokens}


def correct_answers_per_million_generated_tokens(acc: dict, tokens: dict) -> dict:
    return {k: 1_000_000.0 * acc[k] / tokens[k] for k in acc}

acc = {"low": 0.62, "high": 0.74, "max": 0.78}
tok_per_q = {"low": 3_000.0, "high": 11_000.0, "max": 26_000.0}
feasible = request_feasibility(tok_per_q, request_cap=11_000.0)
assert feasible == {"low": True, "high": True, "max": False}
best_feasible = max((k for k in acc if feasible[k]), key=acc.get)
assert best_feasible == "high"
fleet_efficiency = correct_answers_per_million_generated_tokens(acc, tok_per_q)
assert max(fleet_efficiency, key=fleet_efficiency.get) == "low"

print("all reasoning-effort reward assertions passed")
print("  LCPO-Exact crossover distance (tokens):", int(d_cross))
print("  LCPO-Max zero-reward overshoot (tokens):", int(0.5 / ALPHA_MAX))
print("  feasible at 11k/request:", feasible, "best:", best_feasible)
print("  correct answers per 1M generated tokens:",
      {k: round(v, 2) for k, v in fleet_efficiency.items()})

Executed output:

all reasoning-effort reward assertions passed
  LCPO-Exact crossover distance (tokens): 3333
  LCPO-Max zero-reward overshoot (tokens): 1666
  feasible at 11k/request: {'low': True, 'high': True, 'max': False} best: high
  correct answers per 1M generated tokens: {'low': 206.67, 'high': 67.27, 'max': 30.0}

Four results from that block are worth reading carefully, because each is a decision you have to make when you implement this.

Assertion 2 is the reason alpha is the hyperparameter that matters. Under LCPO-Exact the reward is I(correct) - alpha*|d|, so a correct answer that misses the target by more than 1/alpha tokens scores below a wrong answer that hits the target. At alpha = 3e-4 that crossover is 3333 tokens, comfortably outside the range a length-following model wanders into. Set alpha an order of magnitude higher and the crossover moves inside the operating range, at which point you are training a model to prefer being wrong on time over being right late. The paper is explicit about the intent: "a lower value of alpha prioritizes correctness when it is critical, whereas a higher value enforces stricter adherence to the length constraint."

Assertion 3b explains why LCPO-Max's clip(..., 0, 1) upper bound is mostly theoretical at small budgets. The reward only saturates at 1.0 when the model undershoots by (1 - delta)/alpha = 1666 tokens. At a 1000-token budget that is unreachable, so the effective reward range for correct answers is roughly [0, 0.8] and the gradient toward brevity stays alive across the whole range. At a 4096-token budget the ceiling is reachable and the incentive to compress further disappears past that point. The interaction between alpha, delta and your typical budget decides where the model stops trying.

Assertion 4 is the failure mode that will actually bite you. LCPO-Max returns exactly 0 for every incorrect answer and for every correct answer more than delta/alpha over budget. Feed a group of four rollouts where all of them are one or the other into GRPO, and the group standard deviation is zero, so every advantage is zero and the step teaches nothing. This is the same degenerate-group condition GRPO has generally, but the length term makes it much easier to hit, because a hard prompt where the model reliably overshoots produces an all-zero group even when some answers were correct. Watch frac_reward_zero_std.

Assertion 6 shows the K3 reward override is a cliff. Kimi K3 associates each problem x with an initial budget b0(x) estimated from the cold-start model, then "override[s] the task reward with -1 for trajectories whose total token budget T(y) exceeds a scaled threshold tau * b0(x)". There is no ramp. One token past the threshold turns a fully correct long-horizon trajectory into the worst possible training outcome. This is a strong learned preference, not a decode-time constraint: the policy can still generate past the threshold, and only the serving engine can terminate it. Note also what T(y) counts: thinking tokens for general tasks, but for agentic tasks "the cumulative output tokens, including both reasoning traces and tool-call arguments", so a verbose tool call spends the same reward budget as verbose thinking.

How to use it: pick the operating point from the cost model¶

Assertion 9 separates two calculations that must not be collapsed into a fractional response. For a per-request cap of 11k generated tokens, low and high are feasible and max is not; among the feasible modes, high has the best accuracy. For a saturated decode workload, divide expected correct answers by complete-response generation: the synthetic low mode yields 206.67 correct answers per million generated tokens, against 67.27 for high and 30.00 for max. A partial max response is neither 42% of an answer nor 42% of its accuracy, so multiplying accuracy by a budget fraction is not a valid cost metric. This metric excludes prefill and fixed serving costs.

Vendor reports provide workload-specific points, not a common cost curve. Kimi K3 compares selected high-effort results with other systems at stated cost ratios, while DeepSeek-V4 reports that its Max mode outperforms High on its most challenging tasks. Those different prompts, models and accounting methods do not establish that one tier wins a general cost-normalised aggregate. Routing decisions require measurements on the deployment's own workload.

Verify the knob moves anything before you build a cost model on it¶

Every calculation above assumes the effort setting actually changes how many tokens the model emits. On at least one open model family that assumption fails. A study of GPT-OSS-20B and GPT-OSS-120B across three effort levels and six reasoning tasks ran an explicit manipulation check, measuring the high-to-low fold change in mean token count, and found it "small in most (model x task) cells": on both arithmetic tasks the token counts "varied negligibly across effort levels" at a fold change of roughly 1.0.¹ Larger absolute differences appeared on tasks with longer baseline traces, but remained small next to the order-of-magnitude differences between tasks.

The authors' interpretation is the sentence to carry into a design review: the parameter "functions less as a real-time computational dial and more as an upper budget on how many tokens the model may generate, while the actual allocation across items is largely determined by training-time policy."¹ They note this converges with a separate report that reasoning_effort has only limited behavioural consequences.

This is not evidence that effort control is a fiction. It is evidence that the interface and the behaviour are separate things, which is the distinction this page opens with. A model trained with an explicit budget reward (the Kimi K3 -1 override, the DeepSeek-V4 per-mode length penalties) has behaviour behind the knob. A model that merely exposes the parameter may be giving you a ceiling that most requests never approach, in which case the effort tier changes your worst case and not your mean, and a cost model built on mean tokens per tier will be wrong.

Two caveats on the evidence itself. The study measures GPT-OSS specifically, so it does not transfer to the models whose budget-RL recipes this page documents. And its subject is cognitive-cost alignment with human reaction times, not serving economics; the manipulation check is a methodological step in that argument rather than a serving benchmark. The practical instruction it supports is narrow and worth following anyway: before you route traffic by effort tier, measure the completion-token distribution per tier on your own prompts and confirm the tiers actually separate. If they do not, you have a ceiling, not a dial.

Practical recipe:

Measure accuracy and mean completion tokens per effort level on your eval, not the vendor's.
For a per-request limit, discard infeasible modes and choose the highest-accuracy feasible one. For fleet capacity, compare complete correct answers per token or per GPU-hour.
Split traffic only after measuring a router. Send prompts with demonstrated quality gains at max upward; send the remainder down.
Re-measure after any model update. Effort curves move between checkpoints more than raw accuracy does.

How to develop with it: choosing a shape¶

If you need	Use	Because
A prompt-specified token target the user controls	LCPO-Exact	It is the only shape that rewards hitting a target rather than staying under one
A ceiling with no reward for being needlessly short	LCPO-Max	One-sided, and `delta` keeps small violations preferable to being wrong
Token efficiency with no per-prompt target at all	Kimi k1.5 group reward	Needs only the rollout group; no budget estimation step
A strong per-problem penalty beyond a threshold	Kimi K3 override	The `-1` reward cliff is sharper than a linear length penalty
Efficiency without losing the ability to scale up	K2.5 Toggle	Alternation is the documented fix for length overfitting

Two implementation details are easy to get wrong:

The k1.5 shape is group-relative, so it is free of any budget estimate but also unanchored. Every group produces one response at +0.5 and one at -0.5 regardless of whether the group is uniformly verbose. If the whole group is 20k tokens, the shortest one still gets the full bonus. It shrinks the spread within a group and relies on that pressure compounding over training rather than on an absolute target. The paper also handles the degenerate case explicitly: when max_len == min_len the reward is zero for everyone, which assertion 5 checks.
The Toggle budget must be estimated over correct rollouts only (assertion 7b). The operational set is {|y_j| : r(x, y_j) = 1, j = 1, ..., K}. K2.5's printed equation uses r(x, y_i) inside a set indexed by j; that i/j mismatch leaves j unbound and conflicts with the surrounding statement that the subset contains correct responses. If you compute the percentile over all rollouts, a short wrong answer drags the budget down and trains the model toward the length of its failures. K2.5 estimates the budget once at the start of training and holds it fixed, avoiding a feedback loop where the budget chases the policy it shapes.

Warm-up matters too. Kimi k1.5 found that "length penalty may slow down training during the initial phases" and resolved it by running standard policy optimisation first, then switching on a constant length penalty for the rest of training. Turning the length term on at step zero fights the model while it is still learning to be correct at all.

How to maintain it: keep the specialists, not just the merge¶

Both K3 and DeepSeek-V4 train separate models per effort level and then consolidate, and this is a maintenance decision as much as a quality one.

Kimi K3 crosses three domains (general, general agent, coding) with three effort levels (low, high, max) to get nine expert models, then folds them into one checkpoint with multi-teacher on-policy distillation. Its effort experts come from a stage-wise curriculum over the budget multiplier tau: train a max-budget variant with large tau first while still capping the maximum to suppress overthinking, then anneal tau down to get the high- and low-effort experts. The tau schedule is "configured per domain under human-in-the-loop guidance", so it is not a single global number.
DeepSeek-V4 trains distinct specialists per effort mode with "distinct length penalties and context windows during RL", demarcates the modes with <think> and </think> formats, and for its Think Max mode prepends a specific instruction to the system prompt.

Keeping the specialists means a regression in one effort tier can be diagnosed and retrained without touching the others, and a new effort tier can be added by training one more expert rather than redoing the whole run. If you only keep the merged checkpoint you lose that.

For a lighter-weight path, Kimi k1.5's long2short menu still applies and does not require an RL run per level: merge a long-CoT and short-CoT model by averaging weights (model merging); or sample n = 8 responses and SFT on the shortest correct one (shortest rejection sampling); or build DPO pairs with the shortest correct solution as positive and longer responses (wrong ones, plus correct ones more than 1.5x the positive's length) as negatives; or run a short second RL phase with the length penalty on and the max rollout length cut down.

How to run it in production¶

Expose effort as a first-class request field, not a prompt prefix your gateway splices in. K3's design point is worth copying: effort is a global option placed before all input messages because it governs the session and rarely changes. Splicing it into a user turn puts it inside the prompt cache prefix boundary and can invalidate cache hits on every request.
Keep an engine-side token cap, set above the trained budget where the SLO permits. Learned length rewards never hard-bound decoding. The engine cap is the only hard stop, so track how often it truncates an unfinished answer instead of treating it as a harmless backstop.
Alert on completion-token distribution per effort tier, not just the mean. The failure you care about is bimodality: a tier that mostly respects its budget but occasionally runs 10x is a latency SLO problem that a mean hides.
Budget accounting must match the training definition. If you trained agentic budgets over reasoning plus tool-call arguments (the K3 definition of T(y)), meter the same thing in production or your dashboards will disagree with the policy.
Track reasoning-token share. Reasoning tokens are billed output tokens but are not the answer. A tier where reasoning is the large majority of completion tokens is a candidate for a lower effort level.

Failure modes¶

Length overfitting. Trained under a rigid budget, the model cannot use a larger budget later and produces truncated reasoning even when given room. This is the documented motivation for Toggle. Symptom: accuracy flat or falling as you raise the budget at inference.
Degenerate reward groups. With one-sided shapes, groups where everything is wrong or over budget give zero advantage (assertion 4). Symptom: frac_reward_zero_std climbing, gradient norm collapsing.
alpha set too aggressively. Past 1/alpha inside your operating range, the model prefers a wrong answer of the right length (assertion 2). Symptom: accuracy drops while length adherence improves.
Budget estimated over all rollouts. Contaminates the budget with the lengths of failures, biasing the target short.
Length penalty enabled from step zero. Slows early training; warm up instead.
Reward hacking via verbosity in the judge. For non-verifiable tasks scored by a generative reward model, the model learns to win by writing more. K3 applies a budget-based verbosity control to its judge as well: given a cold-start verbosity estimate l0 and multiplier sigma, a candidate longer than sigma * l0 automatically loses the binary comparison. If your reward model is an LLM, it needs its own length gate.
A knob with no behaviour behind it. The effort parameter may set an upper budget the model rarely approaches rather than steering per-request allocation, in which case tiers do not separate on mean tokens and a cost model built on them is wrong. Check the completion-token distribution per tier before routing on it.
Knob and training disagree. Serving low at a cap tighter than the trained low budget produces truncation, not compression.
Effort labels are not portable. One vendor's high is another's max. Any cross-model comparison that does not state the effort setting is uninterpretable; note that published leaderboard comparisons frequently run different models at different effort settings.

References¶

L1 / LCPO, Controlling How Long A Reasoning Model Thinks With Reinforcement Learning: https://arxiv.org/abs/2503.04697
Kimi k1.5, Scaling Reinforcement Learning with LLMs (length penalty, long2short): https://arxiv.org/abs/2501.12599
Kimi K2.5, Visual Agentic Intelligence (Toggle, token-efficient RL): https://arxiv.org/abs/2602.02276
Kimi K3 Technical Report (reasoning-effort RL, nine experts, XTML thinking-effort option): https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf
DeepSeek-V4, Towards Highly Efficient Million-Token Context Intelligence (effort specialists): https://arxiv.org/abs/2606.19348
Qwen3 Technical Report (thinking-mode fusion, /think and /no_think): https://arxiv.org/abs/2505.09388
Sebastian Raschka, Controlling Reasoning Effort in LLMs: https://magazine.sebastianraschka.com/p/controlling-reasoning-effort-in-llms
Hu and Wang, "Effort as Ceiling, Not Dial: Reasoning Budget Does Not Modulate Cognitive Cost Alignment Between Humans and Large Reasoning Models": https://arxiv.org/abs/2605.16938
S1, Simple test-time scaling (budget forcing baseline L1 is compared against): https://arxiv.org/abs/2501.19393

Hu and Wang (arXiv 2605.16938, 2026-05-16). Across GPT-OSS-20B and GPT-OSS-120B, three effort levels and six reasoning tasks, a manipulation check found the high-to-low fold change in mean token count was small in most model-by-task cells and about 1.0 on both arithmetic tasks. The authors conclude the parameter acts as an upper budget rather than a real-time allocation dial, with allocation largely fixed at training time. The paper's subject is cognitive-cost alignment with human reaction times, not serving cost, and its models are GPT-OSS; it is cited here only for the manipulation-check finding. ↩↩