Inference-time scaling for reasoning: temperature, self-consistency, self-refinement¶

Markdown

Scope: three techniques that trade inference compute for answer quality without any weight update, controlling the sampling distribution with temperature and top-p (nucleus) filtering, self-consistency majority voting across multiple sampled answers, and confidence-scored self-refinement, an iterative critique-then-revise loop gated by the model's own average log-probability. This is the sibling of rejection sampling and Best-of-N (which needs an external scorer to pick the best of N) and of RLVR's pass@k metric (which measures exactly what self-consistency tries to approximate with one answer); it is distinct from the agentic reflection loop, which recovers from tool-call failures using external signals, where self-refinement here critiques the content of an answer using only the model itself.

The numpy/stdlib blocks below are executed and asserted in this page (Python 3.9+, numpy 2.4+, no other dependencies); they model the sampling and scoring mechanics on synthetic logits and toy log-probability sequences, standing in for a real model's forward pass so the core maths is verifiable without a live model. The technique lineage (temperature/top-p sampling, self-consistency, self-refinement) follows Sebastian Raschka's Build a Reasoning Model (From Scratch) (Manning, 2026, Ch. 4-5) and the primary papers cited in References; the code is an independent implementation.

What it is¶

All three techniques spend extra inference compute at generation time to improve an answer, none of them change the model's weights.

Temperature and top-p control how tokens are sampled: temperature rescales logits before the softmax (low temperature sharpens toward the argmax, high temperature flattens toward uniform), and top-p keeps only the smallest set of highest-probability tokens whose cumulative mass exceeds p, discarding the long low-probability tail. Neither improves a single sample on its own; they exist to make sampling multiple diverse answers useful.
Self-consistency samples several answers at temperature greater than zero, extracts the final answer from each (the math-verifier extraction step reused here), and takes the majority vote. It is pure frequency counting: no reward model, no verifier score, just "which answer did the model land on most often."
Self-refinement takes a single draft answer, has the model critique its own answer, then revise it using that critique, and only keeps the revision if a confidence score says it is not worse than what came before. The confidence score used here is the average log-probability of the answer's own tokens, a length-normalized measure of how strongly the model backs its own output.

Why use it¶

No training required. All three are inference-only; they ship the same day as the base model and compose with any post-training stage (RLVR, distillation) rather than replacing it.
Self-consistency converts variance into accuracy. A single greedy sample can land on a wrong answer the model would get right more often than not; sampling several and voting exploits the fact that correct reasoning paths tend to agree with each other while wrong ones scatter, the empirical basis of the original paper's accuracy gains.¹
Self-refinement recovers from a bad first draft without resampling from scratch. Instead of throwing away a wrong answer and hoping a fresh sample does better, the model is shown its own attempt and asked to fix it, closer to how a person checks their own work.²
The confidence gate is cheap. Average log-probability falls out of the same forward pass that already produced the tokens; no extra model call, no reward model, no verifier is needed to decide whether to accept a revision.

When to use it (and when not)¶

Use temperature and top-p together whenever you need diverse samples from the same prompt (self-consistency, Best-of-N); for a single best-effort answer, low temperature (or greedy) is usually better.
Use self-consistency when the answer space is small and checkable by equality (a boxed number, a short code snippet's output) and you can afford N full generations per prompt; it needs no verifier or reward model, only a way to compare two final answers.
Prefer Best-of-N instead of self-consistency when you have a verifier or reward model and want to pick the single best-scoring sample rather than the most frequent answer; the two are complementary, and a confidence score can break a self-consistency tie the way it does below.
Use self-refinement when a single answer might be wrong in a fixable way (an arithmetic slip, a missed constraint) and iteration cost is acceptable; skip it for tasks where the model has no way to detect its own error, refinement cannot fix a mistake the model does not recognize as one.
Do not treat logprob confidence as ground truth. It measures the model's own certainty, not correctness; a confidently wrong answer scores higher than a hesitant correct one (demonstrated adversarially below). Use it as a tiebreaker or gate, not a verifier.
Avoid stacking all three uncritically. Self-consistency over N samples, each optionally self-refined over k iterations, costs on the order of N * k generations; measure the accuracy-per-token trade-off before scaling either knob further.

Architecture¶

flowchart LR
  PROMPT["Prompt"] --> SAMPLE["Sample N completions<br/>(temperature > 0, top-p filter)"]
  SAMPLE --> EXTRACT["Extract final answer from each<br/>(math-verifier extraction)"]
  EXTRACT --> VOTE{"Self-consistency vote"}
  VOTE -->|"clear majority"| WINNER["Winning answer"]
  VOTE -->|"tie"| TIEBREAK["Confidence tiebreak<br/>(avg logprob per tied answer)"]
  TIEBREAK --> WINNER
  WINNER --> REFINE["Self-refinement loop"]
  REFINE --> CRITIQUE["LLM critiques its own answer"]
  CRITIQUE --> REVISE["LLM revises using the critique"]
  REVISE --> SCORE{"revised score >= current score?"}
  SCORE -->|"yes"| ACCEPT["Accept revision, continue"]
  SCORE -->|"no"| REJECT["Keep prior answer, continue"]
  ACCEPT --> FINAL["Final answer"]
  REJECT --> FINAL

How to use it¶

1. Temperature and top-p: shaping the sampling distribution¶

# 01_temp_topp.py -- runnable (numpy only). Temperature scaling and nucleus filtering.
import numpy as np


def softmax(logits):
    z = logits - logits.max()
    e = np.exp(z)
    return e / e.sum()


def temperature_scale(logits, temperature):
    """temperature -> 0 sharpens toward argmax (greedy); temperature > 1 flattens."""
    assert temperature > 0.0, "temperature must be positive"
    return softmax(logits / temperature)


def entropy(probs):
    p = probs[probs > 0]
    return float(-np.sum(p * np.log(p)))


def top_p_filter(probs, p):
    """Keep the smallest prefix of sorted-descending probabilities whose
    cumulative mass exceeds p; zero out the rest and renormalize."""
    assert 0.0 < p <= 1.0
    order = np.argsort(-probs)
    sorted_probs = probs[order]
    cumulative = np.cumsum(sorted_probs)
    cutoff = min(int(np.searchsorted(cumulative, p)) + 1, len(probs))
    kept = np.zeros_like(probs)
    kept[order[:cutoff]] = sorted_probs[:cutoff]
    return kept / kept.sum()


logits = np.array([2.0, 1.0, 0.1, -1.0, -3.0])

# --- temperature reshapes mass without changing the argmax ------------------
low_t, high_t = temperature_scale(logits, 0.3), temperature_scale(logits, 2.0)
assert np.argmax(low_t) == np.argmax(high_t) == np.argmax(logits)
assert entropy(low_t) < entropy(softmax(logits)) < entropy(high_t)

# --- temperature -> 0 approaches one-hot (greedy) ---------------------------
assert temperature_scale(logits, 0.01)[np.argmax(logits)] > 0.999

# --- top-p=1.0 keeps every token; smaller p keeps strictly fewer -----------
probs = softmax(logits)
assert np.count_nonzero(top_p_filter(probs, 1.0)) == len(probs)
assert np.count_nonzero(top_p_filter(probs, 0.5)) < len(probs)

# --- ADVERSARIAL: a single dominant token is kept regardless of how small p is
skewed = softmax(np.array([10.0, 0.0, 0.0, 0.0]))
tiny_p = top_p_filter(skewed, 0.01)
assert tiny_p[0] == 1.0 and np.count_nonzero(tiny_p) == 1

print("01_temp_topp: all asserts passed")

2. Self-consistency: majority vote, with a confidence tiebreak¶

# 02_self_consistency.py -- runnable (stdlib + numpy). Majority vote plus a
# logprob-confidence tiebreak (the book leaves this as an exercise; here it
# is implemented and adversarially tested).
from collections import Counter

import numpy as np


def self_consistency_vote(short_answers):
    counts = Counter(short_answers)
    if not counts:
        return None, []
    top_freq = counts.most_common(1)[0][1]
    winners = [ans for ans, freq in counts.items() if freq == top_freq]
    return (winners[0] if len(winners) == 1 else None), winners


def avg_logprob_answer(token_logprobs):
    """Length-normalized confidence: mean (not sum) of per-token logprobs, so
    answers of different lengths compare fairly."""
    assert len(token_logprobs) > 0
    return float(np.mean(token_logprobs))


def resolve_tie_with_confidence(tied_answers, per_answer_logprobs):
    scored = {a: avg_logprob_answer(per_answer_logprobs[a]) for a in tied_answers}
    return max(scored, key=scored.get), scored


# --- clear majority: no tiebreak needed --------------------------------------
winner, ties = self_consistency_vote(["83", "22", "83", "54", "83"])
assert winner == "83" and ties == ["83"]

# --- tie: vote alone returns None, tied candidates are exposed --------------
winner, ties = self_consistency_vote(["83", "22", "83", "22"])
assert winner is None and set(ties) == {"83", "22"}

# --- confidence tiebreak: "83" was generated with tighter (less negative) logprobs
per_answer_logprobs = {"83": [-0.02, -0.01, -0.05, -0.03], "22": [-1.90, -2.40, -0.80, -1.10]}
resolved, scores = resolve_tie_with_confidence(ties, per_answer_logprobs)
assert resolved == "83" and scores["83"] > scores["22"]

# --- length normalization: mean, not sum, so a longer confident answer isn't
# unfairly penalized against a short one at the same per-token confidence ---
assert abs(avg_logprob_answer([-0.05, -0.05]) - avg_logprob_answer([-0.05] * 20)) < 1e-9

# --- ADVERSARIAL: logprob confidence measures the model's OWN certainty, not
# ground truth, a confidently wrong sample can still outscore a hesitant right one
adversarial = {"wrong_but_confident": [-0.01, -0.01, -0.01], "right_but_hesitant": [-2.0, -2.5, -1.8]}
resolved_adv, _ = resolve_tie_with_confidence(list(adversarial), adversarial)
assert resolved_adv == "wrong_but_confident"   # confirms the known failure mode, not a bug

assert self_consistency_vote([]) == (None, [])   # no samples: must not crash

print("02_self_consistency: all asserts passed")

# 03_self_refinement.py -- runnable (stdlib only). The accept/reject gate that
# turns a critique-then-revise pass into a monotonically-non-decreasing loop.
def self_refinement_loop(draft, draft_score, revisions):
    """revisions: [(revised_answer, revised_score), ...] already produced by an
    LLM critique-then-revise pass, one entry per iteration. This function owns
    only the accept/reject decision."""
    current, current_score = draft, draft_score
    history = [{"iteration": 0, "answer": current, "score": current_score, "accepted": True}]
    for i, (revised, revised_score) in enumerate(revisions, start=1):
        accept = revised_score >= current_score
        if accept:
            current, current_score = revised, revised_score
        history.append({"iteration": i, "answer": revised, "score": revised_score, "accepted": accept})
    return current, current_score, history


# --- the textbook case: a confident revision replaces a weak draft ---------
draft_score = sum([-1.2, -1.0, -1.5]) / 3
revised_score = sum([-0.2, -0.1, -0.3]) / 3
final, final_score, history = self_refinement_loop("10", draft_score, [("83", revised_score)])
assert final == "83" and history[-1]["accepted"] is True and final_score > draft_score

# --- a second iteration that makes things WORSE must be rejected -----------
worse_score = sum([-3.0, -2.8, -3.2]) / 3
final2, final2_score, history2 = self_refinement_loop("83", revised_score, [("38", worse_score)])
assert final2 == "83" and final2_score == revised_score and history2[-1]["accepted"] is False

# --- score sequence is monotonically non-decreasing across accepted steps --
scores_over_time = [-1.1, -0.8, -0.5, -0.9, -0.2]   # one dip (must be rejected), rest accepted
running, kept = scores_over_time[0], [scores_over_time[0]]
for s in scores_over_time[1:]:
    running = s if s >= running else running
    kept.append(running)
assert kept == sorted(kept) and kept[-1] == max(scores_over_time)

# --- no scorer: score fixed at 0.0, so every revision is accepted (0.0 >= 0.0)
no_score_final, _, _ = self_refinement_loop("x", 0.0, [("y", 0.0), ("z", 0.0)])
assert no_score_final == "z"

print("03_self_refinement: all asserts passed")

How to develop with it¶

Tune temperature and top-p together for self-consistency, not independently. A common starting point is temperature 0.7-0.9 with top-p 0.9: high enough to diversify samples, low enough that individual samples stay coherent. Sweep both on a held-out set before fixing defaults.
Set the self-consistency sample count N from the accuracy-per-token curve, not a round number. Returns diminish; measure pass@1 accuracy after voting at N = 3, 5, 10, ... and stop where the marginal gain no longer justifies N extra generations (pass@k estimator, RLVR).
Cap self-refinement iterations and watch for oscillation. A revision can also be rejected; if consecutive iterations keep proposing worse answers, that is a signal to stop early rather than spend the full iteration budget.
Prefer a verifiable-reward tiebreak over a logprob tiebreak when one is available. If the task has a math verifier or unit-test checker, use it to break self-consistency ties instead of confidence; confidence is the fallback for tasks with no verifier, not the first choice.

How to run it in production¶

Cost is N (or N * k) full generations per request. Self-consistency at N=5 and self-refinement at k=2 iterations both multiply the per-request generation cost by their factor; this is a latency and GPU-hour cost, not a training cost, and belongs in the same serving-capacity math as any other multi-sample decoding strategy (inference serving).
Parallelize the N samples, do not serialize them. Self-consistency's N completions for one prompt are independent and batch naturally on a continuous-batching engine (continuous batching); self-refinement's critique and revise steps are sequential within one candidate but independent across candidates if refining several drafts at once.
Self-refinement adds request-level latency, not just cost. Each iteration is a full extra round trip through the model; for latency-sensitive serving, bound the iteration count tightly or move self-refinement to an offline/batch path.
Log the acceptance rate. If self-refinement's accept-if-not-worse gate rejects most revisions, the critique step is not adding signal and the extra generations are being spent for nothing, a signal to drop iterations or change the critique prompt.

How to maintain it¶

Re-tune temperature, top-p, and the self-consistency sample count whenever the base model changes; a new model's calibration and output-length habits shift what "diverse but coherent" means at a given temperature. Keep the confidence-tiebreak and self-refinement acceptance-rate metrics in the same dashboard as generation cost, since both techniques trade a measurable amount of compute for an accuracy gain that can silently stop paying for itself after a model upgrade.

Failure modes¶

Confidence rewards certainty, not correctness. A confidently wrong sample beats a hesitant correct one under logprob scoring (demonstrated adversarially above); never substitute it for an actual verifier when one exists.
Unresolved self-consistency ties. Without a tiebreak, a genuine tie returns no answer at all; decide up front whether to tiebreak by confidence, by first-appearance order, or to abstain.
Self-refinement degrading a correct draft. Without the accept-if-not-worse gate, a "revision" can replace a correct answer with a plausible-looking wrong one; always gate on a score, never accept unconditionally.
Systematic self-consistency failure. If the model has one dominant wrong reasoning path, most samples agree on the same wrong answer and majority voting confidently returns it; self-consistency amplifies the mode, it does not detect systematic error.
Runaway cost from stacking. Self-consistency times self-refinement times a large iteration budget grows generation cost multiplicatively for shrinking marginal accuracy; measure before scaling either knob.

References¶

Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models: https://arxiv.org/abs/2203.11171
Madaan et al., Self-Refine: Iterative Refinement with Self-Feedback: https://arxiv.org/abs/2303.17651
Holtzman et al., The Curious Case of Neural Text Degeneration (nucleus/top-p sampling): https://arxiv.org/abs/1904.09751
Wei et al., Chain-of-Thought Prompting Elicits Reasoning in LLMs: https://arxiv.org/abs/2201.11903
Raschka, Build a Reasoning Model (From Scratch), Manning Publications, 2026 (Ch. 4-5).

Wang et al.: sampling diverse reasoning paths and taking a majority vote over final answers improves accuracy over a single greedy decode, because correct reasoning paths tend to converge while incorrect ones scatter. https://arxiv.org/abs/2203.11171 ↩
Madaan et al.: an LLM iteratively critiques and revises its own output without additional training or a separate reward model, generate, critique, revise. https://arxiv.org/abs/2303.17651 ↩