Markdown

Cookbook: build a math answer verifier for RLVR¶

Scope: implementing the deterministic checker that decides whether an LLM's math answer is correct, extraction of the final \boxed{} answer, canonicalization of its many equivalent surface forms, symbolic equivalence via SymPy, and grading of multi-part answers. This is the concrete verifier that RLVR needs as its reward function and that a MATH-500-style eval needs as its grader (LLM benchmarks, evaluation harness); RLVR's "How to engineer the verifier" section names the requirement, this page builds it.

The pipeline shape (extract, then canonicalize, then symbolically compare, then grade) follows the worked example in Sebastian Raschka's Build a Reasoning Model (From Scratch) (Manning, 2026); the code below is an independent implementation, not a transcription, and is executed and asserted in this page against its own adversarial test suite (stdlib re plus SymPy 1.14+, no other dependencies).

What it is¶

A math verifier is a four-stage pure function from (model_output, ground_truth) to a boolean: extract the model's final answer out of its free-form response, canonicalize it into one calculator-style string so that superficially different but equal forms collapse together, check equivalence by parsing both sides into a symbolic-math object and testing whether their difference simplifies to zero, and grade, which extends equivalence to answers with several parts (a coordinate pair, an interval). It is the deterministic checker figure a verifiable-reward pipeline hands its reward function, and the grader an automated math benchmark hands its scoring loop.

The hard part is not the symbolic math, SymPy does that, it is the extraction and canonicalization: a correct answer can arrive as "14/3", "\dfrac{14}{3}", "\left(\frac{14}{3}\right)", "$14/3$", or "(28)/(6)" (an unreduced but equal fraction), and a checker that does not normalize all of these to a common form will reject correct answers, precisely the false-negative failure mode RLVR warns trains a model against correct behaviour.

Why use it¶

A verifier is the reward. In RLVR, whatever this function returns is the training signal; a checker with false negatives punishes correct reasoning, and a checker with loopholes gets gamed. There is no way to skip building this well.
Deterministic, not a model call. No forward pass, no API, no cost per rollout beyond string parsing and a bounded SymPy simplify; it is reproducible and cheap enough to run per-rollout at RL scale.
The same checker serves eval and training. A MATH-500-style benchmark and an RLVR reward function need the identical extract-canonicalize-compare pipeline; building it once means the number you optimize during RL is the same number you report at eval time.

When to use it (and when not)¶

Use this pattern for any domain where the ground truth is a closed-form mathematical expression, number, fraction, or short tuple: MATH, GSM8K-style word problems, and similar closed-answer benchmarks.
Extend, do not reuse as-is, for domains with different answer shapes: multi-choice letters, symbolic proofs, or free-form units (a physics answer with a unit needs unit-aware comparison this page does not implement).
Prefer a code-execution verifier (RLVR's sandboxed unit-test checker) when the task is programming, not arithmetic.
Do not reach for an LLM judge to do this extraction and comparison. It is a mechanical, deterministic task; a judge model adds cost, latency, and its own failure modes for a job regular expressions and a symbolic-math library do exactly and reproducibly.

Architecture¶

flowchart LR
  OUT["Model output (free text, may contain \\boxed{})"] --> EXT["extract_candidate: brace-depth aware \\boxed{} scan, fallback to last number"]
  EXT --> CAN["canonicalize: strip LaTeX, unify \\frac / \\sqrt / superscripts / percent"]
  CAN --> SPLIT{"Tuple-like? (a, b)"}
  SPLIT -->|"yes"| PARTS["split_parts: compare each element"]
  SPLIT -->|"no"| EQ["symbolic_equal: SymPy parse, diff simplifies to 0?"]
  PARTS --> EQ
  EQ --> GRADE["grade: True only if every part matches"]
  GRADE -->|"reward"| RLVR["RLVR reward function"]
  GRADE -->|"correct/incorrect"| BENCH["MATH-500-style eval"]

How to use it¶

1. Extraction: find the answer inside the free-form response¶

The model's response is prose with the final answer somewhere inside it, conventionally in a \boxed{...} at the end. A regex alone cannot extract this reliably because the box's content can itself contain braces (\boxed{\frac{1}{2}}), so extraction has to track brace depth, not just match to the first }. When no box is present, fall back to the last bare number in the text rather than failing outright.

# 01_extract.py -- runnable (stdlib only). Brace-depth-aware \boxed{} extraction.
import re

RE_NUMERIC_TAIL = re.compile(r"-?(?:\d+/\d+|\d+(?:\.\d+)?(?:[eE][+-]?\d+)?)")


def extract_boxed(text):
    """Return the content of the LAST \\boxed{...} in text, brace-depth aware.
    None if no boxed expression is present or its braces are unbalanced."""
    marker = text.rfind(r"\boxed")
    if marker == -1:
        return None
    i = marker + len(r"\boxed")
    while i < len(text) and text[i].isspace():
        i += 1
    if i >= len(text) or text[i] != "{":
        return None
    i += 1
    depth, start = 1, i
    while i < len(text) and depth > 0:
        if text[i] == "{":
            depth += 1
        elif text[i] == "}":
            depth -= 1
        i += 1
    if depth != 0:
        return None
    return text[start:i - 1]


def extract_candidate(text, fallback="last_number"):
    """Prefer the last \\boxed{}; fall back to the last bare number; else ''."""
    if not text:
        return ""
    boxed = extract_boxed(text.strip())
    if boxed is not None:
        return boxed.strip().strip("$ ")
    if fallback == "last_number":
        found = RE_NUMERIC_TAIL.findall(text)
        if found:
            return found[-1]
    return ""


# --- nested braces inside the box must not truncate extraction early -------
assert extract_boxed(r"blah \boxed{\frac{1}{2}} trailing") == r"\frac{1}{2}"
# --- no box at all -----------------------------------------------------------
assert extract_boxed("no box here") is None
# --- ADVERSARIAL: unbalanced braces (truncated generation) -> None, not a crash
assert extract_boxed(r"\boxed{unterminated") is None
assert extract_candidate(r"\boxed{unterminated") == ""
# --- fallback: no box present, take the last bare number --------------------
assert extract_candidate("the answer is 42, not 41") == "41"
assert extract_candidate("no numbers here", fallback="last_number") == ""

print("01_extract: all asserts passed")

2. Canonicalization: collapse equivalent surface forms¶

Canonicalization rewrites LaTeX into a calculator-readable string: \frac{a}{b} becomes (a)/(b), \sqrt{x} becomes sqrt(x), unicode superscripts (²) become ASCII (**2), thousands separators disappear, and degree markers strip. One case is easy to get wrong and worth calling out on its own: a percent sign. Deleting the % character and nothing else turns "50%" into the bare numeral 50, silently comparing unequal to a ground truth of 1/2, a false negative. The fix is to divide by 100, not just delete the sign.

# 02_canonicalize.py -- runnable (stdlib only).
import re

SUPERSCRIPT_DIGITS = str.maketrans("⁰¹²³⁴⁵⁶⁷⁸⁹⁺⁻",
                                    "0123456789+-")

LATEX_STRIP = [
    (r"\\left|\\right", ""),
    (r"\\,|\\!|\\;|\\:", ""),
    (r"\\cdot", "*"),
    (r"\\deg|\^\\circ|\\circ|°", ""),
    (r"\\dfrac|\\tfrac", r"\\frac"),
]


def canonicalize(answer):
    """Rewrite a raw extracted answer into a calculator-style string so that
    equivalent surface forms ("14/3", "\\frac{14}{3}", "(14)/(3)") compare equal."""
    if not answer:
        return ""
    s = answer.strip()
    s = re.sub(r"^\\text\{(.+?)\}$", r"\1", s)          # unwrap a whole-string \text{...}
    s = re.sub(r"\\\(|\\\)|\\\[|\\\]", "", s)            # strip \( \) \[ \] math wrappers
    for pattern, repl in LATEX_STRIP:
        s = re.sub(pattern, repl, s)
    s = s.translate(SUPERSCRIPT_DIGITS)                  # unicode superscripts -> ASCII
    s = re.sub(r"\\sqrt\{([^}]*)\}", r"sqrt(\1)", s)
    s = re.sub(r"\\frac\{([^{}]+)\}\{([^{}]+)\}", r"(\1)/(\2)", s)
    if "%" in s or "\\%" in s:
        s = s.replace("\\%", "%")
        s = re.sub(r"(-?\d+(?:\.\d+)?)\s*%", r"(\1/100)", s)   # 50% -> (50/100), not bare 50
    s = s.replace("$", "").replace("^", "**")
    s = re.sub(r"(?<=\d),(?=\d\d\d(\D|$))", "", s)       # 1,234 -> 1234
    return s.replace("{", "").replace("}", "").strip().lower()


def canonicalize_naive_percent(s):
    """The common bug: delete '%' without dividing by 100. For comparison only."""
    return s.replace("\\%", "%").replace("%", "").strip().lower()


# --- equivalent LaTeX forms collapse to the same calculator string ---------
assert canonicalize(r"\dfrac{14}{3}") == "(14)/(3)"
assert canonicalize(r"\text{\frac{14}{3}}") == "(14)/(3)"
assert canonicalize("1,234") == "1234"
assert canonicalize(r"90^\circ") == "90"
assert canonicalize(r"2^{3}") == "2**3"

# --- ADVERSARIAL: percent must become a fraction, not a stripped integer ----
naive_50pct = canonicalize_naive_percent("50%")   # -> "50", the common footgun
assert naive_50pct == "50" and naive_50pct != "0.5" and naive_50pct != "(1)/(2)"
assert canonicalize("50%") == "(50/100)"          # this module: converts, does not just strip

print("02_canonicalize: all asserts passed")

3. Symbolic equivalence and multi-part grading¶

String equality is not enough: "14/3" and "(14)/(3)" differ as strings but denote the same rational; "0.5" and "1/2" differ in representation entirely. Parse both sides with SymPy and check that their difference simplifies to zero. Guard the parser with a length cap so a pathological or adversarially long candidate cannot be handed to the symbolic engine, and catch every exception class SymPy's tokenizer and parser can raise on malformed input, never let a garbled model output crash the reward function. Finally, extend equivalence to tuple-like answers ("(14/3, 2/3)") by splitting on commas and comparing element-wise.

# 03_equivalence_and_grading.py -- runnable (needs sympy>=1.14; pip install sympy).
from tokenize import TokenError

from sympy import simplify
from sympy.core.sympify import SympifyError
from sympy.parsing import sympy_parser as spp
from sympy.polys.polyerrors import PolynomialError

MAX_CANDIDATE_LEN = 2000  # refuse to symbolically parse unbounded garbage


def _sympy_parse(expr):
    if expr is None or len(expr) > MAX_CANDIDATE_LEN:
        return None
    try:
        return spp.parse_expr(
            expr,
            transformations=(*spp.standard_transformations, spp.implicit_multiplication_application),
            evaluate=True,
        )
    except (SympifyError, SyntaxError, TypeError, AttributeError, IndexError,
            ValueError, PolynomialError, ZeroDivisionError, TokenError):
        return None


def symbolic_equal(a, b):
    """True if two canonicalized answer strings denote the same value."""
    if a == b:
        return True
    pa, pb = _sympy_parse(a), _sympy_parse(b)
    if pa is None or pb is None:
        return False
    try:
        return simplify(pa - pb) == 0
    except (SympifyError, TypeError):
        return False


def split_parts(text):
    """Split a tuple/list-like "(a, b)" answer into its comma-separated parts."""
    if not text:
        return []
    if len(text) >= 2 and text[0] in "([" and text[-1] in ")]" and "," in text[1:-1]:
        parts = [p.strip() for p in text[1:-1].split(",")]
        if all(parts):
            return parts
    return [text]


def grade(predicted, ground_truth, canon):
    """Grade a predicted answer against the ground truth, part-by-part."""
    if predicted is None or ground_truth is None:
        return False
    gold_parts = split_parts(canon(ground_truth))
    pred_parts = split_parts(canon(predicted))
    if not gold_parts or len(gold_parts) != len(pred_parts):
        return False
    return all(symbolic_equal(g, p) for g, p in zip(gold_parts, pred_parts))


# --- equivalence across representations -------------------------------------
assert symbolic_equal("(14)/(3)", "(14)/(3)") is True         # exact-string fast path
assert symbolic_equal("14/3", "28/6") is True                  # unreduced fraction
assert symbolic_equal("0.5", "(1)/(2)") is True                 # decimal vs fraction
assert symbolic_equal("14/3", "15/3") is False                  # genuinely different values

# --- multi-part (tuple) grading, using the identity canonicalizer here ------
identity = lambda s: s
assert grade("(14/3, 2/3)", "(14/3, 4/6)", identity) is True    # 2/3 == 4/6
assert grade("(2, 1)", "(1, 2)", identity) is False             # order matters
assert grade("(1, 2, 3)", "(1, 2)", identity) is False          # part-count mismatch

# --- ADVERSARIAL: malformed / oversized / hostile input never crashes ------
assert grade(None, "1/2", identity) is False
assert grade("not a number at all", "1/2", identity) is False
assert grade("1/0", "1/2", identity) is False                   # division by zero -> unparsable, not a crash
assert _sympy_parse("x" * 5000) is None                         # length guard stops a parser blow-up

print("03_equivalence_and_grading: all asserts passed")

Wire the two files together (canonicalize from step 2 feeding grade from step 3) and the result plugs directly into an RLVR reward function: reward = 1.0 if grade(extract_candidate(completion), ground_truth, canonicalize) else 0.0, or into an eval loop that reports the mean over a MATH-500-style dataset.

How to develop with it¶

Grow the adversarial suite before trusting the checker. Every canonicalization rule above (percent, superscripts, thousands separators) exists because a real model output broke a simpler version of this function; keep adding held-out cases from actual rollouts, both loopholes (wrong answers that pass) and false negatives (right answers that fail).
Extend canonicalize per domain, do not add ad hoc string hacks in grade. Unit handling, interval notation, and set notation each need their own regex pass in canonicalization, kept separate from the comparison logic so the equivalence check stays a single, auditable function.
Prefer over-rejecting to over-accepting during development, then loosen. A verifier that is too strict undercounts correct answers (visible immediately in a lower-than-expected eval score); one that is too loose silently trains on a broken reward, which is far harder to detect (reward design).

How to run it in production¶

Cache by content hash. grade is a pure function of its two string arguments, so cache results keyed on (canonicalized_pred, canonicalized_gold); re-scored rollouts across epochs become free.
Pin sympy. A SymPy upgrade can change what parse_expr accepts or how simplify normalizes an expression, silently shifting which answers pass; pin the version and re-run the adversarial suite on upgrade, the same discipline RLVR recommends for the verifier as a whole.
Bound every parse. The MAX_CANDIDATE_LEN guard above is not optional at RL scale: an unbounded model can and will occasionally emit pathological output, and a symbolic parser given megabytes of garbage is a real wall-clock and memory risk in a rollout loop processing thousands of completions per step.
This checker does not need a sandbox. Unlike a code verifier, there is no code execution here, only string parsing and symbolic simplification, so it carries none of the untrusted-execution risk RLVR flags for code verifiers.

How to maintain it¶

Keep the adversarial test suite as a real regression gate that runs before every training campaign or eval release, re-add a case the moment a real rollout breaks the checker in either direction, and re-verify the whole suite whenever sympy or the base model changes (a new model's answer-formatting habits are a fresh source of canonicalization gaps).

Failure modes¶

Naive percent stripping. Deleting % instead of dividing by 100 turns a correct "50%" into the numeral 50, a false negative against a ground truth of 1/2; convert to a fraction instead (canonicalization step above).
String equality instead of symbolic equality. "14/3" == "28/6" is False as strings but True mathematically; skipping the SymPy comparison step reproduces this false negative for every unreduced fraction.
Unbounded parser input. Handing an unbounded or adversarial string straight to sympy.parse_expr risks a slow or pathological parse in a loop that must score thousands of rollouts; always length-cap before parsing.
Silent brace mismatch. A truncated generation (hit max_new_tokens mid-\boxed{}) has unbalanced braces; extract_boxed must return None, not a truncated or wrong substring, or the checker grades an incomplete answer as if it were complete.
Tuple order treated as unordered. grade("(2, 1)", "(1, 2)") is correctly False here because position encodes meaning (a coordinate is not a set); do not add order-insensitive comparison without confirming the task's semantics call for it.

References¶

Hendrycks et al., Measuring Mathematical Problem Solving With the MATH Dataset: https://arxiv.org/abs/2103.03874
Lightman et al., Let's Verify Step by Step (introduces the MATH-500 test subset): https://arxiv.org/abs/2305.20050
Math-Verify (HuggingFace, symbolic math-answer verification library): https://github.com/huggingface/Math-Verify
SymPy (the symbolic-math engine used for equivalence checking): https://sympy.org
Raschka, Build a Reasoning Model (From Scratch), Manning Publications, 2026 (Ch. 3, the extract-normalize-verify-grade pipeline this page implements independently).