Markdown

Post-training system map: objectives, distributed RL, rollouts, and quantization¶

Scope: reference and decision map for separating LLM learning objectives, reward sources, policy optimizers, parameter-update strategies, trainer parallelism, rollout topology, synchronization, numerical representations, and experiment control, then placing them in one system from a base checkpoint through post-training, evaluation, quantization, and serving. Implementation details live in the focused pages linked below.

The terms in this map are not peers. SFT, DPO, and RL describe objectives; RLVR describes where an RL reward comes from; GRPO describes how an RL policy is updated; LoRA and QLoRA describe which parameters are trainable and how the frozen base is stored; FSDP describes how training state is distributed; rollout and synchronization choices describe how online experience reaches that trainer; quantization describes the numerical representation of weights, activations, or the KV cache. A real pipeline selects one coordinate from several of these axes at the same time.

This is a reference page. It defines boundaries, dependencies, and routing. The focused technology pages carry runnable examples, production templates, and adversarial validation.

Focused pages¶

Question	Focused page
Which post-training method fits the available signal?	Fine-tuning and post-training
How do demonstrations, LoRA, and QLoRA work?	SFT and LoRA/QLoRA
How does group-relative policy optimization work?	GRPO and GRPO variants
What makes a reward verifiable?	RLVR and reward design
When should preference pairs replace online RL?	DPO
How is model and optimizer state sharded?	FSDP, DeepSpeed ZeRO, tensor parallelism, and pipeline parallelism
How should trainer and rollout workers be placed?	Asynchronous RL systems, RL library selection, and rollout fleet sizing
How are repeated prompts and long rollouts made cheaper?	RL rollout redundancy and rejection sampling
How do training weights reach the rollout engine?	Delta weight synchronization
How do prefill and decode separation fit inside a rollout service?	Disaggregated inference
How can experiments be proposed and governed automatically?	Autonomous experimentation loops
Which low-bit format or conversion method fits serving?	Quantization for inference
How are adapters deployed without merging each one?	Multi-LoRA serving
How are checkpoints evaluated and promoted?	LLM evaluation harness and experiment tracking

One system, several orthogonal decisions¶

Each term answers a different engineering question.

Decision axis	Question	Main choices	Output
Learning objective	What signal should change model behaviour?	SFT, DPO or another preference objective, online RL	Loss values and gradients
RL reward source	What makes a sampled completion better?	Learned reward model, judge, environment result, deterministic verifier	Scalar reward per completion
RL policy optimizer	How should rewards update the policy?	PPO, GRPO, DAPO, other policy-gradient variants	Updated policy parameters
Parameter update	Which parameters may change?	Full fine-tuning, LoRA, QLoRA	Full checkpoint or adapter
Trainer parallelism	How is training state and compute distributed?	FSDP2, ZeRO-3, tensor, pipeline, context, or expert parallelism	Sharded training job and checkpoint
Rollout policy	How is experience sampled?	Group sampling, Best-of-N, rejection sampling, multi-turn environment trajectories	Tokens, masks, log probabilities, and rewards
Placement and synchronization	Where do trainer and rollout workers run, and how fresh must actors be?	Colocated, synchronous disaggregated, bounded-asynchronous	Full or delta weight transfers and a staleness contract
Numerical representation	How are tensors stored and computed?	BF16, FP8, INT8, INT4, NF4, FP4	Runtime-specific checkpoint and kernels

| Experiment control | Who proposes, evaluates, stops, and records trials? | Human loop, scheduler, autonomous controller | Reproducible trial ledger and promoted candidate | The resulting configurations are combinations, not competing names:

SFT + LoRA + BF16 compute trains an adapter from demonstrations.
SFT + QLoRA trains an adapter while the frozen base is stored in 4-bit NF4.
RLVR + GRPO + LoRA uses a verifier for rewards, GRPO for the policy update, and adapters for trainable parameters.
DPO + FSDP2 + LoRA optimizes offline chosen/rejected pairs while sharding trainer state, with no online rollout fleet.
full-parameter GRPO + FSDP2 + disaggregated FP8 rollouts updates the full policy while a separate lower-precision inference fleet generates training samples.
merged checkpoint + AWQ INT4 describes a deployment artifact, not a training objective.
autoresearch-rl + inner GRPO trial describes an outer experiment controller wrapped around an ordinary post-training system.

Concept dependency map¶

The RL branch has two independent selections: a reward source and a policy optimizer. RLVR belongs to the reward-source branch. GRPO belongs to the optimizer branch. LoRA and numerical precision cut across both SFT and RL.

flowchart TD
    BASE["Base or instruction checkpoint"] --> SIGNAL{"Available learning signal"}
    SIGNAL -->|"demonstrations"| SFT["SFT objective"]
    SIGNAL -->|"chosen and rejected pairs"| DPO["Preference objective: DPO"]
    SIGNAL -->|"scalar outcome"| RL["RL objective"]

    RL --> RSRC{"Reward source"}
    RSRC --> RM["Learned reward model or judge"]
    RSRC --> VER["Deterministic verifier or environment"]
    VER --> RLVR["RLVR configuration"]

    RL --> OPT{"Policy optimizer"}
    OPT --> PPO["PPO"]
    OPT --> GRPO["GRPO and variants"]
    GRPO -.->|"common pairing"| RLVR

    PARAM["Parameter update: full, LoRA, or QLoRA"] -.-> SFT
    PARAM -.-> DPO
    PARAM -.-> RL
    NUM["Numerics: BF16, FP8, INT8, INT4, NF4, or FP4"] -.-> PARAM
    NUM -.-> SERVE["Rollout and serving engines"]

This map prevents four common category errors:

RLVR is not an optimizer. PPO or GRPO can optimize a policy under a verifiable reward.
GRPO does not require a verifier. It consumes scalar rewards; a verifier is a common, strong source of those rewards.
LoRA is not a learning objective. The adapter receives gradients from SFT, DPO, GRPO, or another differentiable loss.
INT4 is not a quantization algorithm. INT4 names a representation; GPTQ and AWQ are methods that can produce weight-only INT4 checkpoints.

High-level theory¶

SFT: imitate demonstrated tokens¶

SFT minimizes next-token cross-entropy over curated prompt-response demonstrations: L_SFT = -sum_t log p_theta(y_t | x, y_<t). Every target token supplies a training signal. SFT is suited to response format, tool syntax, domain terminology, and known solution patterns. It is a common cold start before preference optimization or RL, but it is not a universal prerequisite. The data contract, loss masking, packing, and adapter implementation are covered in SFT and LoRA/QLoRA and chat rendering and loss masking.

DPO: optimize offline preference pairs¶

DPO consumes a prompt with a chosen and rejected response. Its loss increases the policy's chosen-versus-rejected log-probability margin relative to a fixed reference policy. The preference pair supplies the supervision directly, so the training loop does not contain an online rollout engine, reward service, or actor-to-learner weight synchronization. Responses may have been sampled upstream to construct the dataset, but that data-generation job is not part of each DPO optimizer step.

DPO has an SFT-like distributed systems shape: read a fixed dataset, run policy and reference forwards, backpropagate, and checkpoint. FSDP or ZeRO can shard a full-parameter run; LoRA can reduce trainable state; reference log probabilities can be cached or computed by disabling the active adapter when the framework supports that contract. See DPO for the objective and validation details.

RL: optimize sampled outcomes¶

RL maximizes expected sequence-level reward: J(theta) = E_y~pi_theta[R(x, y)]. The policy generates completions, the system scores them, and the optimizer increases the probability of rewarded behaviour. The signal is usually sparser and more delayed than SFT because a long completion may receive only one final scalar. Online LLM RL therefore adds rollout generation, reward execution, policy updates, and repeated weight synchronization to the training system.

RL is justified when demonstrations no longer provide the needed signal and an outcome can be scored. It is a poor default when the reward is noisy, cheap to exploit, or disconnected from the held-out evaluation. See reward design before selecting an RL library.

RLVR: compute correctness instead of predicting preference¶

RLVR uses a verifier such as an answer-equivalence checker, compiler and unit-test suite, schema validator, proof checker, or environment success condition. In compact form, R(x, y) = verifier(y, ground_truth, environment). This removes the need for a learned reward model where correctness is computable, but it does not remove reward-design risk. Weak tests, leaked answers, permissive parsers, nondeterministic environments, and unsafe code execution all create false rewards. The verifier must be versioned, sandboxed, tested on corrupt inputs, and evaluated against adversarial attempts; see RLVR, the math verifier cookbook, and reward-function sandboxing.

GRPO: compare completions within a prompt¶

GRPO samples a group of completions for each prompt and derives an advantage from their relative rewards. A simplified form is A_i = r_i - mean(r_group); implementations may add scaling, clipping, KL control, or successor fixes. Above-average completions are reinforced and below-average completions are suppressed without training a separate value model. If every completion in a group receives the same reward, every relative advantage is zero and that prompt teaches nothing. Prompt difficulty, group diversity, verifier discrimination, entropy, and the fraction of zero-variance groups are therefore system-level concerns, not minor trainer settings. The core update and current variants are separated in GRPO, GRPO variants, and the training health runbook.

LoRA and QLoRA: constrain the trainable state¶

LoRA freezes a base matrix W_0 and trains a low-rank delta: W' = W_0 + (alpha/r)BA. It reduces trainable parameters, optimizer state, checkpoint size, and per-task storage. The full base still has to be loaded, and activations still consume memory. A rank or target-module choice that is too restrictive can underfit a large distribution shift.

QLoRA adds a storage choice: the frozen base is held in 4-bit NF4 and dequantized for higher-precision computation while LoRA adapters remain trainable. It is a training-memory strategy. It does not imply that the final serving artifact should use NF4, and it should not be conflated with GPTQ or AWQ deployment conversion. Framework and kernel support must be checked before combining QLoRA with a specific RL trainer.

FSDP and the trainer parallelism layer¶

FSDP is neither an objective nor an RL algorithm. PyTorch FSDP2 fully_shard distributes parameters, gradients, and optimizer state across a data-parallel group. It all-gathers parameters before a module computes, reduce-scatters gradients after backward, and releases transient full parameters. ZeRO-3 solves a similar state-sharding problem in DeepSpeed. Tensor and pipeline parallelism split model compute differently and can compose with sharded data parallelism when one replica does not fit or compute scaling requires another axis.

Trainer pressure	First architecture question	Typical mechanism
Optimizer, gradient, and parameter state exceeds one GPU	Can state be sharded across replicas?	FSDP2 or ZeRO-3
One layer's matrix multiplication does not fit or scale on one GPU	Must each layer span devices?	Tensor parallelism
Model depth or interconnect layout favors staged execution	Can layers be partitioned into stages?	Pipeline parallelism
Very long sequences dominate activation memory	Can the sequence dimension be split?	Context or sequence parallelism
Mixture-of-experts routing dominates placement	Can experts be partitioned?	Expert parallelism

The same FSDP trainer can execute SFT, DPO, PPO, or GRPO. Online RL adds an external rollout and synchronization problem around it. The focused implementation and checkpoint contracts live in FSDP.

Quantization: one system, several precision planes¶

Quantization maps a real tensor to low-bit values plus scales, approximately x_hat = scale * (q - zero_point). The design has several independent choices: which tensors are quantized, the format, scale granularity, calibration method, and target kernel. It can reduce HBM capacity and bandwidth requirements; a speedup requires compatible kernels and a workload that uses them efficiently.

An RL design must name the precision of each plane rather than assign one precision to the whole model:

Precision plane	Typical choices	Purpose	Critical boundary
Canonical trainer weights, gradients, and optimizer state	BF16 weights and gradients with FP32 optimizer state; specialized FP8 training paths	Preserve stable optimization while controlling memory	This state defines the policy being optimized
Frozen base during adapter training	NF4 base with BF16/FP16 adapter compute in QLoRA	Reduce frozen-base memory	NF4 is a QLoRA storage format, not automatically the serving format
Rollout actor weights and activations	BF16, FP8, INT8, or NVFP4/MXFP4 on supported hardware and runtimes	Increase generation throughput and actor capacity	The sampled behaviour policy can differ numerically from the trainer policy
Rollout KV cache	BF16, FP8, INT8, or NVFP4 where the engine supports it	Reduce per-sequence memory and admit more concurrent rollouts	It can change generated trajectories without changing checkpoint weights
Trainer-to-actor synchronization payload	Full canonical weights, canonical deltas, or a verified format-aware payload	Refresh actor replicas	Packed layouts and shared quantization scales can make a sparse canonical delta dense or invalid at runtime
Final serving artifact	GPTQ/AWQ INT4, SmoothQuant INT8, FP8, NVFP4/MXFP4, quantized KV cache	Reduce replica size, bandwidth, or compute cost	Re-run evaluation on the exact converted artifact, kernels, and runtime

Method or format family	What becomes low precision	Role in an RL system	Update-cadence boundary
NF4 with QLoRA	Frozen base weights; adapters and compute remain higher precision	Reduce learner memory for SFT, DPO, or supported RL adapter training	No repeated actor conversion is implied; serving format remains a separate choice
FP8 or INT8 actor	Actor weights and often activations; optionally FP8/INT8 KV	Reduce rollout memory and accelerate supported matrix multiplications	Requantize and reload after policy updates; preserve actor log probabilities and correct mismatch
NVFP4 or MXFP4 actor	W4A4 actor paths and, separately, an optional NVFP4 KV cache	Very-low-bit rollout on supported Blackwell paths	Hardware, kernels, scale recalibration, policy mismatch, and update visibility must all be verified
GPTQ or AWQ INT4	Mostly final weight-only serving weights	Compress a promoted policy after RL	Static calibration and packing make per-step actor refresh unattractive
SmoothQuant INT8	Weights and activations through calibrated smoothing	Produce a W8A8 serving artifact	Calibration belongs after candidate promotion unless the entire actor refresh path is engineered for it
Quantization-aware training	Forward numerics simulate or use the target low-bit path while optimization adapts the weights	Train robustness to a planned deployment format	It changes the training method and cost; it is not equivalent to using a quantized rollout actor

Algorithm-aware systems sit above these formats. QeRL combines NVFP4 rollout with LoRA and an adaptive quantization-noise mechanism intended to control exploration. QuRL and FP8-RL address other actor mismatch and update-visibility mechanisms. Treat each correction as part of its reported algorithm, not as a universal flag attached to FP8 or FP4.

Static weight-only methods such as GPTQ and AWQ are generally final-artifact conversions, not cheap per-step actor refreshes. An online actor changes repeatedly, so conversion latency, packed layout, scales, and runtime load time are part of the update cadence. FP8 or another runtime-native block format is often easier to refresh, but that is a property to measure for the exact engine, not a format guarantee.

A quantized actor creates two correctness problems. First, behaviour log probabilities from the actor can differ from log probabilities recomputed by the higher-precision trainer. Preserve exact token IDs, masks, sampling metadata, and actor log probabilities, then use the algorithm's validated importance correction or truncated importance sampling path. Second, small trainer updates can disappear when rounded into a low-bit actor. Monitor actor-versus-trainer KL, probability ratios, reward distribution, entropy, and the fraction of updates that change quantized codes. QuRL and FP8-RL study these mismatch mechanisms; their corrections are not interchangeable defaults for every trainer.

Delta synchronization needs its own format contract. Subtracting packed low-bit tensors is not equivalent to subtracting their canonical weights, and shared scale changes can alter an entire block. The conservative path is to synchronize canonical weights and let the actor engine quantize them. A delta path should be enabled only when the canonical and runtime representations are close enough, or when a format-aware implementation has been verified. See delta weight synchronization.

Quantization for inference provides the detailed NF4, GPTQ, AWQ, SmoothQuant, FP8, FP4, QAT, and KV-cache matrix. NVFP4 across the lifecycle separates NVFP4 from MXFP4 and covers the hardware boundary. Tensor cores and mixed precision covers hardware execution, while KV-cache management covers the separately configured per-request state.

Architecture: end-to-end system workflow¶

SFT is normally one training workload. Online RL is at least two coupled workloads: a rollout inference engine and a trainer. RLVR adds verifier execution, which may itself require isolated CPU workers, test sandboxes, external environments, or proof systems.

flowchart TD
    DATA["Versioned demonstrations, prompts, answers, and held-out evals"]
    BASE["Pinned base model, tokenizer, and chat template"] --> GATE0{"Behaviour already passes the gate?"}
    DATA -.-> GATE0
    DATA -.-> SFT
    GATE0 -->|"no, demonstrations available"| SFT["SFT trainer"]
    UPDATE["Full weights, LoRA, or QLoRA"] -.-> SFT
    SFT --> SFTC["SFT checkpoint or adapter"]
    GATE0 -->|"yes"| POLICY["Candidate policy"]
    SFTC --> POLICY

    POLICY --> RLGATE{"Computable reward and more capability needed?"}
    RLGATE -->|"yes"| ROLLOUT["Rollout engine: generate G completions"]
    DATA -.-> ROLLOUT
    DATA -.-> SCORE
    ROLLOUT --> SCORE["Reward model, judge, verifier, or environment"]
    SCORE --> TRAIN["GRPO, PPO, or another RL trainer"]
    TRAIN -->|"weight or adapter sync"| ROLLOUT
    TRAIN --> RLCKPT["RL checkpoint or adapter"]
    RLGATE -->|"no"| EVAL["Held-out evaluation gate"]
    RLCKPT --> EVAL
    DATA -.-> EVAL

    EVAL -->|"pass"| REG["Registry: model, data, config, reward, lineage"]
    EVAL -->|"fail"| DIAG["Diagnose data, reward, optimizer, or capacity"]
    DIAG --> POLICY
    REG --> ART{"Serving artifact strategy"}
    ART --> KEEP["Keep adapter for multi-LoRA"]
    ART --> MERGE["Merge into a standalone checkpoint"]
    KEEP --> QUANT["Supported serving quantization"]
    MERGE --> QUANT
    QUANT --> QEVAL["Post-conversion quality and performance gate"]
    DATA -.-> QEVAL
    QEVAL --> SERVE["Inference engine and production monitoring"]

The loop has four planes with different ownership and failure modes:

Data plane: demonstrations for SFT; prompts, ground truth, and environment state for RLVR; separate held-out evaluations for promotion.
Training plane: forward/backward workers, optimizer state, sharding, checkpoints, and the selected full/LoRA/QLoRA parameterization.
Rollout and reward plane: inference workers, sampling configuration, reward execution, verifier sandboxes, and weight synchronization. This plane exists only for online methods.
Serving plane: merged or unmerged adapters, quantized artifacts, inference kernels, KV-cache policy, request routing, and production SLOs.

Distributed online RL topology¶

Online RL couples a learner plane to an actor plane. The learner may use FSDP, ZeRO, or Megatron-style parallelism. The actor plane is an inference service, commonly built with vLLM or SGLang, that uses tensor or data parallel replicas, continuous batching, and KV-cache management. A controller such as Ray, KubeRay, a framework-native scheduler, or a Slurm-launched coordinator assigns prompts, collects trajectories, invokes rewards, and advances policy versions. The RL library map compares concrete stacks.

flowchart LR
    CTRL["Controller and policy-version ledger"] --> PROMPTS["Prompt queue"]

    subgraph ACTOR["Rollout actor plane"]
        PROMPTS --> ROUTER["Rollout router"]
        ROUTER --> PREFILL["Optional prefill pool"]
        PREFILL -->|"KV transfer"| DECODE["Decode pool"]
        DECODE --> TRAJ["Tokens, masks, and behaviour log probabilities"]
    end

    TRAJ --> REWARD["Reward model, verifier, or environment"]
    REWARD --> QUEUE["Scored trajectory queue"]

    subgraph LEARNER["Learner plane"]
        QUEUE --> TRAIN["FSDP, ZeRO, or Megatron trainer"]
        TRAIN --> CKPT["Canonical checkpoint or adapter"]
    end

    TRAIN --> SYNC["Full or delta weight synchronization"]
    SYNC --> ROUTER
    CTRL -.-> ROUTER
    CTRL -.-> TRAIN

This is the disaggregated case. In a colocated system, the same GPU set alternates between rollout and learner phases and sees updated weights locally. Reward and environment workers may remain separate CPU or GPU services in either topology.

Architecture	Placement and coordination	Best fit	Main cost or risk
Colocated synchronous	One GPU pool alternates inference and training; each step is a barrier	Small clusters, large models that need all GPUs in both phases, strict on-policy updates	Phase switching, cache teardown, and idle capacity when generation and training rates differ
Dedicated synchronous actor/learner	Fixed actor and learner pools exchange trajectories and weights every iteration	Predictable workloads, heterogeneous actor/trainer hardware, isolated inference kernels	The faster pool waits at the iteration barrier; full synchronization can dominate
Bounded-asynchronous actor/learner	Actors continue from versioned policies; a queue feeds the learner; weights refresh every bounded interval	Large fleets and variable-length or environment-heavy rollouts	Stale off-policy data, queue growth, harder checkpoint recovery, and mandatory importance correction
Hybrid or elastic	Some roles colocate while actor, reward, or environment pools scale independently	Bursty multi-turn workloads and shared clusters	Scheduler complexity, resource fragmentation, and more failure domains

A synchronous topology is not automatically on-policy if the actor uses stale, quantized, or differently configured weights. An asynchronous topology is not automatically invalid if policy versions are bounded, actor log probabilities are preserved, and the optimizer's off-policy correction has been validated. The asynchronous RL systems page covers staleness budgets and correction boundaries.

Two meanings of disaggregated inference¶

Two distinct boundaries are often called disaggregation:

Boundary	Separated components	Data crossing the boundary	Scaling reason
RL system disaggregation	Rollout actors and policy learners	Trajectories toward the learner; weights or adapters toward actors	Generation and training have different kernels, memory shapes, and service times
Inference-engine disaggregation	Prefill and decode pools inside an actor or serving service	KV-cache blocks plus request metadata	Prefill is compute-heavy; decode is memory-bandwidth and KV-capacity sensitive

The two boundaries can nest. A disaggregated RL actor fleet may itself route each request through separate prefill and decode pools. Actor/learner disaggregation does not imply prefill/decode separation, and prefill/decode separation says nothing about whether an RL learner exists. The inner inference topology is covered in disaggregated inference.

Rollout strategies and controls¶

A rollout strategy determines the training distribution, not only throughput. Record the prompt revision, policy version, tokenizer and template, temperature, top-p or top-k, seed policy, maximum response length, stop conditions, tool/environment version, token IDs, loss mask, actor log probabilities, and reward components for every trajectory batch.

Strategy	Produced experience	Use when	System consequence
Group sampling	`G` completions for the same prompt	GRPO or another within-prompt relative objective needs reward variation	Prompt tokens repeat across the group; zero-variance groups waste learner work
Best-of-N or rejection sampling	Many candidates, then retained winners or filtered pairs	Building SFT/DPO data or iteratively improving from a scorer	Scoring and rejected tokens still consume compute; selection can narrow diversity
Multi-turn environment rollout	Messages, tool calls, observations, and terminal outcome	Agentic tasks need state transitions or delayed success	Environment latency, nondeterminism, timeouts, and partial trajectories dominate scheduling
Bounded replay or asynchronous queue	Versioned trajectories from slightly older policies	Actor and learner rates cannot remain locked step by step	Staleness limits and importance correction become part of optimizer correctness

Four controls determine whether the rollout fleet feeds useful work:

Diversity and difficulty: tune sampling and prompt selection so GRPO groups contain meaningful reward variance without drifting into invalid text. Dynamic prompt sampling may skip repeatedly all-pass or all-fail prompts, but its selection bias must be measured.
Redundancy: a group repeats the same prompt tokens. Deduplicate prompt activations in the trainer and use prefix-aware or cascade-attention paths in the actor when supported. Validate outputs and gradients against the repeated-prompt reference path; see RL rollout redundancy.
Stragglers: response length, tool latency, and environment duration create long tails. Use continuous batching, length-aware buckets, timeouts, and partial-batch policies without silently changing which samples reach the learner.
Rate matching: actor capacity must cover approximately prompts_per_update * G * average_generated_tokens within the target learner window. Measure tokens per second on the real sampling distribution, then size and rebalance the actor and learner pools with the rollout fleet sizing runbook.

The rollout record must reach the learner without re-tokenization. A reconstructed chat string can change token boundaries, stop-token handling, or masks and thereby invalidate behaviour-policy ratios even when the visible text is identical.

Outer experiment control with autoresearch-rl¶

autoresearch-rl sits outside the learning system mapped above. It proposes and evaluates experiments; it is not a replacement for SFT, DPO, GRPO, FSDP, or a rollout engine. An inner trial can launch any of those components behind a command, HTTP, or ephemeral remote target. The outer controller decides which code or parameters to try next, while the inner trainer decides how model weights change.

flowchart LR
    SPEC["Fixed objective, budget, and evaluation contract"] --> PROPOSE["Proposal policy"]
    PROPOSE --> CAND["Configuration or code candidate"]
    CAND --> TARGET["Inner target: SFT, DPO, or distributed RL trial"]
    TARGET --> EVAL["Fixed evaluation and progress events"]
    EVAL --> DECIDE{"Keep, reject, or stop"}
    DECIDE --> LEDGER["Trial ledger and checkpoint"]
    LEDGER --> PROPOSE
    DECIDE -->|"promote"| MODEL["Candidate model artifact"]
    TRUST["Frozen preparation and evaluator trust boundary"] -.-> TARGET
    TRUST -.-> EVAL

At the source revision checked for this page, the controller supports grid, random, LLM, LLM-diff, hybrid, and learned proposal policies; serial or parallel trials; resource pools; stop guards; checkpoint and resume; cooperative progress events; and command, HTTP, or Basilica targets. The learned proposal policy uses PPO to select experiments in the outer loop. That PPO policy is separate from an inner language-model policy that might use GRPO. The repository's Basilica GRPO example makes the nesting concrete: the controller edits or configures an inner Qwen2.5-0.5B GSM8K GRPO trial and judges the resulting experiment metric.

Layer	State it owns	Contract that must remain fixed
Outer autoresearch loop	Proposal policy, candidate diff/config, budget, trial ledger, promotion decision	Objective, evaluation function, immutable preparation inputs, stop and resource limits
Inner post-training trial	Dataset batches, model policy, optimizer, FSDP or other parallelism, rollout fleet, rewards, checkpoints	Candidate-specific config plus the fixed evaluator interface

The outer loop magnifies experimental bias if it can edit the evaluator, leak held-out examples, reuse non-comparable metrics, or exceed the declared resource budget. Keep preparation and evaluation outside the mutable candidate surface, sandbox trial execution, version every candidate and result, and replay the winning trial independently before promotion. These controls align with autonomous experimentation loops.

This description was verified against the source repository at commit c2bb32b154d382d3d9be05a312d28f04c3af40ff, reached through the public autoresearch-rl package metadata. The package metadata reports release 0.4.0; untagged source after that release should not be assumed to have the same packaging or compatibility contract.

Validated architecture planner and rollout rate model¶

The following standard-library-only example encodes conservative system invariants and a lower-bound rollout rate calculation. It distinguishes offline DPO from online RL, requires grouped samples for GRPO, rejects disaggregated actors without synchronization, and rejects quantized actors without an explicit correction contract. It also exercises boundary and failure cases. It is an executable design check, not a framework configuration generator.

from dataclasses import dataclass, replace
from math import ceil


@dataclass(frozen=True)
class Plan:
    name: str
    stage: str
    trainer: str
    parameterization: str
    rollout: str
    reward: str
    optimizer: str
    group_size: int
    sync: str
    train_precision: str
    rollout_precision: str
    correction: str
    deployment_quant: str
    outer_controller: str


def validate(plan: Plan) -> None:
    if plan.stage not in {"sft", "dpo", "online_rl"}:
        raise ValueError("stage must be sft, dpo, or online_rl")
    if plan.trainer not in {"single", "fsdp2", "zero3", "megatron"}:
        raise ValueError("unsupported trainer backend")
    if plan.parameterization not in {"full", "lora", "qlora"}:
        raise ValueError("unsupported parameterization")

    if plan.stage == "sft":
        if (plan.reward, plan.optimizer, plan.rollout, plan.sync) != (
            "none", "none", "none", "none"
        ):
            raise ValueError("SFT has no reward, RL optimizer, rollout, or weight sync")

    if plan.stage == "dpo":
        if plan.reward != "preferences":
            raise ValueError("DPO needs chosen/rejected preferences")
        if plan.rollout != "none" or plan.sync != "none":
            raise ValueError("DPO is offline: rollout and weight sync must be none")
        if plan.optimizer != "none":
            raise ValueError("DPO uses its preference loss, not an online RL optimizer")

    if plan.stage == "online_rl":
        if plan.reward not in {"verifier", "learned", "environment"}:
            raise ValueError("online RL needs verifier, learned, or environment reward")
        if plan.optimizer not in {"grpo", "ppo"}:
            raise ValueError("online RL needs GRPO or PPO in this reference planner")
        if plan.optimizer == "grpo" and plan.group_size < 2:
            raise ValueError("GRPO needs group_size >= 2")
        if plan.rollout not in {"colocated", "disaggregated"}:
            raise ValueError("online RL needs colocated or disaggregated rollouts")
        if plan.rollout == "colocated" and plan.sync != "local":
            raise ValueError("colocated RL uses local phase-to-phase weight visibility")
        if plan.rollout == "disaggregated" and plan.sync not in {"full", "delta"}:
            raise ValueError("disaggregated RL needs full or delta weight sync")
        quantized_actor = (
            plan.train_precision == "bf16"
            and plan.rollout_precision in {"fp8", "int8"}
        )
        if quantized_actor and plan.correction not in {"importance_sampling", "tis"}:
            raise ValueError(
                "BF16 trainer with FP8/INT8 rollouts needs an importance correction"
            )
        if plan.sync == "delta" and plan.rollout_precision == "int8":
            raise ValueError(
                "INT8 rollout delta needs a verified format-aware synchronization path"
            )

    if plan.stage != "online_rl" and plan.rollout_precision != "none":
        raise ValueError("offline stages do not have rollout precision")


def rollout_gpu_floor(
    prompts: int,
    group_size: int,
    average_tokens: int,
    tokens_per_second_per_gpu: int,
    rollout_window_seconds: int,
) -> int:
    values = (
        prompts,
        group_size,
        average_tokens,
        tokens_per_second_per_gpu,
        rollout_window_seconds,
    )
    if any(value <= 0 for value in values):
        raise ValueError("rate-model inputs must all be positive")
    demand = prompts * group_size * average_tokens
    capacity = tokens_per_second_per_gpu * rollout_window_seconds
    return ceil(demand / capacity)


def must_reject(name: str, plan: Plan, expected: str) -> None:
    try:
        validate(plan)
    except ValueError as error:
        assert str(error) == expected
        print(f"REJECT {name}: {error}")
    else:
        raise AssertionError(f"{name} was accepted")


sft = Plan(
    "sft-qlora", "sft", "single", "qlora", "none", "none", "none", 0,
    "none", "bf16", "none", "none", "awq_int4", "none"
)
dpo = Plan(
    "dpo-offline", "dpo", "fsdp2", "lora", "none", "preferences", "none", 0,
    "none", "bf16", "none", "none", "awq_int4", "none"
)
grpo = Plan(
    "grpo-rlvr", "online_rl", "fsdp2", "lora", "disaggregated", "verifier",
    "grpo", 8, "full", "bf16", "fp8", "tis", "awq_int4", "autoresearch-rl"
)

for plan in (sft, dpo, grpo):
    validate(plan)

print("VALID dpo-offline: trainer=fsdp2 rollout=none reward=preferences sync=none")
print(
    "VALID grpo-rlvr: trainer=fsdp2 rollout=disaggregated/fp8 "
    "reward=verifier sync=full correction=tis outer=autoresearch-rl"
)
required = rollout_gpu_floor(64, grpo.group_size, 2048, 10_000, 20)
assert required == (64 * 8 * 2048 + 10_000 * 20 - 1) // (10_000 * 20)
assert required == 6
assert rollout_gpu_floor(64, 16, 2048, 10_000, 20) >= required
print(
    "RATE grpo-rlvr: 64 prompts x 8 x 2048 tokens needs >= "
    f"{required} rollout GPUs"
)

must_reject(
    "grpo-group-one", replace(grpo, group_size=1), "GRPO needs group_size >= 2"
)
must_reject(
    "dpo-rollout",
    replace(dpo, rollout="disaggregated", sync="full", rollout_precision="bf16"),
    "DPO is offline: rollout and weight sync must be none",
)
must_reject(
    "rlvr-without-verifier",
    replace(grpo, reward="none"),
    "online RL needs verifier, learned, or environment reward",
)
must_reject(
    "disaggregated-without-sync",
    replace(grpo, sync="none"),
    "disaggregated RL needs full or delta weight sync",
)
must_reject(
    "quantized-without-correction",
    replace(grpo, correction="none"),
    "BF16 trainer with FP8/INT8 rollouts needs an importance correction",
)
must_reject(
    "int8-delta",
    replace(grpo, rollout_precision="int8", sync="delta"),
    "INT8 rollout delta needs a verified format-aware synchronization path",
)
try:
    rollout_gpu_floor(64, 0, 2048, 10_000, 20)
except ValueError as error:
    assert str(error) == "rate-model inputs must all be positive"
else:
    raise AssertionError("zero group size was accepted")

print("ALL ASSERTS PASSED")

Executed output:

VALID dpo-offline: trainer=fsdp2 rollout=none reward=preferences sync=none
VALID grpo-rlvr: trainer=fsdp2 rollout=disaggregated/fp8 reward=verifier sync=full correction=tis outer=autoresearch-rl
RATE grpo-rlvr: 64 prompts x 8 x 2048 tokens needs >= 6 rollout GPUs
REJECT grpo-group-one: GRPO needs group_size >= 2
REJECT dpo-rollout: DPO is offline: rollout and weight sync must be none
REJECT rlvr-without-verifier: online RL needs verifier, learned, or environment reward
REJECT disaggregated-without-sync: disaggregated RL needs full or delta weight sync
REJECT quantized-without-correction: BF16 trainer with FP8/INT8 rollouts needs an importance correction
REJECT int8-delta: INT8 rollout delta needs a verified format-aware synchronization path
ALL ASSERTS PASSED

System dependencies and artifacts¶

Configuration	Additional dependencies	Persistent artifact	Does not require
SFT with full weights	Demonstration dataset, tokenizer/template, trainer, optimizer, checkpoint storage	Full checkpoint	Rollout engine or reward service
SFT with LoRA	SFT dependencies plus adapter injection and target-module config	Base reference plus adapter	Full optimizer state for frozen weights
SFT with QLoRA	LoRA dependencies plus 4-bit loading, dequantization kernels, and compatible hardware/runtime	Base reference, quantization config, and adapter	A fully materialized BF16 base in GPU memory
DPO with FSDP2 or ZeRO	Chosen/rejected dataset, fixed reference policy or cached reference log probabilities, sharded trainer	Full checkpoint or adapter plus dataset and reference lineage	Online rollout engine, reward service, or actor weight sync
FSDP trainer backend	Distributed process group, sharding plan, mixed-precision policy, distributed checkpoint contract	Sharded optimizer and model state or consolidated export	Any particular objective; SFT, DPO, PPO, and GRPO can share it
GRPO with learned reward	Prompt set, rollout engine, scorer, trainer, weight sync, sampling config, optional reference policy	Policy checkpoint/adapter, reward version, rollout and metric lineage	Separate value model in the original critic-free formulation
GRPO with RLVR	GRPO dependencies plus ground truth, deterministic verifier, sandbox, timeouts, and corruption tests	Policy plus exact verifier and environment version	Learned reward model when the verifier fully supplies the reward
Synchronous disaggregated RL	Dedicated actor and learner pools, trajectory transport, per-step policy version, weight synchronization	Policy plus trajectory, reward, topology, and synchronization lineage	Phase switching on one GPU pool
Bounded-asynchronous RL	Synchronous dependencies plus queues, staleness bound, importance correction, recovery protocol	Versioned trajectories and actor/learner checkpoints	A global barrier after every rollout batch
Quantized RL actors	Supported low-bit inference kernels, repeated conversion/refresh path, actor log probabilities, mismatch correction	Canonical policy plus actor format, scales, engine version, and correction config	A quantized trainer or quantized final artifact
Post-training quantization	Final candidate checkpoint, representative calibration data where required, conversion tool, supported kernels, exact runtime	Quantized checkpoint and scale metadata	New behaviour data unless QAT is selected

Every promoted model should preserve the base revision, tokenizer and chat template, dataset revisions, objective, adapter configuration, trainer topology, reward or verifier revision, rollout sampling settings, actor policy version, synchronization mode, precision plane configuration, optimizer configuration, checkpoint lineage, conversion method, runtime version, and held-out evaluation result. Without that chain, a regression cannot be assigned to learning, actor staleness, synchronization, merging, quantization, or runtime drift.

Selection workflow¶

Start from the evaluation gap. If the current checkpoint already meets the target, do not add a training stage.
Select the objective from the signal. Use SFT for correct demonstrations, DPO for chosen/rejected pairs, and online RL only when sampled outcomes can be scored.
Select parameterization separately. Choose full fine-tuning for maximum capacity, LoRA for a small reversible delta, or QLoRA when frozen-base memory is the constraint.
Select the reward and optimizer separately for RL. Call the setup RLVR only when the reward is verifiable; choose GRPO, PPO, or another optimizer on its own merits.
Fit the trainer. Start with FSDP2 or ZeRO state sharding, then add tensor, pipeline, context, or expert parallelism only for a measured capacity or scaling constraint.
Define the rollout contract. Fix group size or trajectory strategy, sampling, exact token records, reward execution, and the actor-versus-learner policy-version rule.
Choose placement from measured rates. Compare colocated, synchronous disaggregated, and bounded-asynchronous designs using observed rollout and update service times.
Name every precision plane. Treat trainer, frozen QLoRA base, actor weights/activations, KV cache, synchronization payload, and final serving precision independently.
Automate experiments only after freezing evaluation. An outer controller such as autoresearch-rl may explore configurations but must not mutate the held-out evaluator or trust boundary.
Promote, convert, and evaluate again. Keep or merge adapters, select the supported serving format, then gate the exact converted artifact and runtime.

This sequence is an operational default, not a mandatory algorithmic chain. Pure RL from a base checkpoint, iterative rejection sampling, distillation, and interleaved methods are valid branches when their signals and evaluation results justify them. The broader branches are mapped in fine-tuning and post-training, rejection sampling, and knowledge distillation method selection.

Don't-miss checklist¶

Classify every proposal by objective, reward source, optimizer, parameter update, trainer parallelism, rollout strategy, placement, synchronization, precision planes, and experiment control before comparing it with another proposal.
Treat the tokenizer and chat template as versioned model dependencies; a mismatched template can invalidate both SFT and RL results.
Keep training precision, rollout precision, and final serving precision as separate configuration fields.
Preserve actor policy version, exact token IDs, masks, sampling metadata, and behaviour log probabilities in every online trajectory.
Test distributed checkpoint save, restore, reshard, and consolidated export before a long FSDP or ZeRO run.
State whether disaggregation means actor versus learner, prefill versus decode, or both.
Test verifiers on correct equivalents, boundary values, malformed outputs, timeouts, and deliberate reward-hacking attempts.
Track reward variance within each GRPO group; all-equal groups produce no relative learning signal.
Budget online RL as rollout, reward execution, training, and weight synchronization, not as a normal training job with one extra loss.
Keep adapter lineage tied to the exact base revision; a LoRA adapter is not a self-contained model.
Run the held-out quality gate after adapter merge and after quantization on the exact serving runtime.
Freeze the evaluator and immutable preparation inputs before allowing an autonomous controller to modify a trial.

Failure modes¶

Category substitution: proposing LoRA as an alternative to SFT or GRPO confuses parameterization with the learning objective. Specify both dimensions.
RLVR treated as automatically correct: a deterministic verifier can still reward an exploit or reject a valid equivalent. Build adversarial verifier tests before training.
Zero-variance GRPO groups: all completions pass or all fail, so the relative advantage is zero. Adjust prompt difficulty, sampling, group size, or reward resolution.
Rollout starvation: too little rollout capacity leaves trainer GPUs idle. Measure generation and update time separately and rebalance the pools.
Adapter/base mismatch: an adapter loads against the wrong base, tokenizer, or target modules. Pin and validate the full composite artifact.
Training quantization confused with serving quantization: QLoRA's NF4 base solves training memory, while GPTQ, AWQ, SmoothQuant, FP8, and FP4 address different serving constraints.
Kernel-blind quantization: a smaller checkpoint does not guarantee lower latency. Confirm hardware support and profile the actual engine, batch, and context distribution.
Skipped post-conversion evaluation: merging, quantization, KV-cache precision, or a runtime upgrade changes numerics after the training gate. Block deployment on a second gate.
FSDP expected to solve rollout cost: FSDP shards the learner; it does not accelerate autoregressive actors. Size and optimize the rollout plane separately.
Ambiguous disaggregation: an architecture says only "disaggregated inference," leaving KV transfer confused with policy weight synchronization. Name the two boundaries explicitly.
Unbounded actor staleness: a fast learner consumes trajectories from unknown policy versions. Version actors, bound the lag, preserve behaviour log probabilities, and apply the validated correction.
Quantized actor mismatch ignored: the actor and trainer produce different token probabilities or tiny updates disappear under rounding. Track ratios, KL, code changes, and training health against a BF16 baseline.
Low-bit tensors differenced directly: packed codes or changed scales are treated as canonical weight deltas. Synchronize canonical weights or prove a format-aware delta path.
Outer-loop evaluator leakage: an experiment controller changes the code that judges its own candidate. Keep evaluation outside the mutable surface and replay the winner independently.

Open questions and validation¶

Does the base or instruction checkpoint already pass the held-out task gate?
Are there demonstrations, preference pairs, a scalar learned reward, or a deterministic verifier? Which signal actually exists today?
Can the verifier distinguish correct equivalents from formatted but wrong outputs, malformed inputs, and timeouts?
Do sampled groups contain enough reward variation to support a group-relative update?
Does LoRA match full fine-tuning on the held-out distribution at the selected rank and target modules?
Does the RL stack support the chosen adapter and base-quantization combination without changing optimizer semantics?
Does lower-precision rollout generation preserve the reward distribution observed with the training policy?
Does the final quantized artifact pass quality, latency, throughput, memory, and maximum-context gates on the target GPU and runtime?
Which trainer state exceeds one GPU, and does that require FSDP/ZeRO state sharding, model parallelism, or both?
Do measured actor and learner service times favor colocated, synchronous disaggregated, or bounded-asynchronous execution?
What is the maximum permitted actor policy lag, and which correction remains valid at that lag?
Does the actor need separate prefill and decode pools, and has KV transfer been profiled against the actual context distribution?
Do full and delta weight synchronization preserve the same actor output distribution at each supported precision?
Can an outer experiment controller reproduce the promoted trial from a clean checkpoint without access to held-out evaluator internals?

References¶

Hu et al., LoRA: Low-Rank Adaptation of Large Language Models. https://arxiv.org/abs/2106.09685
Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs. https://arxiv.org/abs/2305.14314
Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO). https://arxiv.org/abs/2402.03300
Lambert et al., Tulu 3: Pushing Frontiers in Open Language Model Post-Training (SFT, DPO, RLVR pipeline). https://arxiv.org/abs/2411.15124
Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. https://arxiv.org/abs/2210.17323
Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. https://arxiv.org/abs/2306.00978
Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. https://arxiv.org/abs/2211.10438
Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model. https://arxiv.org/abs/2305.18290
PyTorch, fully_shard API documentation. https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_shard.html
Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models. https://arxiv.org/abs/2410.18252
Li et al., QuRL: Efficient Reinforcement Learning with Quantized Rollout. https://arxiv.org/abs/2602.13953
Qiu et al., FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learning. https://arxiv.org/abs/2601.18150
Huang et al., QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs. https://arxiv.org/abs/2510.11696
vLLM, Disaggregated Prefilling. https://docs.vllm.ai/en/latest/features/disagg_prefill.html
autoresearch-rl, source repository pinned at commit c2bb32b154d382d3d9be05a312d28f04c3af40ff; the package metadata provides the source route. https://pypi.org/project/autoresearch-rl/ Related: Inference system map · Fine-tuning and post-training · SFT and LoRA/QLoRA · DPO · PPO · GRPO · GRPO variants · RLVR · Reward design · FSDP · Asynchronous RL systems · RL library selection · RL rollout redundancy · Rollout fleet sizing · Delta weight synchronization · Disaggregated inference · Quantization for inference · Multi-LoRA serving · Autonomous experimentation loops · Experiment tracking and model registry · LLM evaluation harness · Glossary