Skip to content
Markdown

Molt (NVIDIA NeMo Labs)

Scope: NVIDIA's agentic-first, PyTorch-native RL post-training framework, a ~8.6K-LOC stack of Ray placement + vLLM rollout + NVIDIA AutoModel/FSDP2 training, built for fully-async multi-turn and multimodal agentic RL up to 1T-class MoE.

Reference templates on real APIs; pin versions and validate before production use. The two python blocks below are dependency-free numpy re-implementations of Molt's actual advantage estimators and importance-sampling gate (molt/trainer/algorithm/advantage.py, molt/models/loss.py), executed and asserted, not illustrative pseudocode.

What it is

Molt (NVIDIA-NeMo/labs-molt) is a research-focused RL framework built around three thin layers: Ray for placement and async queues, vLLM for rollout, and NVIDIA AutoModel + FSDP2 for training in plain PyTorch. There is one trainable actor (plus an optional PPO critic or KL-reference), one Gymnasium-aligned agent API, and a token-first contract, token ids, logprobs, action ranges, rewards, and multimodal tensors stay aligned end-to-end from rollout through the policy loss. The framework targets AutoModel checkpoints (Qwen3.x, Nemotron-Omni3, Kimi-K2.6, GLM, Gemma, DeepSeek) natively, with tensor/expert/context parallelism (TP/EP/CP) that scales the same launch script from an 8B dense model to DeepSeek-V3-class MoE at --fsdp.ep_size 256. It ships molt.cli.train_sft and molt.cli.train_rl_ray, both CLI-flag-configured (no Hydra/YAML layer).

Why use it

  • Agentic-first, not RLHF-first. The agent is the program: Env.step() or ChatAgent.run() returns a Result(reward=...), and reward is any Python you write, graders, multi-turn tool loops, VLM environments, LLM-as-judge calls back through the same vLLM engines driving rollout. This is the same design axis as agentic RL, pushed all the way into the framework's core abstraction rather than bolted on.
  • Small enough to read end-to-end. ~8.6K lines of RL code (trainer, rollout, Ray orchestration, advantage/KL/loss, actor/critic) by the repo's own traced-import-graph count, versus ~62K for verl and ~25K for slime on the same basis. One actor, one page-sized RL graph, every gradient traceable to one file.
  • Frontier-scale MoE on plain PyTorch. AutoModel + FSDP2 with TP/EP/CP and Adam CPU offload reaches DeepSeek-V3-class MoE without a Megatron dependency, the same script that trains an 8B model.
  • OpenAI- and Anthropic-wire-compatible rollout. A ChatAgent subclass gets a real vLLM server on loopback speaking both /v1/chat/completions and /v1/messages, so external harnesses (browser automation, eval suites, even opaque tools like Claude Code or opencode) can drive the policy through a stock SDK while the framework still captures a token-exact trajectory, including automatic re-segmentation across context-compaction boundaries.
  • Explicit train/rollout drift correction. Async and partial rollout make the FSDP actor's recomputed logprobs diverge from vLLM's generation-time logprobs (different kernels, mid-request weight swaps); Molt's importance-sampling correction and MoE router replay/freeze address this directly rather than treating it as noise (see How to develop with it).

When to use it (and when not)

  • Use Molt when the RL loop is agent/tool/multi-turn-shaped and you want a small, single-actor, PyTorch-native codebase you can fork one layer of without touching the others, or when the target model is an NVIDIA AutoModel checkpoint and you want native TP/EP/CP without a Megatron dependency.
  • Prefer verl for the most battle-tested, throughput-first colocated stack across a broad model zoo (FSDP/FSDP2/Megatron, vLLM/SGLang/TRT-LLM). Prefer slime for the Megatron+SGLang decoupled stack that powers GLM. Prefer NeMo-RL when Megatron-Core-scale parallelism or FP8 end-to-end training is the priority over a minimal codebase. Molt's HF-transformers fallback (used only when AutoModel has no native class for a model) is text-only with no CP/EP/TP, so a model without an AutoModel-native path loses Molt's main selling point.
  • Compare the full landscape in RL libraries.

Architecture

flowchart LR
  subgraph Ray["Ray: placement + async queue"]
    Q["Async rollout queue"]
  end
  subgraph Roll["Rollout"]
    VLLM["vLLM engines (TP / EP / DP)"]
    AGT["Env / ChatAgent (reward = your Python)"]
  end
  subgraph Train["Trainer (single actor)"]
    FSDP["AutoModel + FSDP2 (TP / EP / CP)"]
    CRIT["Optional PPO critic"]
    REF["Optional KL reference / distill teacher"]
  end
  AGT -->|"token ids, logprobs, action ranges, reward"| Q
  VLLM --> AGT
  Q --> FSDP
  FSDP ==>|"weight broadcast"| VLLM
  REF -.->|"KL / reverse-KL"| FSDP
  CRIT -.->|"GAE value"| FSDP

How to use it

The recommended path is the project container (CUDA 13, torch 2.11, vLLM, TransformerEngine, flash-attn, DeepEP, AutoModel, prebuilt for A100/H100/H200/B200-GB200):

git clone https://github.com/NVIDIA-NeMo/labs-molt.git
cd labs-molt
docker pull hijkzzz/molt:latest          # or a pinned release, e.g. hijkzzz/molt:0.1.2
bash examples/scripts/docker_run.sh "python -c 'import vllm, transformer_engine; print(vllm.__version__)'"

Local (non-container) development installs the vllm extra directly: pip install -e ".[vllm]".

SFT and RL share the AutoModel/FSDP2 loading path and are both CLI-flag-configured, no YAML/Hydra layer:

# SFT
torchrun --standalone --nproc_per_node=8 -m molt.cli.train_sft \
  --model.model_name_or_path /path/to/automodel \
  --data.dataset /path/to/sft.jsonl --data.input_key input --data.output_key output \
  --ckpt.output_dir ./ckpt/sft --fsdp.attn_implementation te

# RL (single node, math grader agent)
python3 -m molt.cli.train_rl_ray \
  --actor.model_name_or_path /path/to/automodel \
  --data.prompt_dataset /path/to/prompts.jsonl --data.input_key input \
  --train.agent_path examples/python/agents/math.py \
  --vllm.num_engines 2 --vllm.tensor_parallel_size 2 \
  --rollout.batch_size 128 --train.batch_size 128 --train.micro_batch_size 1 \
  --algo.advantage.estimator reinforce --algo.kl.init_coef 0 \
  --fsdp.attn_implementation te --ckpt.output_dir ./ckpt/rl

Reference agents ship under examples/python/agents/: math.py (single-turn boxed grader, Env), geo3k.py (VLM multi-turn + Python tool, Env), chat_minimal.py (hello-world, ChatAgent), chat_geo3k.py (VLM multi-turn + Python tool over the chat wire, ChatAgent).

How to develop with it

The agent contract

--train.agent_path points at a module exporting AgentRunner. Choose one of two shapes, verified against molt/agents/base.py and molt/agents/chat_agent.py:

# Env: framework owns the LLM loop (Gymnasium-style step/reset)
from molt.agents import Env, Result, StepEnvRunner

class MathEnv(Env):
    async def step(self, state) -> Result:
        reward = grade(state["action_text"], state["label"])
        return Result(reward=reward, terminated=True)

class AgentRunner(StepEnvRunner):
    def __init__(self):
        super().__init__(MathEnv)

# ChatAgent: you own the loop via the OpenAI SDK (Anthropic SDK works identically
# against ctx.session_url, the same server auto-launched on loopback)
from openai import AsyncOpenAI
from molt.agents import ChatAgent, ChatAgentRunner, ChatContext, Result

class MyAgent(ChatAgent):
    async def run(self, ctx: ChatContext) -> Result:
        client = AsyncOpenAI(base_url=ctx.base_url, api_key=ctx.api_key)
        resp = await client.chat.completions.create(
            model=ctx.model_name, messages=[{"role": "user", "content": ctx.prompt}],
            max_tokens=ctx.sampling_params.max_tokens,
        )
        return Result(reward=grade(resp.choices[0].message.content, ctx.label))

class AgentRunner(ChatAgentRunner):
    def __init__(self):
        super().__init__(MyAgent)

Result (a dataclass in molt/agents/base.py) mirrors Gymnasium's step return: reward, observation, terminated, truncated, info, plus Molt's score (dynamic-filtering/dashboard metric, defaults to reward), images, and a per-turn sampling_params override.

Advantage estimators: RLOO, GRPO, and Dr.GRPO are different baselines, not aliases

Molt registers seven advantage estimators (reinforce, reinforce_baseline, rloo, grpo, dr_grpo, gae, on_policy_distill) behind one --algo.advantage.estimator switch. Three of them, rloo, grpo, dr_grpo, share the same group-baseline shape but diverge in exactly how they center and scale, which is easy to get wrong by eye. The block below is a numpy port of molt/trainer/algorithm/advantage.py's three functions, executed with asserts:

import numpy as np

def group_advantages(rewards, groups, method):
    """RLOO / GRPO / Dr.GRPO group baselines (molt/trainer/algorithm/advantage.py)."""
    adv = rewards.astype(float).copy()
    for group in groups:
        r = rewards[group]
        if method == "rloo":
            if len(group) > 1:
                adv[group] = r - (r.sum() - r) / (len(group) - 1)
            # singleton: raw reward, no baseline (matches molt's intentional no-op)
        elif method == "grpo":
            std = r.std() if len(r) > 1 else 0.0
            adv[group] = (r - r.mean()) / (std + 1e-9)
        elif method == "dr_grpo":
            adv[group] = r - r.mean()  # mean-centered, NOT divided by std
        else:
            raise ValueError(method)
    return adv

# Group 0: mixed rewards; Group 1: zero-variance (all-correct) group.
rewards = np.array([1.0, 0.0, 0.0, 0.0,   2.0, 2.0, 2.0, 2.0])
groups = [[0, 1, 2, 3], [4, 5, 6, 7]]

rloo = group_advantages(rewards, groups, "rloo")
grpo = group_advantages(rewards, groups, "grpo")
dr = group_advantages(rewards, groups, "dr_grpo")

# 1) All three baselines are mean-centered per group (sum ~ 0).
for adv in (rloo, grpo, dr):
    assert abs(adv[:4].sum()) < 1e-8
    assert abs(adv[4:].sum()) < 1e-8

# 2) GRPO divides by group std; dr_grpo does not, same sign pattern, different magnitude.
assert np.sign(grpo[:4]).tolist() == np.sign(dr[:4]).tolist()
assert not np.allclose(np.abs(grpo[:4]), np.abs(dr[:4]))

# 3) Zero-variance group collapses to exactly 0 under all three (no NaN/inf).
assert np.allclose(grpo[4:], 0.0) and np.allclose(dr[4:], 0.0) and np.allclose(rloo[4:], 0.0)

# 4) RLOO's baseline is the OTHER samples' mean, not the full group's mean, so its
#    correct-sample advantage is strictly larger than Dr.GRPO's on the same group.
assert rloo[0] > dr[0] > 0
assert abs(rloo[0] - 1.0) < 1e-12   # baseline = mean([0,0,0]) = 0  ->  adv = 1 - 0
assert abs(dr[0] - 0.75) < 1e-12    # baseline = mean([1,0,0,0]) = 0.25  ->  adv = 1 - 0.25

# 5) Singleton group: RLOO has no leave-one-out baseline, raw reward passes through untouched.
assert group_advantages(np.array([3.0]), [[0]], "rloo")[0] == 3.0

print("advantage estimators OK:", {"rloo": rloo.tolist(), "grpo": grpo.tolist(), "dr_grpo": dr.tolist()})

gae adds a learned value baseline (own Ray-actor critic, --algo.advantage.estimator gae, NeMoAutoModelForCausalLM + scalar value head, clipped value loss); on_policy_distill drops the scalar reward entirely and trains on per-token reverse KL to a frozen teacher (--ref.model_name_or_path), the only estimator that needs no reward function or task agent at all.

Importance-sampling correction: the aggregation level changes what gets rejected, not just how much

Async and partial rollout make the training-time recomputed logprob diverge from vLLM's generation-time logprob (different kernels, a weight swap mid-request the HTTP router cannot see). Molt corrects this with a per-token ratio pi_train / pi_rollout, gated two ways: is_correction_level {off, token, seq, geo} (off disables the correction entirely) picks how the ratio is aggregated before gating, is_correction_mode {mask, clip, trunc} picks what happens to a unit outside the [low, high] band. seq/geo are rejection filters over a whole sequence and, per molt/models/loss.py, only support mode=mask (clip/trunc would replace a per-token weight with one clamped sequence weight, which the code explicitly rejects at construction). The recipe default is level geo, mode mask ("seq-mask-tis"), a deliberately gentler filter than raw level seq (product-ratio reject). The block below reimplements the mask-mode gate from PolicyLoss.forward and demonstrates why geo and seq disagree on a real case, small, consistent per-token drift across a longer sequence:

import numpy as np

def is_correction_mask(train_logp, rollout_logp, action_mask, level, low, high):
    """molt/models/loss.py PolicyLoss.forward, mode='mask' only (the only mode
    seq/geo support; token also supports clip/trunc, omitted here)."""
    log_ratio = np.clip(train_logp - rollout_logp, -30.0, 30.0)
    token_ratio = np.exp(log_ratio)

    if level == "token":
        unit_ratio = token_ratio                                # per-token, no aggregation
    elif level == "seq":
        seq_log = (log_ratio * action_mask).sum(axis=-1, keepdims=True)
        unit_ratio = np.exp(np.clip(seq_log, -30.0, 30.0))       # product of token ratios
    elif level == "geo":
        denom = action_mask.sum(axis=-1, keepdims=True)
        seq_log = (log_ratio * action_mask).sum(axis=-1, keepdims=True) / denom
        unit_ratio = np.exp(seq_log)                             # geometric mean of token ratios
    else:
        raise ValueError(level)

    keep = (unit_ratio >= low) & (unit_ratio <= high)
    coef = np.where(np.broadcast_to(keep, token_ratio.shape), token_ratio, 0.0)
    return coef, keep

# One 8-token sequence, a small but CONSISTENT drift on every token (kernel-numerics gap,
# not one bad outlier).
n = 8
train_logp = np.full((1, n), -1.0 + 0.005)
rollout_logp = np.full((1, n), -1.0)
action_mask = np.ones((1, n))
low, high = 0.99, 1.01   # tight recipe-default band

# TOKEN level: each token's own ratio (exp(0.005) ~ 1.005) is inside the band -> all kept.
_, keep_tok = is_correction_mask(train_logp, rollout_logp, action_mask, "token", low, high)
assert np.all(keep_tok)

# SEQ level: the PRODUCT of 8 ratios compounds (exp(0.04) ~ 1.041, outside the band) ->
# the whole sequence is rejected even though no single token looked bad.
_, keep_seq = is_correction_mask(train_logp, rollout_logp, action_mask, "seq", low, high)
assert not keep_seq[0, 0]

# GEO level: the geometric MEAN of the same 8 ratios stays ~exp(0.005) ~ 1.005, inside the
# band, so it survives where "seq" rejected it. This is why seq-mask-tis (geo) is the
# recipe default: sequence length alone should not drive rejection.
_, keep_geo = is_correction_mask(train_logp, rollout_logp, action_mask, "geo", low, high)
assert keep_geo[0, 0]

# Adversarial: a genuinely large single-token blowup (weight-sync race) is still caught
# at token level, and its coefficient is exactly zeroed.
blown = rollout_logp.copy(); blown_train = rollout_logp.copy()
blown_train[0, 3] += 2.0   # ratio ~ exp(2) ~ 7.4, far outside the band
coef, keep_blown = is_correction_mask(blown_train, rollout_logp, action_mask, "token", low, high)
assert not keep_blown[0, 3] and coef[0, 3] == 0.0 and keep_blown[0, 0]

print("IS-correction gating OK: token survives per-token noise, seq compounds it away, geo doesn't")

MoE routing stability

MoE RL is unstable because vLLM's rollout router and the FSDP training router pick top-k experts independently; even at identical weights, numerical differences flip a fraction of the top-k per layer, breaking the importance-sampling assumption GRPO/GSPO rely on. Molt closes the gap at three levels, the first two on by default in the qwen3.5-moe recipes: fp32 router precision (matches vLLM's fp32 router, override with MOLT_GATE_PRECISION=bfloat16), Rollout Routing Replay (R3, --train.routing_replay), which replays vLLM's exact per-token expert selection in the training forward while leaving the router logits (and gradient) live (arXiv:2510.11370), and the blunter, off-by-default router freeze (--actor.freeze_moe_router), which excludes the gate/router weights from the optimizer and the refit entirely so rollout and actor route identically by construction, redundant with R3 and reached for only when drift still dominates.

How to run it in production

Scaling knobs

Mode Flag
Actor Tensor parallel --fsdp.tp_size 2
Expert parallel --fsdp.ep_size 8 (up to 256 for DeepSeek-V3-class MoE)
Context parallel --fsdp.cp_size 8 (32K+ sequences; packed batches disabled under CP)
Optimizer CPU offload --fsdp.offload optimizer
vLLM rollout Tensor / expert / data parallel --vllm.tensor_parallel_size, --vllm.enable_expert_parallel, --vllm.data_parallel_size
Scheduler token budget --vllm.max_num_batched_tokens 32768
MTP speculative decoding --vllm.mtp_num_speculative_tokens 1

MTP speculative decoding is rollout-only: like verl and NeMo-RL, Molt trains the main head alone; the draft head shares embed_tokens/lm_head with the target and refreshes on every weight broadcast, so it tracks the updating policy and acceptance degrades gracefully rather than off a cliff. It is lossless for the RL objective (accepted tokens follow the target's distribution) and vLLM auto-detects the per-architecture draft (Qwen3.6-MoE works out of the box; the Nemotron-Nano-Omni checkpoint ships no MTP head as of the README's writing, so rollout-MTP on it is unavailable until upstream adds it).

Fabric and precision

  • NVLink/NVSwitch: keep a TP group inside one node's NVLink domain; the trainer to vLLM weight broadcast (every step, or every few under async) and TP collectives both ride it (Blackwell platform).
  • InfiniBand/RoCE + GDR: multi-node EP/CP and cross-node weight sync want GPUDirect RDMA. Set NCCL_IB_HCA to the right HCAs, confirm [GDRDMA] in NCCL_DEBUG=INFO, ACS off (networking fabric, performance tuning).
  • Router precision: fp32 gate/combine is the default and matches vLLM; a bf16 router silently drifts and shows up as climbing vllm_kl.
  • Weight-update auditing: --train.check_weight_update_equal warns which vLLM params a broadcast left stale, catch a partial/failed sync before it silently trains on the wrong policy.

Slurm and quick-start recipes

Four reference launches ship under examples/scripts/{quick_start,slurm}/: Qwen3.6-35B-A3B VLM SFT/RL on geo3k (multi-turn Python tool), and Qwen3-4B dense SFT/RL on text math.

# Single-node smoke test
MODEL_PATH=/path/to/Qwen3-4B bash examples/scripts/quick_start/rl_qwen3_4b.sh

# Slurm: 2-node RL smoke, then scale to 4 nodes for convergence
sbatch examples/scripts/slurm/rl_qwen3_6_35b.sh
sbatch --nodes=4 examples/scripts/slurm/rl_qwen3_6_35b.sh

The geo3k VLM scripts auto-prepare the dataset on first run (examples/python/utils/prepare_geo3k.py --num-proc 8 --out-dir .tmp/geo3k), or point PROMPT_DATASET/EVAL_DATASET at your own data.

Monitoring

Watch vllm_kl (train/rollout logprob divergence, the same signal the IS-correction and router-replay machinery exist to bound) alongside the standard reward/entropy/KL triad for RL collapse (observability); a climbing vllm_kl with R3 and fp32 routing both on points at a genuine weight-sync problem, not routing noise.

How to maintain it

  • Pin the container. dockerfile/Dockerfile bakes CUDA 13 + torch 2.11 + vLLM + TransformerEngine + flash-attn + mamba + DeepEP + AutoModel as one matched set; pin a tagged image (hijkzzz/molt:0.1.2) rather than :latest for reproducibility, and rebuild locally only to change one of those pins deliberately.
  • Validate before scaling. python -m compileall -q molt examples/python tests && pytest -q locally; SKIP_BUILD=1 DOCKER_GPUS=all DOCKER_SHM_SIZE=32g bash examples/scripts/docker_run.sh "pytest -q" in-container before a multi-node run.
  • Custom advantage estimators register, they don't fork the trainer. @register_advantage_estimator("name") on a function of (rewards, groups, ctx) -> (advantages, returns) is the extension point; it never touches the framework's Experience objects, so a custom baseline stays a small standalone file.
  • The router-replay dependency is pinned to an upstream PR. R3 needs AutoModel's RouterReplay (nemo_automodel.components.moe.router_replay, PR #2797); re-check that AutoModel pin on any MoE-RL upgrade, and note R3 is incompatible with --train.partial_rollout_enable (vLLM frees routing state on preemption).
  • muon is experimental. Dion-style Newton-Schulz Muon for 2D weights/grouped MoE experts runs distributed under FSDP/EP but has shown no consistent win over adam yet; adam (with CPU offload for the largest actors) stays the recommended default.

Cookbook (common use cases)

1. Single-turn verifiable-reward RL (math grader, single node):

python3 -m molt.cli.train_rl_ray \
  --actor.model_name_or_path /path/to/Qwen3-4B \
  --data.prompt_dataset /path/to/math_prompts.jsonl --data.input_key input \
  --train.agent_path examples/python/agents/math.py \
  --algo.advantage.estimator grpo --rollout.n_samples_per_prompt 8 \
  --vllm.num_engines 2 --vllm.tensor_parallel_size 2 \
  --fsdp.attn_implementation te --ckpt.output_dir ./ckpt/rl

2. VLM multi-turn tool-use RL (geo3k, Slurm, 4 nodes):

sbatch --nodes=4 examples/scripts/slurm/rl_qwen3_6_35b.sh

3. Frontier MoE RL with routing stability on (DeepSeek-V3-class):

python3 -m molt.cli.train_rl_ray \
  --actor.model_name_or_path /path/to/deepseek-v3-automodel \
  --fsdp.ep_size 256 --fsdp.offload optimizer \
  --vllm.enable_expert_parallel --vllm.data_parallel_size 4 \
  --train.routing_replay \
  --algo.advantage.estimator grpo --algo.advantage.is_correction_level geo \
  --algo.advantage.is_correction_mode mask --algo.advantage.is_correction_threshold 0.99 1.01 \
  --train.agent_path examples/python/agents/chat_geo3k.py

4. On-policy distillation onto a frozen teacher:

python3 -m molt.cli.train_rl_ray \
  --actor.model_name_or_path /path/to/student \
  --ref.model_name_or_path /path/to/teacher \
  --algo.advantage.estimator on_policy_distill \
  --data.prompt_dataset /path/to/prompts.jsonl --data.input_key input

Failure modes

  • Non-AutoModel-native model. The HF transformers fallback (used only when AutoModel has no native class) supports text + flash_attention_2 + packing only, no CP/EP/TP, which erases Molt's scaling story for that model.
  • Routing drift misread as a reward bug. MoE rollout and training routers pick top-k independently; a climbing vllm_kl on an MoE actor is a routing-drift symptom before it is anything else. Check fp32 router precision and R3 are actually on before chasing the reward function.
  • seq/geo IS-correction with mode other than mask. The loss constructor raises at init (by design): sequence-level rejection filters carry a per-token IS weight for kept sequences and cannot be clip/trunc'd as one sequence-level number.
  • R3 plus partial rollout together. --train.routing_replay and --train.partial_rollout_enable are incompatible; vLLM frees per-token routing state on preemption, so replay has nothing to replay.
  • CP with packed batches. Packing is off by default and CP explicitly rejects --fsdp.packing_samples; a config that both enables CP and forces packing fails, not degrades silently.
  • Chasing muon for a win it hasn't shown yet. It is explicitly marked experimental with no consistent advantage over adam; don't treat its presence as a recommendation.
  • Pinning :latest instead of a release tag. The Docker image bundles CUDA/vLLM/TransformerEngine/AutoModel as one matched set; floating on :latest reintroduces exactly the dependency-wrangling the container exists to avoid.

References

  • Molt repo: https://github.com/NVIDIA-NeMo/labs-molt
  • Molt tech report: https://www.researchgate.net/publication/409325071_Molt_A_Scalable_PyTorch-Native_Training_Framework_for_Agentic_Reinforcement_Learning (DOI 10.13140/RG.2.2.23375.65447)
  • Rollout Routing Replay (R3): https://arxiv.org/abs/2510.11370
  • RLOO: https://arxiv.org/abs/2402.14740 · Dr.GRPO: https://arxiv.org/abs/2503.20783 · GRPO (DeepSeekMath): https://arxiv.org/abs/2402.03300
  • REINFORCE++: https://arxiv.org/abs/2501.03262 · Dual-clip PPO: https://arxiv.org/pdf/1912.09729
  • Anyscale, Open Source RL Libraries for LLMs: https://www.anyscale.com/blog/open-source-rl-libraries-for-llms

Related: RL Libraries · Agentic RL · GRPO · NeMo-RL · verl · slime · Token-In, Token-Out · Ray · Fine-tuning · Chat rendering and loss masking · Glossary