Markdown

OpenTinker: open RL-as-a-service¶

Scope: how an open-source stack turns reinforcement-learning post-training into a service, so a client submits a training job to a shared GPU pool instead of owning and operating the trainer. OpenTinker (open-tinker/OpenTinker, Apache-2.0) is the worked example: it separates a lightweight client, a scheduler that allocates GPUs to jobs, a verl-based training core, and an environment framework organized as a 2x2 design space. This is the self-hostable, open counterpart to the managed Tinker training-as-a-service API (whose paradigm it echoes by name; the repository states no affiliation), and it sits alongside the other RL libraries and the broader distributed training as a platform service. Its training core is verl; its algorithms are the GRPO family and agentic RL.

Facts and snippets are from the OpenTinker repository (README, scheduler guide, config, and file tree) and its paper (arXiv 2601.07376, "Separating Concerns in Agentic Reinforcement Learning," Zhu and You), read at research time; the last code push was March 2026 and the project self-classifies as Alpha, so verify current behavior on the repo. The install, config, and CLI blocks are reference templates on the real interface. The Python model of the scheduler and environment concerns is executed and asserted (stdlib only). Note the security caveat under "Running it in production": the repository ships a demo API key and a committed user database.

flowchart TB
  CLIENT["Client (lightweight)<br/>math_rl.py / gomoku_rl.py / ...<br/>Hydra-style CLI overrides"] -->|"POST /submit_job (+ otk_ API key)"| SCHED
  subgraph SCHED_PLANE["Scheduler concern"]
    SCHED["Job scheduler (FastAPI + dashboard)<br/>available_gpus, gpus_per_job<br/>submit / list / status / cancel / complete"]
  end
  SCHED -->|"allocate GPU block, spawn server"| CORE
  subgraph CORE_PLANE["Training core concern"]
    CORE["verl (git submodule, patched)<br/>PPO / GRPO / GRPO-per-step<br/>FSDP or Megatron engine"]
  end
  CORE <-->|"rollouts / rewards over HTTP"| ENV
  subgraph ENV_PLANE["Environment concern (2x2 design space)"]
    ENV["Env server: math / geo3k / gomoku /<br/>alfworld / android_world<br/>Data source x Interaction mode"]
  end

What it is¶

OpenTinker is an RL-as-a-service infrastructure for foundation models whose stated goal is "Democratizing Agentic Reinforcement Learning as a Service."¹ The organizing idea, and the title of its paper, is separating concerns: instead of one monolithic training script, the system is four cooperating pieces, each addressable on its own.

A client is a thin entrypoint (math_rl.py, gomoku_rl.py, and so on) that submits a job and takes Hydra-style key=value overrides. The README notes the client still borrows a small subset of functions from verl but is meant to become fully decoupled and lightweight.²
A scheduler (FastAPI plus a web dashboard) owns a GPU pool and hands blocks of GPUs to jobs. It exposes REST endpoints (/submit_job, /list_jobs, /job_status, /cancel_job, /complete_job, /register) and gates them behind otk_-prefixed API keys stored in a SQLite database.³
A training core is volcengine's verl, vendored as a git submodule and patched, carrying the PPO, GRPO, and per-step-GRPO trainers and the FSDP or Megatron engines.²
An environment framework serves prompts and rewards over HTTP, organized as a 2x2 design space that the next section details.

Because the concerns are separated, the client can run on a laptop with no GPU while the scheduler and training core run on the cluster, which is what "as a service" means here: the project page frames the system as enabling RL "without local GPU resources."¹ The paper's own emphasis is on managing multiple LoRA-backed policies on shared resources, including their optimizer states and training-data consistency.¹

Why use it¶

You want managed-service ergonomics on your own hardware. The managed Tinker API trades cluster ownership for a hosted endpoint; OpenTinker gives a similar submit-a-job workflow while keeping the GPUs, the weights, and the data on infrastructure you control, under an Apache-2.0 license.
Multi-tenant GPU sharing is built in. The scheduler allocates whole GPU blocks per job, queues the rest, and reclaims them on completion, so several users (or several experiments) can share one pool without hand-coordinating who gets which GPU.
Agentic and multi-turn RL are first-class. The environment framework and the grpo_per_step advantage estimator are aimed at multi-turn, tool-using agents (Gomoku, ALFWorld, AndroidWorld), not only single-turn verifiable-reward tasks, which is where much of the RL tooling still stops.⁵
LoRA and VLM paths ship as examples. There are documented recipes for LoRA math fine-tuning and for a vision-language geometry task, not just full-parameter text RL.⁵

When to use it (and when not)¶

Use it when you have a shared GPU cluster and want an open, self-hosted way for multiple people to launch agentic-RL jobs against it without each rebuilding the scheduler and rollout plumbing; when your tasks are genuinely multi-turn (games, embodied or GUI agents) and you want per-step credit assignment; or when you want a verl-based stack with a service layer already wrapped around it.

Do not adopt it for a production training platform without hardening it first: the project self-classifies as Alpha, its last code push was March 2026, and it ships demo credentials and a committed user database (see "Running it in production"). Prefer the managed Tinker API if you specifically want someone else to operate the cluster, or a bare verl or SkyRL setup if you do not need the multi-tenant service layer and would rather not carry a patched submodule.

Architecture¶

The scheduler is the service boundary. Its config (scheduler.yaml) declares the GPU pool (available_gpus: [0, 1, 2, 3]), the block size per job (gpus_per_job: 4), a port range for spawned training servers, and the scheduler's own port; a client authenticates with an otk_ key and calls /submit_job, then polls /job_status/{job_id} and finally /complete_job to release the GPUs.³ The scheduler spawns a training server per admitted job, which runs the verl core.

The environment framework's 2x2 design space crosses two orthogonal axes.⁴ The data source axis is data-dependent (load a structured dataset, for example parquet files, to provide prompts) versus data-free (generate prompts dynamically from a simulator or game engine). The interaction mode axis is single-turn (one-shot responses) versus multi-turn (iterative interaction with tool calls and feedback). The four cells map to distinct paradigms: data-dependent single-turn (math reasoning, QA), data-dependent multi-turn (tool-assisted problem solving), data-free single-turn (a bandit), and data-free multi-turn (game playing, dialogue agents). Each concrete environment (math, geo3k, gomoku, alfworld, android_world) runs as its own HTTP server started separately from training, which is what lets the environment concern evolve independently of the trainer.

The training core is verl at a pinned commit with a patch set under backend_patch/ touching the agent loop, the PPO core algorithms, the trainer, and the reward path; the documented advantage estimators are gae, grpo, and grpo_per_step, the last being the per-step credit assignment that the "separating concerns" work targets for multi-turn agents.²⁵

How to use it¶

Install with submodules (the verl core is a submodule), then run the four-step pattern that is identical across tasks: start the scheduler, start the environment server, prepare data if the task is data-dependent, and launch training. Reference templates (verify against the repo):

# Reference template (repo README). Clone with the verl submodule, install both.
git clone --recurse-submodules https://github.com/open-tinker/OpenTinker.git
cd OpenTinker && pip install -e .
cd verl && pip install -e . && cd ..          # the vendored training core

# Reference template (repo docs): a math single-turn run, end to end.
bash opentinker/scripts/launch_scheduler.sh --scheduler-port <scheduler_port>   # 1. scheduler
python opentinker/environment/math/math_server.py --port <env_port>             # 2. env server
python opentinker/data_preprocess/math_multiturn_w_interaction.py \
    --local_save_dir=<dir>                                                       # 3. data (data-dependent)
python opentinker/client/math_rl.py \                                            # 4. train
    tokenizer_path=Qwen/Qwen2.5-1.5B batch_size=16 num_epochs=5 \
    data_path=data/math_agentloop/train.parquet \
    scheduler_url=http://<host>:<scheduler_port> \
    interaction.config.env_port=<env_port>

A LoRA run swaps the client config (--config-name math_lora_param.yaml); a multi-turn game (Gomoku, ALFWorld, AndroidWorld) swaps the environment server and uses the grpo_per_step estimator. The client stays thin: it submits, monitors, and releases.

How to develop with it¶

The two properties worth validating before trusting a service layer are that the scheduler never double-allocates a GPU (the correctness that makes multi-tenant sharing safe) and that the environment router enforces the data contract of each 2x2 cell (a data-free environment must generate its own prompts, a data-dependent one must be given a dataset). This executed model implements both from the repository's documented behavior:

# opentinker_service.py -- validated: the two "separation of concerns" that make
# OpenTinker an RL-as-a-service layer, modeled from the repo's documented behavior.
# Pure stdlib.
#
# Part A: the scheduler concern. A central scheduler owns a fixed GPU pool and
# hands whole blocks of `gpus_per_job` GPUs to jobs, queueing the rest and never
# double-allocating a GPU. This is the correctness property that lets many clients
# submit training jobs to one cluster (scheduler.yaml: available_gpus, gpus_per_job).
#
# Part B: the environment concern. The 2x2 design space (Data source x Interaction)
# routes a task to one of four paradigms; the load-bearing invariant is that a
# Data-Free environment generates its own prompts (needs no dataset) while a
# Data-Dependent one must be given a data path.

from dataclasses import dataclass, field


# ---------------- Part A: the GPU job scheduler ----------------
@dataclass
class Scheduler:
    available_gpus: list[int]                  # scheduler.yaml: [0,1,2,3]
    gpus_per_job: int                          # scheduler.yaml: 4
    free: list[int] = field(init=False)
    running: dict[str, list[int]] = field(default_factory=dict)
    queue: list[str] = field(default_factory=list)

    def __post_init__(self):
        self.free = list(self.available_gpus)

    def submit(self, job_id: str) -> str:
        """Admit if a full block of gpus_per_job GPUs is free, else queue."""
        if len(self.free) >= self.gpus_per_job:
            gpus = [self.free.pop(0) for _ in range(self.gpus_per_job)]
            self.running[job_id] = gpus
            return "RUNNING"
        self.queue.append(job_id)
        return "QUEUED"

    def complete(self, job_id: str) -> None:
        """Release a job's GPUs (client calls /complete_job) and admit the next
        queued job if the freed block is now large enough."""
        self.free.extend(self.running.pop(job_id))
        if self.queue and len(self.free) >= self.gpus_per_job:
            self.submit(self.queue.pop(0))

    def allocated_gpus(self) -> list[int]:
        return [g for gpus in self.running.values() for g in gpus]


# A single-node pool of 4 GPUs, 4 GPUs per job -> exactly one job at a time.
sched = Scheduler(available_gpus=[0, 1, 2, 3], gpus_per_job=4)
assert sched.submit("job_a") == "RUNNING"
assert sched.submit("job_b") == "QUEUED"          # pool exhausted, job_b waits
assert sched.running["job_a"] == [0, 1, 2, 3] and sched.free == []

# CORRECTNESS: no GPU is ever handed to two jobs at once.
alloc = sched.allocated_gpus()
assert len(alloc) == len(set(alloc)), "a GPU must never be double-allocated"

# Completing job_a frees its block and auto-admits the queued job_b.
sched.complete("job_a")
assert sched.running.get("job_b") == [0, 1, 2, 3] and sched.queue == []
assert "job_a" not in sched.running

# A larger pool runs jobs concurrently without overlap.
big = Scheduler(available_gpus=[0, 1, 2, 3, 4, 5, 6, 7], gpus_per_job=4)
assert big.submit("j1") == "RUNNING" and big.submit("j2") == "RUNNING"
assert big.submit("j3") == "QUEUED"               # 8 GPUs / 4 = 2 concurrent jobs
a = big.allocated_gpus()
assert len(a) == len(set(a)) == 8                  # both jobs, disjoint GPUs
# Boundary: a job needing more GPUs than exist never runs, it just queues.
tiny = Scheduler(available_gpus=[0, 1], gpus_per_job=4)
assert tiny.submit("too_big") == "QUEUED" and tiny.running == {}


# ---------------- Part B: the 2x2 environment design space ----------------
PARADIGMS = {
    ("data_dependent", "single_turn"): "Math reasoning / QA (dataset, one-shot)",
    ("data_dependent", "multi_turn"): "Tool-assisted problem solving (dataset, iterative)",
    ("data_free", "single_turn"): "Bandit (simulator, one-shot)",
    ("data_free", "multi_turn"): "Game playing / dialogue (simulator, iterative)",
}


def route(data_source: str, interaction: str, data_path: str | None) -> str:
    """Dispatch a task to its paradigm and enforce the data invariant: a
    data-dependent environment loads a dataset (needs a data_path); a data-free
    environment generates prompts from a simulator (must NOT be given one)."""
    key = (data_source, interaction)
    assert key in PARADIGMS, f"unknown paradigm {key}"
    if data_source == "data_dependent":
        assert data_path is not None, "data-dependent env requires a dataset path"
    else:
        assert data_path is None, "data-free env generates prompts, needs no dataset"
    return PARADIGMS[key]


# All four documented paradigms route correctly with the right data contract.
assert route("data_dependent", "single_turn", "data/math/train.parquet").startswith("Math")
assert route("data_dependent", "multi_turn", "data/math/train.parquet").startswith("Tool")
assert route("data_free", "single_turn", None).startswith("Bandit")
assert route("data_free", "multi_turn", None).startswith("Game")

# Invariant violations are rejected, not silently mis-run.
for bad in [lambda: route("data_dependent", "single_turn", None),      # missing dataset
            lambda: route("data_free", "multi_turn", "data/x.parquet")]:  # spurious dataset
    try:
        bad(); raise SystemExit("should have raised")
    except AssertionError:
        pass

print(f"OK Part A: 4-GPU pool runs one 4-GPU job, queues the next, auto-admits on "
      f"complete, never double-allocates; 8-GPU pool runs 2 disjoint jobs. "
      f"OK Part B: all four 2x2 paradigms route with the correct data contract "
      f"(data-free needs no dataset, data-dependent requires one).")

Run output:

OK Part A: 4-GPU pool runs one 4-GPU job, queues the next, auto-admits on complete, never double-allocates; 8-GPU pool runs 2 disjoint jobs. OK Part B: all four 2x2 paradigms route with the correct data contract (data-free needs no dataset, data-dependent requires one).

The scheduler model captures why block allocation matters: a job needs a whole gpus_per_job block or it waits, so a pool that is not a multiple of the block size strands GPUs, and a /complete_job that a crashed client never sends leaks a block permanently. The environment model captures the data contract that keeps the 2x2 cells honest.

How to maintain it¶

Track the verl pin. The training core is verl at a specific commit (the patch notes base it on verl 0.7.0.dev0), applied through backend_patch/. A verl upgrade can break the patch, so treat the submodule commit and the patch set as one coupled unit and re-test after bumping either.
Keep the client thin. The README flags the client's dependency on verl as transitional; if you extend the client, avoid deepening that coupling so the intended lightweight, GPU-free client stays achievable.
Prefer the Docker path for servers. The repo recommends the pinned verlai/verl Docker image for the training server and warns that manual installation "may introduce version conflicts"; use the image to keep the core reproducible.

Running it in production¶

The load-bearing production caveat is credentials, because the repository ships insecure defaults that are convenient for a demo and dangerous on a shared cluster. The committed scheduler.yaml sets enable_auth: false (the README example shows true), a real-looking API key otk_... is hardcoded in the client config and appears verbatim in the guide, and a scheduler_users.db SQLite file is committed to the repository tree.⁶ Before any multi-user or internet-reachable deployment: turn authentication on, rotate to freshly generated keys (registration returns one that "cannot be retrieved after registration," so store it securely), remove the committed user database, and put the scheduler behind network controls rather than exposing its port directly. The scheduler is the GPU allocation authority for the whole pool; an unauthenticated one lets any reachable client submit jobs against your GPUs.

Beyond credentials, size the pool to the block: because the scheduler allocates whole gpus_per_job blocks, a pool whose size is not a multiple of the block size leaves GPUs permanently idle, and a client that dies without calling /complete_job leaks its block until you cancel the job manually. Monitor /list_jobs for jobs stuck running with no live client.

Failure modes¶

Shipped demo credentials treated as safe. The default enable_auth: false, the hardcoded otk_ key, and the committed user database are demo conveniences; leaving them in a real deployment exposes the GPU pool to any reachable client.⁶
Leaked GPU blocks from crashed clients. A job whose client never sends /complete_job holds its GPU block indefinitely; without monitoring, the pool silently loses capacity job by job.
Stranded GPUs from block-size mismatch. A pool size that is not a multiple of gpus_per_job can never fill, so some GPUs sit idle by construction; the executed model above shows a 2-GPU pool with a 4-GPU block admitting nothing.
Broken verl patch after an upgrade. The training core is a pinned, patched submodule; bumping verl without re-applying and testing the patch set silently changes trainer behavior or fails to build.
Assuming an official Tinker relationship. The name echoes the managed Tinker API, but the repository states no affiliation; do not infer compatibility or a shared API from the name.
Depending on an Alpha project unpinned. The project self-classifies as Alpha with its last code push in March 2026; pin to a commit you have tested rather than tracking main, and expect gaps (the LoRA example's performance table reads "TBA").

References¶

Zhu and You, "OpenTinker: Separating Concerns in Agentic Reinforcement Learning" (arXiv 2601.07376): https://arxiv.org/abs/2601.07376
OpenTinker repository (README, scheduler guide, examples; Apache-2.0): https://github.com/open-tinker/OpenTinker
OpenTinker project page: https://open-tinker.github.io/opentinker-page/
verl (volcengine, the vendored training core): https://github.com/volcengine/verl
ALFWorld (text-based embodied environment used as a multi-turn example): https://github.com/alfworld/alfworld
AndroidWorld (GUI-agent benchmark environment used as a multi-turn example): https://github.com/google-research/android_world

OpenTinker repository README and paper (arXiv 2601.07376, "Separating Concerns in Agentic Reinforcement Learning," Siqi Zhu and Jiaxuan You, UIUC): the tagline is "Democratizing Agentic Reinforcement Learning as a Service"; the GitHub description reads "OpenTinker is an RL-as-a-Service infrastructure for foundation models"; the project page frames it as RL training and inference "without local GPU resources." The arXiv abstract describes infrastructure for training language-model agents using multiple LoRA-backed policies on shared computational resources, managing policy-lifecycle elements including optimizer states and training-data consistency. The repository does not mention Thinking Machines or the Tinker API; any relationship is nominal, not stated. ↩↩↩
OpenTinker README, .gitmodules, and backend_patch/note.md: four components, a thin client (opentinker/client/, entrypoints such as math_rl.py), a scheduler (opentinker/scheduler/), a training core (verl vendored as a git submodule at verl/, pointing at github.com/volcengine/verl, patched under opentinker/backend_patch/ against verl 0.7.0.dev0), and an environment framework (opentinker/environment/). The README states the client "currently relies on a small subset of functions from verl. This dependency is transitional." Patch files touch the agent loop, PPO core algorithms, the trainer, the reward path, and rollout config. ↩↩↩
OpenTinker scheduler/SCHEDULER_GUIDE.md and scheduler/config/scheduler.yaml: the scheduler is FastAPI plus a web dashboard; config declares available_gpus: [0, 1, 2, 3], gpus_per_job: 4, a port_range, and scheduler_port: 8765. REST endpoints: POST /submit_job, GET /list_jobs, GET /job_status/{job_id}, DELETE /cancel_job/{job_id}, POST /complete_job/{job_id}, POST /register. API keys are otk_-prefixed and stored in a SQLite database (user_db_path, default scheduler_users.db); a key is returned once at registration and "cannot be retrieved after registration." ↩↩
OpenTinker README "Environments": a 2x2 design space over two orthogonal axes. Data source: data-dependent environments "load structured datasets (e.g., parquet files) to provide prompts," data-free environments "generate prompts dynamically from simulators or game engines." Interaction mode: single-turn (one-shot responses) versus multi-turn (iterative interactions with tool calls and feedback loops). The four cells map to math/QA (data-dependent single-turn), tool-assisted problem solving (data-dependent multi-turn), a bandit (data-free single-turn), and game playing or dialogue agents (data-free multi-turn). Concrete environments (math, geo3k, gomoku, alfworld, android_world) run as separate HTTP servers. ↩
OpenTinker README Quick Start and docs: documented examples include single-turn and multi-turn math (Qwen2.5-1.5B, the multi-turn variant adding a code-interpreter tool), LoRA single-turn math (Qwen2.5-0.5B, --config-name math_lora_param.yaml, performance "TBA"), a vision-language geometry task (Geometry3K, Qwen2-VL-2B), and multi-turn game or embodied agents Gomoku, ALFWorld, and AndroidWorld (Qwen2.5-3B-Instruct). Documented advantage estimators (alfworld/android docs) are gae, grpo, and grpo_per_step. ↩↩↩
OpenTinker scheduler/config/scheduler.yaml, client/client_config/opentinker_param.yaml, and repository tree: the committed scheduler config ships enable_auth: false while the README example shows enable_auth: true; a real-looking API key otk_98b8db24ccd64c92e1fdd9a232e209fa is hardcoded in the client config and appears as the guide's example; and a scheduler_users.db SQLite database is committed to the repo. These are demo defaults, not live secrets, but must be changed (auth on, keys rotated, database removed, network controls added) before any shared or internet-reachable deployment. The project self-classifies as Alpha (pyproject.toml "Development Status :: 3 - Alpha") with its last code push in March 2026. ↩↩