Markdown

Self-play with human-data regularization¶

Scope: a post-training method that keeps a self-play RL loop on a minimal safe reward and adds a small regularization term toward a human behavioral-cloning anchor, so the policy lands on the human-compatible equilibrium instead of an arbitrary one. Introduced in "Human-like autonomy emerges from self-play and a pinch of human data" (Cornelisse et al., 2026), demonstrated on autonomous driving. The technique itself is general: whenever self-play is cheap but reaches many equally-good equilibria and only some coordinate with people or with legacy systems, a small human-data term selects the compatible one. It sits in the same post-training family as GRPO and reward design, and shares the on-policy-divergence idea with on-policy distillation.

The numpy block under Architecture is executed and asserted in this page; it models the equilibrium-selection mechanism on a toy coordination game. The PPO block in "How to use it" is a labelled reference template (needs an RL stack, not run here). This is a single recent (2026) paper on one domain with one regularization weight, so validate the constants on your own setting.

What it is¶

Self-play RL trains a policy against copies of itself in simulation, with no human data, substituting cheap large-scale rollouts for expensive human demonstrations. Its known failure is that pure self-play can converge to effective but alien conventions: behaviors that are internally consistent and high-reward yet incompatible with how people act, because the self-play objective is indifferent to which of several equally-good equilibria it settles on.¹

The method keeps that self-play loop but changes the objective. The base reward stays minimal and safety-only (in the driving instance: +1 for reaching the goal, -1 for a collision or off-road event, 0 otherwise), and a human-data regularizer is added on top. A small set of human demonstrations is first distilled into a behavioral-cloning anchor tau_phi by minimizing negative log-likelihood on the human (observation, action) pairs. The training loss is then the PPO objective plus a KL term that pulls the policy toward that anchor, evaluated on the states the policy actually visits:¹

L(theta) = L_PPO(theta) + lambda * E_{o ~ rho_pi}[ D_KL( tau_phi(.|o) || pi_theta(.|o) ) ]

Two design choices carry the method. First, the KL is taken under the policy's own on-policy state distribution rho_pi, not the offline human distribution, so the regularizer shapes behavior on states the policy reaches rather than on a distribution it will drift away from (the classic imitation distribution-shift problem). Second, lambda is small: the reward, not the anchor, does most of the work, and the human term only breaks the tie between equilibria.¹

Why use it¶

Pure self-play picks an arbitrary equilibrium. A symmetric coordination problem (which side to pass on, which merge order, which turn-taking convention) has several equally-optimal solutions; self-play is indifferent among them and lands on whichever its initialization falls toward. That convention need not be the human one, so the policy is competent but cannot coordinate with people.¹
Reward engineering is the brittle fix. The usual alternative, hand-shaping the reward and adding domain randomization until behavior looks human, is labor-intensive and fragile; each new behavior needs a new term, and terms interact. Treating human data as a regularizer replaces that hand-tuning with a single anchor and one coefficient.¹
A little human data goes a long way. The demonstrated system uses 30 minutes of human demonstrations, which the paper reports as 0.04% of the training set and 2500x fewer than comparable imitation-learning baselines. The human term selects the equilibrium; it does not have to teach the whole task.¹
It stays cheap. Training completes in 15 hours on a single consumer-grade GPU, because the expensive part remains fast self-play simulation and the human term is one behavioral-cloning anchor, not a large offline dataset in the loop.¹

When to use it (and when not)¶

Use it when self-play (or any cheap self-generated experience) reaches a good policy but not a compatible one, you hold a small amount of human or reference behavior to anchor to, and you want to avoid a long reward-shaping campaign. The regularizer is the right tool for equilibrium selection.
Prefer plain self-play / GRPO when the task has a unique optimum, or when human compatibility is irrelevant (the policy never has to coordinate with people or legacy agents). The human term then only adds cost.
Prefer imitation learning or offline preference methods (DPO) when you have abundant demonstrations and no safe, fast simulator to self-play in; the point of this method is to need little human data, which only pays off when self-play supplies the bulk of the signal.
Do not use it to rescue an unsafe base reward. The regularizer selects among the equilibria the reward already allows; if the minimal reward does not encode safety (goal, collision, off-road), no anchor weight fixes that. Keep the safety terms in the reward.
Evidence caveat. This is one 2026 paper, in driving, with a single fixed regularization weight (lambda = 0.075) and limited ablation on it. Treat the mechanism as general but re-derive the usable weight for your setting.

Architecture¶

The loop is ordinary PPO self-play with one added term. All agents in an episode are driven by the same policy (decentralized self-play), the minimal safe reward scores each rollout, and PPO updates the policy on the resulting return. In parallel, the small human set is compiled once into a behavioral-cloning anchor; during training, a KL term between that anchor and the policy is evaluated on the states the policy visits and added to the loss with a small weight. That term is what selects the human-compatible equilibrium out of the many the reward alone would accept.

flowchart LR
  ENV["Self-play episodes<br/>(one shared policy drives all agents)"] --> RWD["Minimal safe reward<br/>(+1 goal, -1 collision / off-road)"]
  RWD --> PPO["PPO update on self-play return"]
  HUM["~30 min human demonstrations"] --> BC["Behavioral-cloning anchor tau_phi<br/>(NLL on human (obs, action) pairs)"]
  ENV --> VIS["States the policy actually visits (rho_pi)"]
  BC --> KL["lambda * KL(anchor || policy)<br/>on visited states"]
  VIS --> KL
  KL --> PPO
  PPO -->|"selects the human-compatible equilibrium"| ENV

The mechanism (runnable): equilibrium selection under a small human term¶

The core claim is not about driving; it is that a small regularizer breaks the symmetry between equally-good self-play equilibria toward the human one, without giving up the self-play gains. This numpy block models exactly that on a two-part toy: a symmetric convention (two equally-optimal coordination equilibria, p = 0 or p = 1) plus a competence dimension where self-play strictly beats the imperfect human anchor. It asserts the four behaviors the method rests on: (a) pure self-play (lambda = 0) lands on an arbitrary, possibly human-incompatible convention; (b) a small in-window lambda selects the human-compatible one while keeping competence near the self-play optimum; (c) too-large lambda collapses toward pure imitation and sacrifices that competence; (d) if self-play and the human already agree, the term is inert. Run: python3 toy_equilibrium.py.

# toy_equilibrium.py -- equilibrium selection by a small human-data regularizer. numpy only.
# Objective maximized by gradient ascent:
#   J(p, s) = [ p^2 + (1-p)^2 ]        # coordination: two symmetric optima (p=0 or p=1)
#           + w_task * s                # competence: self-play wants s -> 1
#           - lambda * [ KL(Bern(h_conv) || Bern(p)) + KL(Bern(h_skill) || Bern(s)) ]
# p = P(policy picks the R convention); humans use R (h_conv high) so the human-compatible
# equilibrium is high p. Pure self-play is indifferent between the two conventions.
import numpy as np

H_CONV = 0.9    # humans pick the R convention 90% of the time -> compatible eq = high p
H_SKILL = 0.5   # humans are imperfect; self-play competence beats this
W_TASK = 4.0    # self-play values competence strongly (strictly prefers s -> 1)


def bern_kl(a, p, eps=1e-9):
    a = np.clip(a, eps, 1 - eps); p = np.clip(p, eps, 1 - eps)
    return a * np.log(a / p) + (1 - a) * np.log((1 - a) / (1 - p))


def grad_kl_wrt_p(a, p, eps=1e-9):          # d/dp KL(Bern(a) || Bern(p))
    p = np.clip(p, eps, 1 - eps)
    return -a / p + (1 - a) / (1 - p)


def coordination(p):                         # shared-policy self-play: both agents ~ Bern(p)
    return p ** 2 + (1 - p) ** 2             # symmetric double-well: optima at p=0 and p=1


def train(p0, s0, lam, h_skill=H_SKILL, lr=0.01, steps=8000, gclip=10.0):
    """Gradient ascent on J = self_play_return - lam * divergence_from_human. Array-safe:
    p0, s0, lam broadcast, so a whole grid of lambdas ascends in one loop. The forward
    KL(anchor || policy) is mode-covering: its gradient diverges as the policy tries to drop
    an action the anchor takes, so clip the gradient (as PPO does)."""
    p, s, lam = np.broadcast_arrays(np.asarray(p0, float), np.asarray(s0, float),
                                    np.asarray(lam, float))
    p, s = p.astype(float).copy(), s.astype(float).copy()
    for _ in range(steps):
        gp = np.clip((4 * p - 2) - lam * grad_kl_wrt_p(H_CONV, p), -gclip, gclip)
        gs = np.clip(W_TASK - lam * grad_kl_wrt_p(h_skill, s), -gclip, gclip)
        p = np.clip(p + lr * gp, 1e-4, 1 - 1e-4)
        s = np.clip(s + lr * gs, 1e-4, 1 - 1e-4)
    return p, s


# (a) pure self-play: symmetry NOT broken -> convention decided by the init alone.
p_lo, s_lo = train(0.30, 0.5, lam=0.0)   # init biased toward L (the human-INCOMPATIBLE eq)
p_hi, s_hi = train(0.70, 0.5, lam=0.0)   # init biased toward R (compatible)
assert p_lo < 0.05 and p_hi > 0.95, (p_lo, p_hi)          # lands on opposite conventions
assert (p_lo < 0.5) and (p_hi > 0.5), "convention is init-dependent, i.e. arbitrary"
assert coordination(p_lo) > 0.99 and coordination(p_hi) > 0.99   # effective either way
assert s_lo > 0.99 and s_hi > 0.99, (s_lo, s_hi)          # and competent either way
p_tie, _ = train(0.5, 0.5, lam=0.0)                       # knife-edge symmetric init
assert abs(p_tie - 0.5) < 1e-6, p_tie                     # tie never broken without human data

# window bounds: lambda_min flips a wrongly-initialized policy (p0=0.30) onto the human
# convention; lambda_max is the largest weight keeping competence >= 0.9 of the optimum.
grid = np.linspace(0.0, 2.0, 401)[1:]
p_final, s_final = train(0.30, 0.5, lam=grid)             # whole grid trains at once
selects_R = p_final > 0.5
lam_min = float(grid[np.argmax(selects_R)])
compat, comp_s = grid[selects_R], s_final[selects_R]
lam_max = float(compat[comp_s >= 0.9].max())
assert lam_min < lam_max, (lam_min, lam_max)              # usable window is non-empty
assert not np.any(selects_R[grid < lam_min - 1e-9]), "flip must be a clean threshold"

# (b) a small in-window lambda selects the human eq WITHOUT collapsing self-play.
lam_demo = float(np.sqrt(lam_min * lam_max))              # geometric-mean point in the window
p_d, s_d = train(0.30, 0.5, lam=lam_demo)                # SAME wrong init as the lambda=0 case
assert p_d > 0.5, p_d                                     # convention now human-COMPATIBLE
assert s_d >= 0.9, s_d                                    # competence NOT collapsed
assert coordination(p_d) > 0.9, coordination(p_d)        # self-play coordination intact
# a pinch of human data: the selecting penalty is a small share of the return it overrides.
conv_share = (lam_demo * bern_kl(H_CONV, p_d)) / coordination(p_d)
assert conv_share < 0.05, conv_share                      # < 5% of the coordination return

# (c) too-large lambda collapses toward imitation: competence dragged to the human level.
p_big, s_big = train(0.30, 0.5, lam=20.0)
assert p_big > 0.5, p_big                                 # still R (imitation also coordinates)
assert s_big < 0.7 and abs(s_big - H_SKILL) < 0.1, s_big  # competence dragged to human's 0.5
assert s_big < s_d, (s_big, s_d)                          # strictly worse than the in-window run

# (d) boundary: if self-play and the human already agree, the term is inert.
p_agree0, _ = train(0.70, 0.5, lam=0.0)                   # self-play alone -> R
p_agreeL, _ = train(0.70, 0.5, lam=lam_demo)             # + human term (which also wants R)
assert abs(p_agree0 - p_agreeL) < 0.05, (p_agree0, p_agreeL)   # same equilibrium, lambda inert
_, s_a0 = train(0.30, 0.5, lam=0.0, h_skill=1.0)         # anchor already optimal on competence
_, s_aL = train(0.30, 0.5, lam=5.0, h_skill=1.0)
assert s_a0 > 0.99 and s_aL > 0.99, (s_a0, s_aL)          # agreement -> regularizer inert

print("(a) self-play lam=0: p from init 0.30 =", round(float(p_lo), 4),
      "(L, INCOMPATIBLE); from 0.70 =", round(float(p_hi), 4), "(R, compatible)")
print("    effective either way: coordination =", round(float(coordination(p_lo)), 4),
      "competence =", round(float(s_lo), 4), "| knife-edge init 0.5 stays", round(float(p_tie), 6))
print("window: lam_min =", round(lam_min, 4), "lam_max =", round(lam_max, 4),
      "demo =", round(lam_demo, 4))
print("(b) demo lam, SAME wrong init 0.30: p =", round(float(p_d), 4),
      "(R, COMPATIBLE), competence =", round(float(s_d), 4),
      "| selecting penalty share =", round(float(conv_share), 4))
print("(c) lam=20 over-regularized: p =", round(float(p_big), 4), "competence =", round(float(s_big), 4),
      "(collapsed toward human", H_SKILL, ")")
print("(d) already-agree: p(lam=0)=", round(float(p_agree0), 4), "p(lam=demo)=", round(float(p_agreeL), 4),
      "| h_skill=1: s(lam=0)=", round(float(s_a0), 4), "s(lam=5)=", round(float(s_aL), 4))
print("ALL ASSERTIONS PASSED")

Run output:

(a) self-play lam=0: p from init 0.30 = 0.0001 (L, INCOMPATIBLE); from 0.70 = 0.9999 (R, compatible)
    effective either way: coordination = 0.9998 competence = 0.9999 | knife-edge init 0.5 stays 0.5
window: lam_min = 0.285 lam_max = 0.895 demo = 0.505
(b) demo lam, SAME wrong init 0.30: p = 0.9788 (R, COMPATIBLE), competence = 0.9408 | selecting penalty share = 0.0419
(c) lam=20 over-regularized: p = 0.838 competence = 0.5495 (collapsed toward human 0.5 )
(d) already-agree: p(lam=0)= 0.9999 p(lam=demo)= 0.9788 | h_skill=1: s(lam=0)= 0.9999 s(lam=5)= 0.9999
ALL ASSERTIONS PASSED

The window [0.285, 0.895] is specific to this toy and its scales, not the paper's number; what transfers is the shape. There is a usable band for lambda: large enough to flip a policy that self-play would otherwise leave on the alien convention, small enough that the self-play competence gain survives. Below the band the symmetry is not broken; above it the policy regresses toward the imperfect anchor. The < 5% share confirms the data-efficiency intuition: the term that decides the equilibrium is a small fraction of the return it overrides.

How to use it¶

Assemble three pieces on top of a working self-play loop: a minimal safe reward, a behavioral-cloning anchor compiled once from a small human set, and an on-policy KL term with a small weight. Initialize episodes from real logged scenarios so the state distribution is realistic, run PPO self-play with all agents sharing the policy, and add the anchor KL to the loss. The block below is a reference template (needs an RL stack and a simulator; not run here); the equilibrium-selection math it depends on is the executed block above.

# reference template: pin your RL stack and simulator; verify APIs before use.
# Base: PPO self-play, all agents share one policy, minimal safety-only reward.
def safe_reward(event):
    if event == "goal":                 return  1.0
    if event in ("collision", "offroad"): return -1.0
    return 0.0

# Anchor: behavioral cloning on a SMALL human set (compiled once, then frozen).
anchor = train_bc(human_obs_action_pairs)     # minimize NLL; ~30 min of demos in the paper

def loss(batch, policy, lam=0.075):
    l_ppo = ppo_objective(batch, policy)      # self-play return drives the direction
    # KL on the states the POLICY visits (on-policy), not the offline human distribution.
    obs = batch.observations                  # sampled from rho_pi
    l_reg = lam * kl(anchor.dist(obs), policy.dist(obs)).mean()
    return l_ppo + l_reg                       # maximize return, minimize divergence from anchor

The two non-obvious points are that the anchor KL is evaluated on the policy's own visited states (which is what avoids the imitation distribution-shift trap), and that lambda is small: the reward is the objective and the anchor only selects among its equilibria.

How to develop and tune it¶

Iterate on three things, in order: the base reward, the anchor, and the weight lambda.

Keep the base reward minimal and safety-complete. The method's premise is that human compatibility comes from the regularizer, not from reward shaping, so resist adding behavioral terms to the reward. But the reward must still contain every safety constraint (goal, collision, off-road), because the anchor selects only among the equilibria the reward permits.
The anchor is a behavioral-cloning model, and its data amount is the real knob. The paper studies subsets of roughly 10 minutes, 30 minutes, 3 hours, and 30 hours; more anchor data sharpens the target distribution but the reported system already coordinates from 30 minutes. Anchor quality bounds the ceiling: a biased or narrow human sample selects a biased equilibrium.
Tune lambda for the window, not a point. As the runnable block shows, there is a usable range: too small and the symmetry is not broken (the policy keeps its arbitrary convention); too large and the policy regresses toward the imperfect anchor, giving up the self-play competence gain. The paper fixes lambda = 0.075 for its setting with limited ablation, so re-derive the window on your task rather than copying the value.
Watch coordination and self-play return together. The failure you are tuning against is invisible in self-play return alone (an alien policy scores well against copies of itself). Track a coordination metric against held-out human behavior alongside the return, and move lambda up only until coordination is achieved, not further.

How to run it in production and at scale¶

The budget is small and the bottleneck is simulation. The reported run is 15 hours on a single consumer-grade GPU; the simulator does the heavy lifting (the paper's PufferDrive reports about 390k steps per second on one high-end consumer card), and the run consumes on the order of 20 billion self-play transitions. Provision for fast simulation throughput, not for a large data pipeline.
The anchor is cheap and static. It is one behavioral-cloning model, compiled once from a small human set and frozen; there is no large offline dataset streaming through the loop and no second policy to keep in sync. The added per-step cost is one anchor forward pass over the visited observations.
Gate promotion on held-out human coordination, not on self-play score. Evaluate the policy against logged human trajectories it never trained on (the paper's human-replay setting, where the controlled agent must coordinate with non-reactive logged co-players). Self-play score is exactly the number an alien policy inflates, so it cannot be the promotion gate.
Keep the KL on the policy's state distribution. At scale it is tempting to precompute the anchor KL on a fixed offline batch; do not. The on-policy expectation rho_pi is load-bearing, because it regularizes the states the policy will actually reach.

How to maintain it¶

Keep both baselines as regression anchors. Run pure self-play (lambda = 0) and pure imitation (lambda large) on the same task alongside the method. The regularized policy should beat pure imitation on task competence and beat pure self-play on human coordination; if it stops beating either, lambda or the anchor is misconfigured (the two extremes are the (a) and (c) cases in the runnable block).
Re-validate coordination after any change. A new base reward, a new anchor dataset, or a new simulator version can move the equilibrium the policy selects. Re-check held-out human coordination, not just self-play return, on every such change.
Treat the constants as version-specific. lambda = 0.075, the 30-minute anchor, and the driving reward come from one paper on one domain. Cover the equilibrium-selection behavior with the library-independent assertions above and re-derive the window per setting.
Guard the anchor data provenance. Because the anchor defines "human-compatible," a shift in who the demonstrations came from silently shifts the target the policy is pulled toward.

Results¶

Reported for the driving instance (self-play PPO on a fast simulator, initialized from Waymo Open Motion Dataset scenarios):¹

Human data: 30 minutes of demonstrations, which the paper states is 0.04% of the training set and 2500x fewer than comparable imitation-learning approaches. The imitation baseline it contrasts with (SMART) is trained on tens to hundreds of millions of logged transitions, on the order of dozens of days to months of driving.
Compute: 15 hours on a single consumer-grade GPU. The run consumes about 20 billion self-play transitions (roughly 63 years of experience at 10 Hz), made affordable by simulator throughput near 390k steps per second on one high-end consumer card.
Coordination: the resulting policies coordinate with held-out human trajectories. The paper evaluates in three settings (human-replay against logged non-reactive co-players, co-players following the Intelligent Driver Model, and full self-play) on a held-out validation split, and reports an aggregate Score (goal reached with no collision or off-road event), plus collision rate, at-fault collision rate, off-road rate, route progress, completion rate, distributional realism, and collision severity. The headline is that the regularized policy attains human-compatible behavior that pure self-play does not, at a fraction of the data an imitation approach needs.

The exact per-metric tables are in the paper; the load-bearing result for this page is the qualitative one, that a 30-minute anchor plus a small on-policy KL term converts a competent-but-alien self-play policy into a human-compatible one on a single-GPU budget.

Failure modes¶

lambda too large: collapse to imitation. The policy regresses toward the imperfect anchor and gives up the self-play competence gain (validated: the lambda = 20 case drags competence to the human level). Symptom: task competence falls toward the anchor's, coordination no longer improves.
lambda too small: symmetry not broken. The human term is too weak to move the policy off the arbitrary convention self-play would pick, so behavior stays alien (validated: below lam_min the wrongly-initialized policy keeps the incompatible convention).
Unsafe or over-shaped base reward. The regularizer selects among the equilibria the reward allows; it cannot add a missing safety constraint, and a heavily shaped reward defeats the point of using the anchor for behavior. Keep the reward minimal but safety-complete.
Offline KL reintroduces distribution shift. Evaluating the anchor KL on a fixed offline batch instead of the policy's visited states rho_pi recreates the imitation-learning mismatch the method was built to avoid.
Biased or narrow anchor. The anchor defines the human-compatible target, so demonstrations from an unrepresentative population or regime select a correspondingly biased equilibrium.
Over-generalizing the evidence. Results are from one 2026 paper, in driving, with a single lambda. The mechanism (a small human-data term selecting a human-compatible equilibrium) is general; the specific constants are not.

References¶

Cornelisse, Hunt, Zhang, Doulazmi, Joseph, Fisac, Vinitsky, "Human-like autonomy emerges from self-play and a pinch of human data" (arXiv 2606.19370, submitted 2026-06-11): https://arxiv.org/abs/2606.19370
Schulman et al., "Proximal Policy Optimization Algorithms" (the base RL algorithm): https://arxiv.org/abs/1707.06347
SMART, "Scalable Multi-agent Real-time Motion Generation via Next-token Prediction" (the imitation-learning contrast point): https://arxiv.org/abs/2405.15677
Ettinger et al., "Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset" (scenario source and held-out evaluation data): https://arxiv.org/abs/2104.10133

Cornelisse et al., arXiv 2606.19370. Method: self-play PPO with all agents sharing one policy, on a minimal safe reward (+1 goal, -1 collision/off-road, 0 otherwise), plus a regularizer L = L_PPO + lambda * E_{o ~ rho_pi}[ D_KL( tau_phi(.|o) || pi_theta(.|o) ) ], where tau_phi is a behavioral-cloning anchor trained by NLL on the human demonstrations and the KL is taken on the policy's on-policy state distribution rho_pi (as reported). Reported quantities: lambda = 0.075; anchor subsets of about {10 min, 30 min, 3 h, 30 h}; the headline system uses 30 minutes, stated as 0.04% of the training set and 2500x fewer than comparable imitation learning; training completes in 15 hours on a single consumer-grade GPU using about 20 billion transitions (~63 years at 10 Hz), with simulator throughput near 390k steps/sec on a high-end consumer GPU; evaluation in human-replay, IDM co-player, and self-play settings on a held-out validation split, reporting an aggregate Score, collision rate, at-fault collision rate, off-road rate, route progress, completion rate, distributional realism, and collision severity. Numbers are the paper's; the KL direction and the exact objective are transcribed from its method section and should be verified against the source. ↩↩↩↩↩↩↩↩