Skip to content
Markdown

Own or rent a coding model

Scope: the decision layer for putting an open-weight coding model behind developers instead of buying hosted seats. It answers whether to do it, at what size, at what cost per developer, and how you would know the model is good enough: the five deployment states from hosted API to continued pretraining, sizing one replica and then a fleet, the per-developer economics against a hosted API, an internal eval that can actually detect a capability difference, and the four gates that should stop the project. It is not the how-to. It owns the seat economics, the replica count, and the eval sizing; it reuses the utilisation and headroom conventions of GPU capacity planning and the GPU cost basis of the build-vs-rent model, and it delegates model selection to serving open-weight models, every vllm serve flag and launch shape to the per-model cookbooks, adaptation to SFT and LoRA, and the eval harness to the LLM evaluation harness. The single-developer, single-box build is running local coding agents. Read this page first; if it says yes, the end-to-end bring-up (pin, serve, verify tool calling, gateway, harness wiring, sandbox, live-endpoint acceptance) is Cookbook: own and run an open-weight coding model, and the remaining mechanics are in the pages it points to.

The three Python blocks below are executed and asserted (stdlib only, Python 3.9+), including adversarial, boundary, and sweep cases. The eval block records two genuine discrepancies this page found and resolved: a closed-form sample size that the exact test contradicts, and the naive remedy for it, which also fails. Every dollar figure, token count, throughput number, cache multiplier, and GPU count is an illustrative input you must replace with your own measurements; nothing here is a benchmark result. The gateway and serving snippets are unexecuted reference templates. Model specifications are as published on the vendors' cards and are not independently verified. The Qwen worked example uses the official FP8 checkpoint, not the BF16 checkpoint.

flowchart LR
  DEV["Developers (N seats)"] --> HARNESS["Coding harness (IDE or CLI)"]
  HARNESS --> GW["Gateway: auth, quota, routing"]
  GW -->|"most tasks"| FLEET["Owned fleet: vLLM replicas of an open-weight coder"]
  GW -->|"escape hatch"| API["Frontier API: hardest tasks, overflow"]
  FLEET --> CACHE["Prefix cache, salted per tenant"]
  FLEET --> TEL["Telemetry: tokens, TTFT, acceptance"]
  API --> TEL
  TEL --> EVAL["Internal eval on your repositories"]
  EVAL -->|"gates every promotion"| FLEET

What it is

Owning a coding model means holding four assets that a hosted seat rents to you: the weights, the serving stack, the evaluation that says whether the model is good enough, and the adaptation that bends it toward the codebase. For most organizations it does not mean pretraining; that remains a lab-scale activity and appears below only as a stop sign.

Ownership comes in five deployment states, and the cost and difficulty rise steeply with each one. Only the last two train weights:

State You control You do not control What pushes you here
0. Hosted API or seat prompts, harness weights, price, deprecation, roadmap the default
1. Open weights, rented GPUs weights, version pinning, serving the hardware data residency, price control, pinning
2. Open weights, owned GPUs state 1 plus hardware and residency base model capability sustained load, strict residency
3. Adapted weights (LoRA) state 2 plus house style and internal APIs base model capability recurring house-specific failures
4. Continued pretraining state 3 plus base capability frontier capability almost never justified

Most businesses that say "we want to own our coding model" want state 1 or state 2. State 3 is worth it only after the harness and the context pipeline are exhausted, for reasons the development section makes concrete.

The word "SOTA" deserves care. Open-weight coders publish strong vendor-reported coding scores, and the model cheat-sheet lists the current flagships. Those numbers are vendor-reported, and some of the public benchmarks behind them are measurably contaminated. What an organization can verify is how a model performs on its repositories, which is why the eval step is not optional.

Why use it

  • Data residency and IP. Prompts from a coding agent carry proprietary source, secrets in context, and architecture. For a codebase under contractual or regulatory constraints this is the deciding factor, and it is the only one that API pricing cannot answer.
  • Version pinning. A hosted model can change under you between Tuesday and Wednesday. An owned checkpoint plus a pinned engine version gives a reproducible agent, which matters when the agent's output is reviewed, audited, or replayed.
  • Cost, but only sometimes, and only if you measure. Self-hosting is a fixed monthly cost against a linear per-token bill. Whether it can ever win is decided by one comparison, given exactly in step 3: the API's bill per seat against the fleet's marginal cost per seat. At the illustrative throughput below, self-hosting cannot beat a budget-priced API at any seat count, and it beats a frontier-priced one from a few dozen seats. Both answers move when your measured throughput moves.
  • Capability control. You choose when to upgrade, you can serve a per-team adapter, and you can refuse a regression that a vendor ships.

When to use it (and when not)

Four gates, in order. Fail any one and the answer is a hosted API.

  1. The capability gate. If the open model cannot do the work, no amount of infrastructure rescues it. Below the capability floor a better harness cannot manufacture missing base capability; above it, the harness often matters more than the model (harness architecture). Measure this on your own eval before you buy anything.
  2. The economics gate. Compute the marginal cost per seat from your measured throughput and compare it to the per-seat bill of the API tier you would actually buy. If the API bills less than your marginal cost, no seat count ever makes self-hosting cheaper.
  3. The evaluation gate. If you cannot tell whether a model change helped, you cannot own a model, because owning means deciding when to upgrade. A 30-task paired eval cannot see anything smaller than an 18-point difference at 25% discordance.
  4. The operations gate. An owned fleet needs on-call, capacity planning, driver and engine upgrades, and an SLO. If nobody owns the pager, the fleet degrades quietly, and the salary of whoever does own it is usually larger than the GPU bill at small scale.

Use it when residency or pinning is a hard requirement, or when your measured marginal cost per seat sits clearly below the API bill per seat. Avoid it when the workload is bursty and small (a fleet idles at the same cost), when the team has no eval capability, or when the deciding argument is "it will be cheaper" and nobody has run the arithmetic.

Architecture

Two planes. The serving plane is a gateway in front of N replicas of one open-weight coder, with a prefix cache and an escape hatch to a frontier API for the hardest tasks. The evidence plane is telemetry flowing from both back into an internal eval that gates every promotion of a new checkpoint, engine version, or harness change. The top diagram shows both.

The load-bearing property of the serving plane is that agentic coding is input-heavy: each turn resends the repository context and the conversation history. In the illustrative workload below the input-to-output ratio is 28.6 to 1. That makes the prefix cache the dominant cost lever, and it means the bottleneck can move as the cache warms. vLLM's automatic prefix caching accelerates the prefill phase only and explicitly does not speed up decoding, so caching removes prefill work and never removes decode work.

The economic structure is a step function meeting a line, which is why there is no single break-even seat count. The fleet costs a whole replica at a time; the API bills per token. Whether the two lines ever cross at all is decided before any scan:

flowchart TD
  M["Marginal fleet cost per seat<br/>(set by your measured throughput)"] --> CMP{"API bill per seat<br/>&gt; marginal fleet cost?"}
  CMP -->|"no"| NEVER["Self-hosting never wins on cost.<br/>Own it for residency, pinning, or control."]
  CMP -->|"yes"| BAND["A finite settle point exists.<br/>Below it the decision oscillates<br/>as each replica step lands."]
  BAND --> OPS["Add staffing. It is a fixed cost<br/>and it dominates at small seat counts."]

Routing is what keeps the capability gate honest. Rather than choosing one model for everything, route per task and keep a frontier escape hatch. Agent-as-a-Router formalises this for coding specifically and reports that giving a router task-dimension performance statistics yields a 15.3% relative gain over a vanilla LLM router. The mechanics live in LLM request routing.

How to use it

1. Decide which deployment state you are buying

Pick a state from the table above and write down the reason. If the reason is cost, size the replica in step 2 and run the economics in step 3 before committing to any hardware. If the reason is residency, the economics still matter but they no longer decide.

2. Pick the model and size one replica

Choose from the coding rows of the cheat-sheet, then compute the memory floor. Memory is set by total parameters; compute is set by active parameters. Sizing an MoE model on its active count is the most common and most expensive mistake in this whole exercise.

This block is executed and asserted. It is a memory-floor check, not a production launch certificate; validate the production launch shape on the target engine, model revision, context length, and GPU type:

# replica_fit.py -- validated: the memory floor for one coding-model replica.
# Memory comes from TOTAL params (all experts are resident). Compute comes from ACTIVE params.
from math import ceil

def weights_gb(total_params_b, bytes_per_param):
    return total_params_b * 1e9 * bytes_per_param / 1e9      # fp8/int8=1, bf16=2, int4=0.5

def gpus_per_replica(total_params_b, bytes_per_param, gpu_gb, overhead=0.15):
    """overhead reserves activations plus a KV floor; see the KV planner in
    cookbook-thunderagent-agentic-serving.md for a context-aware budget."""
    return ceil(weights_gb(total_params_b, bytes_per_param) / (gpu_gb * (1 - overhead)))

def max_overhead(total_params_b, bytes_per_param, gpu_gb, gpus):
    """The largest non-weight budget under which `gpus` still holds the weights."""
    return 1 - weights_gb(total_params_b, bytes_per_param) / (gpus * gpu_gb)

# --- CROSS-CHECK 1: GLM-5.2 at ~744B fp8 is served on an 8xH200/H20 (141 GB) node and
# never on an 8xH100 (80 GB) node. Weights alone (744 GB) exceed 8x80GB = 640 GB, so the
# arithmetic reproduces the published split with no vendor input.
assert weights_gb(744, 1) > 8 * 80                 # cannot fit at ANY overhead
assert gpus_per_replica(744, 1, 80) > 8            # 11 GPUs
assert gpus_per_replica(744, 1, 141) <= 8          # 7 GPUs at the default 15% overhead

# --- CROSS-CHECK 2: Ornith-1.0 at 397B fp8 fits one 8x80GB node.
assert gpus_per_replica(397, 1, 80) <= 8           # 6 GPUs

# --- ADVERSARIAL: the 8-GPU shapes are OVERHEAD-SENSITIVE, and GLM-5.2 is the tightest.
# The published shape only tells you the weights fit; it bounds the non-weight budget.
assert round(max_overhead(744, 1, 141, 8), 3) == 0.340   # GLM-5.2 on 8xH200: <=34.0%
assert round(max_overhead(397, 1, 80, 8), 3) == 0.380    # Ornith on 8x80GB:   <=38.0%
assert round(max_overhead(80, 1, 80, 2), 3) == 0.500     # Qwen FP8 on 2x80GB: <=50.0%
# So at the 35% overhead the sibling local-coding-agents.md assumes, GLM-5.2 tips to 9 GPUs
# while Ornith and Qwen still fit. This is why GLM-5.2's cookbook caps --max-num-seqs.
assert gpus_per_replica(744, 1, 141, overhead=0.35) == 9
assert gpus_per_replica(397, 1, 80, overhead=0.35) <= 8
assert gpus_per_replica(80, 1, 80, overhead=0.35) == 2

# --- TRAP 1, the MoE trap. Qwen3-Coder-Next is 80B total / 3B ACTIVE. Sizing on the
# active count says one GPU, and the server then fails to load 80 GB of weights.
assert gpus_per_replica(3, 1, 80) == 1 < gpus_per_replica(80, 1, 80)

# --- TRAP 2, the precision trap. The FP8 card's vLLM line uses `--tensor-parallel-size 2`
# but states no GPU memory size. TP=2 is consistent with 2x80GB only for the FP8
# checkpoint; the BF16 checkpoint is 160 GB and does not fit two 80 GB GPUs.
assert gpus_per_replica(80, 1, 80) == 2            # FP8 on 2x80GB: consistent with the card
assert gpus_per_replica(80, 2, 80) == 3            # BF16 (160 GB) needs THREE 80GB GPUs
assert gpus_per_replica(80, 2, 141) == 2           # ...but BF16 does fit 2x141GB
# A parallelism flag alone therefore tells you nothing about the GPU you need.

# --- BOUNDARY: bf16 is exactly 2x fp8; more overhead never lowers the count; no zero GPUs.
assert weights_gb(80, 2) == 2 * weights_gb(80, 1)
assert gpus_per_replica(80, 1, 80, 0.35) >= gpus_per_replica(80, 1, 80, 0.15)
assert gpus_per_replica(1, 1, 80) == 1

The checks matter more than the formula. They catch the three mistakes that make fleet budgets wrong: using active parameters for memory, pricing the BF16 checkpoint as if it were the FP8 checkpoint, and reading a published --tensor-parallel-size as if it pinned a GPU type. The GLM and Ornith checks reproduce the 8-GPU launch shapes recorded in the GLM-5.2 cookbook and the Ornith cookbook, and the max_overhead assertions show how little slack each one leaves. The GLM figure is quoted from the cheat-sheet as ~744B; its own cookbook records ~743-753B, so treat any single number as approximate.

3. Size the fleet and run the seat economics

Now convert developers into replicas into dollars. The hosted side uses the Anthropic-style five-minute prompt-cache schedule from prompt caching, which has three input tiers, not two: a cache read bills 0.1x the base input price, a five-minute cache write bills 1.25x, and input that is never cached bills the plain 1.0x. Which tier the non-hit tokens land in depends on where you place cache breakpoints, so write_frac exposes it and the block asserts the answer at both ends. Replace every multiplier for OpenAI, Gemini, or any other API. The fleet side reuses the target-utilisation and headroom conventions of GPU capacity planning and the illustrative neocloud rent of the build-vs-rent model.

# seat_economics.py -- validated: developers -> replicas -> $/seat, against a hosted API.
# EVERY constant below is an illustrative input. Replace all of them, especially the two
# throughput numbers, which you must MEASURE on your own hardware and batch size: they
# decide not just the fleet size but whether self-hosting can EVER win.
from math import ceil, inf

HOURS_PER_MONTH = 730
READ_MULT, WRITE_MULT, PLAIN_MULT = 0.10, 1.25, 1.00   # prompt-caching.md: three tiers
TARGET_UTIL, HEADROOM = 0.65, 0.20                     # gpu-capacity-planning.md conventions
RENT_GPU_HR = 2.40                                     # build-vs-rent-gpu-cost-model.md
GPUS_PER_REPLICA = 2                                   # Qwen3-Coder-Next-FP8 memory floor
TURNS, CTX, OUT = 120, 20_000, 700                     # per developer per day
WORK_H, WORK_DAYS = 8, 21
PREFILL_R, DECODE_R = 9_000.0, 900.0                   # tokens/s PER REPLICA. MEASURE THESE.

def tokens_per_dev_day(turns, ctx, out, cache_hit):
    """Every agentic turn resends the context; cache_hit is the prefix-cache fraction."""
    assert 0.0 <= cache_hit <= 1.0
    total_in = turns * ctx
    return total_in * (1 - cache_hit), total_in * cache_hit, turns * out

def replicas_needed(seats, per_dev, prefill_r=PREFILL_R, decode_r=DECODE_R):
    """Size on the PEAK working window, never on the 24-hour mean."""
    uncached_in, _cached, out = per_dev
    window_s = WORK_H * 3600
    r_pre = seats * uncached_in / window_s / prefill_r   # cached prefix skips prefill
    r_dec = seats * out / window_s / decode_r            # decode always runs
    bound = "decode" if r_dec >= r_pre else "prefill"    # ties resolve to decode
    return max(ceil(max(r_pre, r_dec) / TARGET_UTIL * (1 + HEADROOM)), 1), bound

def self_host_monthly(seats, per_dev, ops=0.0):
    """ops = staffing/on-call, amortized monthly. Default 0 to isolate the GPU term;
    the assertions below show it is NOT ignorable."""
    r, _ = replicas_needed(seats, per_dev)
    return r * GPUS_PER_REPLICA * RENT_GPU_HR * HOURS_PER_MONTH + ops

def api_monthly(seats, per_dev, base_in, p_out, write_frac=0.0):
    """write_frac = share of UNCACHED input billed as a cache write (1.25x); the rest is
    plain input (1.0x). write_frac=0 makes the API as cheap as the contract allows, so a
    self-host win asserted at write_frac=0 is the conservative one."""
    assert 0.0 <= write_frac <= 1.0
    u, c, o = per_dev
    in_mult = write_frac * WRITE_MULT + (1 - write_frac) * PLAIN_MULT
    return seats * (u * base_in * in_mult + c * base_in * READ_MULT + o * p_out) / 1e6 * WORK_DAYS

pd = tokens_per_dev_day(TURNS, CTX, OUT, cache_hit=0.80)
assert abs(pd[0] - 480_000) < 1e-6 and abs(pd[1] - 1_920_000) < 1e-6 and pd[2] == 84_000
assert round((pd[0] + pd[1]) / pd[2], 1) == 28.6         # input-heavy: 28.6:1

# --- Caching removes PREFILL work and never removes DECODE work, so the binding phase
# can move as the cache warms. At the illustrative throughput it does.
assert replicas_needed(500, tokens_per_dev_day(TURNS, CTX, OUT, 0.00)) == (9, "prefill")
assert replicas_needed(500, tokens_per_dev_day(TURNS, CTX, OUT, 0.80)) == (3, "decode")
# ADVERSARIAL: the flip is NOT a law. At 50:1 the fleet is decode-bound even cold; at 1:1
# it stays prefill-bound even at a 95% hit rate.
assert replicas_needed(500, tokens_per_dev_day(TURNS, CTX, OUT, 0.0), 30_000., 600.)[1] == "decode"
assert replicas_needed(500, tokens_per_dev_day(TURNS, CTX, OUT, 0.95), 4_000., 4_000.)[1] == "prefill"
# The documented tie-break, exercised exactly (r_pre == r_dec):
assert replicas_needed(10, (1000.0, 0.0, 1000.0), 100.0, 100.0)[1] == "decode"
# The floor is live: zero seats still means one replica, never zero.
assert replicas_needed(0, pd)[0] == 1

# --- ADVERSARIAL: sizing on the 24h mean instead of the 8h peak under-provisions.
def replicas_on_24h_mean(seats, per_dev):
    day_s = 24 * 3600
    raw = max(seats * per_dev[0] / day_s / PREFILL_R, seats * per_dev[2] / day_s / DECODE_R)
    return max(ceil(raw / TARGET_UTIL * (1 + HEADROOM)), 1)
assert replicas_needed(1000, pd)[0] == 6 and replicas_on_24h_mean(1000, pd) == 2   # 3x short
assert all(replicas_needed(s, pd)[0] >= replicas_on_24h_mean(s, pd) for s in (1, 50, 200, 3000))

# --- The idle premium. The fleet bills 730 h/month and serves 168 of them.
duty = (WORK_H * WORK_DAYS) / HOURS_PER_MONTH
assert abs(duty - 0.230) < 0.001 and abs(1 / duty - 4.35) < 0.01

# === THE CRITERION. Do not scan; solve. The fleet's MARGINAL cost per seat is fixed by
# the binding phase. If the API bills less than that, both sides grow linearly and NO seat
# count ever makes self-hosting cheaper. This decides the economics gate.
def marginal_self_host_per_seat(per_dev, prefill_r=PREFILL_R, decode_r=DECODE_R):
    window_s = WORK_H * 3600
    reps = max(per_dev[0] / window_s / prefill_r, per_dev[2] / window_s / decode_r)
    reps = reps / TARGET_UTIL * (1 + HEADROOM)                    # continuous limit, no ceil
    return reps * GPUS_PER_REPLICA * RENT_GPU_HR * HOURS_PER_MONTH

def api_per_seat(per_dev, base_in, p_out, write_frac=0.0):
    return api_monthly(1, per_dev, base_in, p_out, write_frac)

BUDGET, MID, FRONTIER = (0.30, 1.20), (1.00, 5.00), (3.00, 15.00)
marg = marginal_self_host_per_seat(pd)
assert round(marg, 2) == 20.96
assert round(api_per_seat(pd, *BUDGET), 2) == 6.35        # below marginal -> hopeless
assert round(api_per_seat(pd, *MID), 2) == 22.93          # barely above -> long staircase
assert round(api_per_seat(pd, *FRONTIER), 2) == 68.80     # far above -> quick crossover
assert api_per_seat(pd, *BUDGET) < marg < api_per_seat(pd, *MID) < api_per_seat(pd, *FRONTIER)

# --- THE CRITERION IS THROUGHPUT-DEPENDENT, and that overturns any "always" claim.
# Faster decode lowers the marginal cost and can make even a budget API beatable; slow
# decode raises it and can make even a frontier API unbeatable.
def can_ever_win(tier, prefill_r, decode_r):
    return api_per_seat(pd, *tier) > marginal_self_host_per_seat(pd, prefill_r, decode_r)
assert not can_ever_win(BUDGET, 9_000., 900.)         # illustrative: budget is hopeless
assert can_ever_win(BUDGET, 100_000., 5_000.)         # ...but NOT at high throughput
assert not can_ever_win(MID, 30_000., 600.)           # slow decode: mid is hopeless too
assert not can_ever_win(FRONTIER, 20_000., 200.)      # very slow decode: even frontier loses
assert can_ever_win(FRONTIER, 9_000., 900.)
# ADVERSARIAL: only the BINDING phase sets the marginal cost. Speeding decode 5.6x while
# leaving prefill alone just hands the bottleneck to prefill and does not rescue the budget
# tier; both phases have to move. This is the mistake the criterion is designed to expose.
assert round(marginal_self_host_per_seat(pd, 9_000., 900.), 2) == 20.96      # decode binds
assert round(marginal_self_host_per_seat(pd, 9_000., 5_000.), 2) == 11.98    # now prefill binds
assert replicas_needed(500, pd, 9_000., 5_000.)[1] == "prefill"
assert not can_ever_win(BUDGET, 9_000., 5_000.)                              # still hopeless
assert round(marginal_self_host_per_seat(pd, 100_000., 5_000.), 2) == 3.77   # both moved

# === THE STAIRCASE. When a crossover exists, it is not a single seat count: the fleet
# steps by a whole replica while the API bill rises as a line. Scan only to the proven
# bound, where no remaining step can be tall enough to flip the decision back.
def bound_from_margin(margin, step):
    """A replica step can only flip the decision back while step > margin * seats."""
    return inf if margin <= 0 else step / margin
# The degenerate tie is real: if the API bills EXACTLY the marginal cost, the fleet's
# rounding-up never catches up, so there is still no crossover.
assert bound_from_margin(0.0, 3504.0) == inf and bound_from_margin(-1.0, 3504.0) == inf
assert bound_from_margin(2.0, 3504.0) == 1752.0

def settle_bound(tier, write_frac=0.0, ops=0.0):
    margin = api_per_seat(pd, *tier, write_frac) - marginal_self_host_per_seat(pd)
    step = GPUS_PER_REPLICA * RENT_GPU_HR * HOURS_PER_MONTH + ops
    return bound_from_margin(margin, step)

def decide(self_cost, api_cost):
    return "self" if self_cost <= api_cost else "api"  # ties resolve to the fleet
assert decide(100.0, 100.0) == "self"                  # the tie-break, exercised directly

def winner(seats, tier, write_frac=0.0, ops=0.0):
    return decide(self_host_monthly(seats, pd, ops), api_monthly(seats, pd, *tier, write_frac))

def flips(tier, write_frac=0.0, ops=0.0):
    bound = settle_bound(tier, write_frac, ops)
    upto = 2000 if bound == inf else int(bound) + 2     # scan past the proven bound
    out, prev = [], None
    for s in range(1, upto + 1):
        w = winner(s, tier, write_frac, ops)
        if prev and w != prev:
            out.append((s, f"{prev}->{w}"))
        prev = w
    return out, prev

# BUDGET: no crossover exists at all, under either pricing assumption. Proven, not scanned.
assert settle_bound(BUDGET) == inf and settle_bound(BUDGET, write_frac=1.0) == inf
for wf in (0.0, 1.0):
    f, final = flips(BUDGET, wf)
    assert f == [] and final == "api"

# MID: with plain-input pricing the decision reverses 21 times before settling at 1681.
mid, mid_final = flips(MID)
assert len(mid) == 21 and mid[0] == (153, "api->self") and mid[-1] == (1681, "api->self")
assert mid_final == "self" and mid[-1][0] < settle_bound(MID)      # the bound holds
assert winner(160, MID) == "self" and winner(200, MID) == "api"    # it really oscillates

# The write premium is not a rounding detail: billing every miss as a cache write moves
# the first crossing from 153 to 138 and hides 12 of the 21 reversals.
mid_w, _ = flips(MID, write_frac=1.0)
assert mid_w[0] == (138, "api->self") and len(mid_w) == 9

# FRONTIER: one clean crossover under both pricing assumptions.
assert flips(FRONTIER)[0] == [(51, "api->self")]
assert flips(FRONTIER, write_frac=1.0)[0] == [(46, "api->self")]

# The first crossing is the minimum replica cost divided by the API's per-seat bill,
# exactly, for as long as one replica still serves the load.
min_replica = GPUS_PER_REPLICA * RENT_GPU_HR * HOURS_PER_MONTH
assert round(min_replica) == 3504
assert ceil(min_replica / api_per_seat(pd, *MID)) == 153 and replicas_needed(153, pd)[0] == 1
assert ceil(min_replica / api_per_seat(pd, *MID, 1.0)) == 138 and replicas_needed(138, pd)[0] == 1

# === STAFFING DOMINATES AT SMALL SCALE, and a GPU-only model hides it.
HALF_FTE = 200_000 / 2 / 12                            # $8,333/month, illustrative
assert round(HALF_FTE) == 8333 and HALF_FTE > 2 * min_replica   # costs more than TWO replicas
def first_self(tier, write_frac=0.0, ops=0.0):
    f, _ = flips(tier, write_frac, ops)
    wins = [s for s, t in f if t.endswith("self")]
    return wins[0] if wins else None
assert first_self(FRONTIER) == 51 and first_self(FRONTIER, ops=HALF_FTE) == 223   # 4.4x later
assert first_self(MID) == 153 and first_self(MID, ops=HALF_FTE) == 4337           # 28x later

# === $/Mtok, self-hosted (GPU rent only). Two denominators, two very different numbers.
def per_mtok_delivered(seats):
    """All tokens the developer sees, cached input included. The engine does not prefill
    the cached ones, so this is a floor, not a compute cost."""
    return self_host_monthly(seats, pd) / (seats * sum(pd) * WORK_DAYS / 1e6)
def per_mtok_computed(seats):
    """Only tokens the engine actually computes: uncached prefill plus every decoded token."""
    return self_host_monthly(seats, pd) / (seats * (pd[0] + pd[2]) * WORK_DAYS / 1e6)
assert round(per_mtok_delivered(10), 2) == 6.72          # a 2-GPU replica for 10 people
assert round(per_mtok_delivered(500), 3) == 0.403
assert round(per_mtok_computed(500), 3) == 1.775         # 4.4x higher on the honest denominator
# ADVERSARIAL: 0.403 is a SAWTOOTH MINIMUM, not the typical rate. 500 and 5000 seats both
# sit at the top of a replica's capacity band; ten more seats buy a whole replica.
assert round(per_mtok_delivered(5000), 3) == 0.403
assert round(per_mtok_delivered(510), 3) == 0.527        # +31% from ten extra developers
assert per_mtok_delivered(510) > per_mtok_delivered(500) < per_mtok_delivered(400)

Read the assertions, not the dollar signs. At the illustrative throughput the block produces four conclusions, and each one names the regime it holds in. Re-run it after changing throughput, GPU price, GPU count, cache multipliers, staffing, or API price; the sweep assertions above show that changing throughput alone can reverse the first two.

  • Whether self-hosting can ever win is decided before any seat count. Compare the API's bill per seat against the fleet's marginal cost per seat ($20.96 here). A budget tier bills $6.35 and can never be beaten at this throughput; a frontier tier bills $68.80 and is beaten quickly. Only the binding phase sets that marginal cost, which is why partial optimisation is a trap: raising decode alone from 900 to 5,000 tokens/s per replica hands the bottleneck to prefill and lands at $11.98, still hopeless against the budget tier. Move both phases (100,000 prefill, 5,000 decode) and the marginal cost falls to $3.77, at which point even a budget API becomes beatable. Measure both before you conclude.
  • When a crossover exists, it is not one number. Between 153 and 1681 seats the mid-tier decision reverses 21 times, because each new replica is an indivisible step. A claim of the form "self-hosting pays off above N developers" is describing one point on a staircase.
  • The first crossing is the minimum replica cost divided by the API bill per seat, exactly, while one replica still serves the load. Two GPUs you cannot subdivide is the real minimum ticket, and at $2.40/GPU-hour that is $3,504 a month before anyone is paid.
  • Staffing dominates at small scale. Half an engineer at $200k/year is $8,333 a month, more than two replicas of GPU rent. Adding it moves the frontier crossover from 51 to 223 seats and the mid-tier crossover from 153 to 4,337. A GPU-only TCO is not a TCO.

Two more traps the assertions pin down. The $0.403/Mtok figure is a sawtooth minimum, not a typical rate: ten extra developers past 500 seats push it to $0.527. And it counts cached input tokens the engine never prefills; on the tokens the engine actually computes, the same fleet costs $1.775/Mtok. Never compare the delivered-token rate to an API list price. Finally, a fleet sized for an eight-hour working window serves 23% of the hours it bills, so every served token carries a 4.35x idle multiplier unless you backfill overnight.

4. Build the internal eval before you migrate

Public coding benchmarks cannot decide this for you. SWE-Bench+ manually screened successful SWE-Agent plus GPT-4 patches on SWE-bench and found that 32.67% of them had the solution already present in the issue report or its comments, and it flagged a further 31.08% as suspicious because the test cases were weak. Filtering those out dropped the reported resolution rate from 12.47% to 3.97%. Over 94% of the issues predate the models' knowledge cutoffs. The paper reports the same data-quality problems in SWE-bench Lite and SWE-bench Verified.

So build a paired eval on your own repositories, and size it so it can see the difference you care about.

# eval_gate.py -- validated: can your internal eval detect the difference you care about?
# Paired McNemar exact test, sample sizing, and a contamination model. stdlib only.
from math import ceil, comb, sqrt

Z = 1.959963984540054   # two-sided 95%

def mcnemar_exact_p(b, c):
    """b = tasks only the candidate passes; c = tasks only the incumbent passes.
    Tasks both models pass, or both fail, carry no information about which is better."""
    n = b + c
    if n == 0:
        return 1.0                                   # no discordant evidence at all
    k = max(b, c)
    return min(1.0, 2 * sum(comb(n, i) for i in range(k, n + 1)) / 2 ** n)

def required_tasks(delta, discordance, z=Z):
    """Closed form from z = (b-c)/sqrt(b+c): detectable delta = z*sqrt(discordance/n)."""
    assert 0 < delta <= 1 and 0 < discordance <= 1
    return ceil((z / delta) ** 2 * discordance)

def p_at(n, delta=0.05, d=0.25):
    """The exact p for a paired eval of n tasks showing a `delta` gap at discordance `d`."""
    nd, net = int(n * d), round(delta * n)
    b = (nd + net) // 2
    return mcnemar_exact_p(b, nd - b)

# --- The trap: the SAME +5pp effect is uncallable at n=100 and decisive at n=1000.
assert round(mcnemar_exact_p(15, 10), 3) == 0.424        # 100 tasks, 60 -> 65 pass
assert mcnemar_exact_p(150, 100) < 0.002                 # 1000 tasks, same +5pp
assert mcnemar_exact_p(15, 10) > 0.05 > mcnemar_exact_p(150, 100)

# --- DISCREPANCY 1: the closed form is optimistic. It says 385 paired tasks suffice for a
# 5pp gap at 25% discordance; the exact test at n=385 returns p = 0.082, not significant.
assert required_tasks(0.05, 0.25) == 385
assert round(p_at(385), 3) == 0.082 > 0.05

# --- DISCREPANCY 2: "the first n that clears" is a sawtooth artifact, and the obvious
# remedy fails. n=412 is the first n with p < 0.05, but p(n) is NOT monotone in n, because
# b and c are integers. Adding the tempting 10% lands on n=424, which does NOT clear.
assert p_at(412) < 0.05                                  # first crossing
assert p_at(424) > 0.05 and round(p_at(424), 3) == 0.064 # 385 * 1.10 -> still not significant
assert p_at(420) > 0.05 and p_at(416) > 0.05             # neither do these
assert p_at(432) < 0.05                                  # ...but 432 clears again
assert sum(1 for n in range(380, 460) if p_at(n + 1) > p_at(n)) == 8   # sawtooth, not a curve

# --- The checked answer for this planning window: 468 clears every n through 1199, i.e.
# the closed form plus 22%. Below it, whether you clear depends on integer luck.
STABLE = 468
assert all(p_at(n) < 0.05 for n in range(STABLE, 1200))
assert any(p_at(n) >= 0.05 for n in range(412, STABLE))  # the sawtooth region really is unsafe
assert round(STABLE / required_tasks(0.05, 0.25), 2) == 1.22
# Always run the exact test on the b and c you actually observe.

# --- ADVERSARIAL boundaries: no crash, no false confidence, and the tails are pinned.
assert mcnemar_exact_p(0, 0) == 1.0                      # zero discordant pairs
assert mcnemar_exact_p(10, 0) == 2 / 2 ** 10             # pins the endpoint: 0.001953125
assert mcnemar_exact_p(1, 0) == 1.0                      # one discordant pair proves nothing
assert mcnemar_exact_p(5, 5) == 1.0                      # a perfect split proves nothing
assert mcnemar_exact_p(3, 7) == mcnemar_exact_p(7, 3)    # symmetric in b and c
assert mcnemar_exact_p(20, 5) < mcnemar_exact_p(15, 10)  # wider margin, same discordant count

# --- What a small internal eval can actually see.
def min_detectable_pp(n, discordance=0.25):
    return Z * sqrt(discordance / n) * 100
assert round(min_detectable_pp(30), 1) == 17.9           # a 30-task eval is nearly blind
assert round(min_detectable_pp(100), 1) == 9.8
assert round(min_detectable_pp(1000), 1) == 3.1

# --- CONTAMINATION: why a leaked benchmark can invert the ranking.
def measured_pass(true_skill, leak_fraction, leak_pass=1.0):
    """Leaked tasks are recited, not solved."""
    assert 0 <= leak_fraction <= 1
    return leak_fraction * leak_pass + (1 - leak_fraction) * true_skill

a_true, b_true = 0.50, 0.60                              # B is genuinely the better model
a_meas = measured_pass(a_true, leak_fraction=0.30)       # weaker model, 30% leaked
b_meas = measured_pass(b_true, leak_fraction=0.00)       # stronger model, clean
assert abs(a_meas - 0.65) < 1e-9 and abs(b_meas - 0.60) < 1e-9
assert a_true < b_true and a_meas > b_meas               # the benchmark INVERTS the ranking
assert measured_pass(0.5, 0.0) == 0.5                    # zero leakage is the identity
assert measured_pass(0.1247, 0.30) > 0.1247              # de-leaking always lowers the score
assert all(measured_pass(0.5, l) < measured_pass(0.5, l + 0.1) for l in (0.0, 0.2, 0.5))

The contamination assertions are the whole argument for an internal eval. A model that memorised 30% of the benchmark measures 65% while genuinely solving 50%, and it outranks a clean model that genuinely solves 60%. Select on a contaminated public benchmark and you may deploy the worse model.

The sizing assertions are the argument for a large internal eval, and they contain a warning about sizing itself. The closed-form required_tasks is optimistic by about 22%, and because pass counts are integers the exact p-value zig-zags with n: 412 clears, 416, 420 and 424 do not, 432 does. Do not size to "the first n that clears". Size at or above 468 tasks, the point from which every n clears for the rest of the checked window; the block verifies n = 468 through 1199, and extending the same check to 29999 finds no failure either. Then run the exact test on the b and c you actually observe. At that discordance a 30-task eval cannot see anything under 18 points, which is larger than the gap between most adjacent models; measure your own discordance rate, since a lower one sharpens the test. Gate promotions with the floors-plus-no-regression pattern in the LLM evaluation harness, score agent trajectories with agent evaluation, and regenerate tasks from your own pull requests the way SWE-Bench++ does at scale.

5. Keep a frontier escape hatch

Do not route everything to the owned fleet. Send the hardest tasks and the overflow to a frontier API, and record which tasks went where. This bounds the capability gate, gives an honest ongoing comparison, and absorbs peak load that would otherwise size the fleet. Wire it in the gateway, not in each developer's harness:

# Reference template, not executed. A gateway routing rule; see llm-request-routing.md.
routes:
  - name: owned-coder
    match: {task_class: [edit, explain, test-gen, refactor]}
    backend: http://vllm-coder.svc/v1        # the owned fleet
    fallback: frontier-api                   # on 429, 5xx, or admission rejection
  - name: frontier-api
    match: {task_class: [architect, debug-hard], escalated: true}
    backend: https://api.vendor.example/v1
budgets:
  frontier-api: {monthly_usd: 5000, on_exhaustion: reject}

How to develop with it

Adapt in this order, and stop as soon as the eval clears. Each rung is materially more expensive than the one before it.

  1. Fix the harness first. The evidence that this dominates a model upgrade is strong and repeated. SWE-agent reported that a purpose-built agent-computer interface significantly enhances a model's ability to create and edit files, navigate a repository, and run tests, reaching 12.5% pass@1 on SWE-bench at the time. Agentic Harness Engineering evolved the harness alone over ten iterations and took Terminal-Bench 2 from 69.7% to 77.0%, then froze it and transferred it without re-evolution: on Terminal-Bench 2 it gained 5.1 to 10.1 points across three other model families, and on SWE-bench-verified it led on aggregate success at 12% fewer tokens than the seed harness. HarnessX reports an average gain of 14.5% (up to 44.0%) across five benchmarks, with gains largest where baselines are lowest on four of the five; WebShop is the exception in the paper's own results table, where the lowest-baseline model gains the least. See harness architecture and self-improving harnesses.
  2. Fix the context. Retrieval over your repository, house conventions in the system prompt, and a warm prefix cache. This is where the token bill actually lives, given the 28.6-to-1 input ratio. See RAG vs CAG and prompt caching.
  3. Then, and only then, adapt the weights. LoRA on your own merged pull requests teaches house style and internal APIs, not general coding ability. Serve per-team adapters on one base with multi-LoRA serving rather than forking the fleet. The recipes are in SFT and LoRA.

Note that a model trained with self-scaffolding methods does not ship its scaffold: the released weights behave as an ordinary chat model and the harness remains yours to build, as the Ornith cookbook records.

How to run it in production

  • Serve it. Take the launch shape from the model's cookbook (GLM-5.2, GLM-4.7, Ornith-1.0, Qwen3-235B, Kimi K2) and the deployment mechanics from the vLLM inference deployment recipe. Prefix caching is the highest-value setting for a coding fleet, and in current vLLM it is already on: enable_prefix_caching defaults to True in vllm/config/cache.py, so the job is to confirm nothing disabled it and to watch the hit rate, not to pass a flag.
  • Isolate the cache. A shared prefix cache across teams is a confidentiality boundary and a fairness boundary at once. Salt cache keys per tenant and reserve KV partitions, per tenant cache isolation.
  • Admit and shed. N developers share one endpoint. Give interactive completions priority over background agent loops and shed the least important traffic rather than degrading everyone, per inference QoS and admission control. Set TTFT and TPOT targets from inference serving SLOs.
  • Sandbox the agent. A coding agent runs model-generated code with file and shell access. Container defaults are not sufficient for untrusted code; see agent sandboxing and isolation and prompt-injection defense, because a poisoned dependency or a hostile issue comment is now an execution path into your build.
  • Instrument the seat. Export per-developer tokens, cache hit rate, escalation rate to the frontier hatch, and acceptance rate of the agent's diffs, via GenAI observability. Acceptance rate is the only one of these that measures value; the rest measure cost.
  • When agent loops thrash the KV cache, move to program-aware scheduling (ThunderAgent).

How to maintain it

Re-run the whole decision on a schedule, because every input moves. Model capability moves monthly, API prices fall, throughput changes with the engine, and seat count grows.

  • Every release of a candidate model, run the paired eval against the incumbent and promote only on a significant, non-regressing win. Record the exact p-value, not a vibe.
  • Every quarter, re-measure per-replica prefill and decode throughput, recompute the marginal cost per seat, and re-run the criterion against the current API price tier. A price cut on the hosted side, or an engine regression on yours, can move you back across the staircase.
  • Whenever seat count changes materially, re-check the replica count. Growth crosses staircase steps, and a step you cross costs a whole replica.
  • Pin and stage. Pin the engine version, the model revision, and the harness. Stage upgrades behind the eval gate; an engine upgrade can change throughput enough to move the bottleneck between prefill and decode, which changes the fleet size and, through the marginal cost, the economics gate itself.
  • Re-baseline the eval as tasks leak into training corpora. Regenerate from recent merged pull requests rather than reusing a fixed set forever.

Failure modes

  • Sizing an MoE model on active parameters. The weights do not fit and the server fails to load. Memory follows total parameters. The replica_fit.py assertion covers exactly this.
  • Reading a parallelism flag as a hardware requirement. A published --tensor-parallel-size 2 fixes neither the GPU nor the precision. The same 80B model needs two 80 GB GPUs at FP8 and three at BF16.
  • Trusting a published 8-GPU shape without its overhead budget. GLM-5.2 on an 8xH200 node leaves at most 34% of HBM for activations and KV; Ornith leaves 38%. Raise your KV budget past that and the shape silently needs another GPU.
  • Quoting a single break-even seat count. The fleet cost is a staircase and the API bill is a line; in the middle price band the decision reverses 21 times between 153 and 1681 seats. Quote the price tier, the throughput, and the seat count together, or quote nothing.
  • Concluding "self-hosting never pays" (or "always pays") from one throughput. The criterion is the API bill per seat against the fleet's marginal cost per seat, and that marginal cost is set by whichever phase binds. Speed both phases up and even a budget API becomes beatable; slow either one down and even a frontier API becomes unbeatable.
  • Optimising the phase that is not binding. Raising decode alone from 900 to 5,000 tokens/s per replica hands the bottleneck to prefill and stops at $11.98 per seat, still above the budget tier's $6.35 bill. Find the binding phase first, then spend.
  • Billing the API side wrong. Charging every uncached token at the 1.25x cache-write premium, when most of it is plain 1.0x input, overstates the hosted bill and flatters self-hosting. Here it moves the first crossing from 153 seats to 138 and hides 12 of the 21 reversals.
  • A GPU-only TCO. Half an engineer costs more per month than two replicas of GPU rent, and moves the frontier crossover from 51 to 223 seats. If gate 4 says you need a pager, gate 2 has to pay for it.
  • Sizing on the daily mean. Developers work in an eight-hour window. A fleet sized on the 24-hour average under-provisions by up to 3x and misses its TTFT SLO exactly when everyone is working.
  • Forgetting the idle premium. The fleet bills 730 hours and serves 168. Any $/token computed at full utilisation understates the true cost by roughly 4.35x unless the fleet is backfilled overnight.
  • Quoting the delivered-token rate as the cost of compute. $0.403/Mtok counts cached input the engine never prefills, and it is a sawtooth minimum besides: ten developers past 500 seats make it $0.527. On tokens actually computed the same fleet costs $1.775/Mtok.
  • Selecting a model on a public benchmark. With 32.67% solution leakage measured on SWE-bench, a contaminated benchmark can rank a genuinely weaker model above a stronger one. The eval_gate.py inversion demonstrates it.
  • Sizing an eval to "the first n that clears". The exact p-value zig-zags with n: 412 clears, 424 does not. Use a checked sizing window and test the b and c you observe.
  • Shipping on a 30-task eval. At 25% discordance it cannot resolve anything below an 18-point difference. Every "the new model feels better" decision made on such a set is noise.
  • Buying a bigger model to fix a harness problem. Above the capability floor the harness moves the number more than the model does, at a fraction of the cost, and the gain transfers across model families.
  • No pager, no SLO. An owned fleet without on-call degrades silently, and developers quietly route around it back to a hosted product on a personal card, which reintroduces the residency problem you paid to solve.

References

Related: Cookbook: own and run an open-weight coding model · Serving open-weight models · Running local coding agents · Build-vs-rent GPU cost model · GPU capacity planning · Agent harness architecture · LLM evaluation harness · Agent evaluation · Prompt caching · Tenant cache isolation · Inference QoS and admission control · Multi-LoRA serving · SFT and LoRA · LLM request routing · Agent sandboxing and isolation · Glossary