Skip to content
Markdown

Colibri (colibrì): GLM-5.2 local inference on 25 GB of RAM

Scope: colibri (JustVugg/colibri), a single-file, zero-dependency C inference engine that runs exactly one checkpoint, Z.ai's GLM-5.2 (a roughly 744B-parameter Mixture-of-Experts model), on a consumer machine with about 25 GB of RAM by keeping the dense skeleton resident at int4 and streaming the routed experts from local NVMe with a per-layer LRU cache. This page covers the memory-hierarchy design, the sigmoid MoE router, native MTP speculative decoding and its documented acceptance and determinism caveats, and the project's actual maturity as of the commit pinned below, thirteen days after repo creation with same-day commits still landing. It is the counterpart to DwarfStar (ds4): DeepSeek V4 local inference (same shape of problem, a different model family and quantization mix) and the small-box end of the spectrum from serving GLM-5.2 with vLLM (the datacenter, multi-GPU, multi-tenant deployment of the same model family). It extends open-weight serving and quantization for inference.

Shell commands below are reference templates pinned to commit 37ffb61 (2026-07-14T14:13:19Z) of the JustVugg/colibri repository; verify flags against the commit you actually build. As of that commit the repository is 13 days old (created 2026-07-01), has 11,704 stars, 928 forks, 28 open issues, is Apache-2.0 licensed, and ships no tagged releases (gh api repos/JustVugg/colibri/releases and .../tags both return empty). Commits are landing same day, including crash and memory-safety hardening (see "How to maintain it"). Treat every performance number here as the repository's or a community contributor's own single-run measurement, not independently reproduced. The numpy block is self-contained, executed, and asserted on this page; it validates colibri's sigmoid MoE router mechanism, matching how DwarfStar (ds4) handles the same constraint for a similar project. Separately, on 2026-07-17 this page's authoring host (18 cores, 94 GiB RAM, no GPU, far too little free disk for the roughly 384 GB container) built the engine for real at the pinned commit, ran the repository's own teacher-forcing self-test to its expected 32/32 positions, and executed one real coli serve round trip (health probe, one completion, auth rejection) against the repo's 2.4 MB random-weight oracle model; the actual 744B GLM-5.2 checkpoint was neither downloaded nor run here, so every full-model number remains the repository's or a community contributor's own. Upstream was re-checked the same day: still no tags or releases, and main (72d3d372, 2026-07-16) sits 117 commits ahead of the pin, four of them security fixes (see "How to maintain it").

What it is

Colibri is a single C file, c/glm.c (238,244 bytes on disk at the pinned commit; the README describes it as "~2,400 lines," but a direct line count on that file at the same commit is 4,212 lines, 4,121 non-blank, so treat the README's line-count figure as stale rather than as this page's own measurement), plus small headers, with zero runtime dependencies: no BLAS, no Python at runtime, no GPU required. It is not a general GGUF or safetensors runner; it loads exactly one model family, GLM-5.2, converted into its own int4 container. Python appears only in the one-time offline FP8-to-int4 converter. Colibri's README states GLM-5.2 is "744B-parameter" with about 40B active per token; the vLLM serving page for the same checkpoint cites the vLLM recipe's roughly 743B total, roughly 39B active, with the Hugging Face safetensors metadata totaling 753B. Treat the small spread across these independent sources as a rounding or counting-convention gap, not something this page resolves.

The engine treats VRAM, RAM, and disk as one managed memory hierarchy, split by how often each part of the model changes per token:

  • The dense skeleton (attention, shared expert, embeddings, router, roughly 17B parameters) stays resident in RAM at int4, about 9.9 GB.
  • The 21,504 routed experts ("75 MoE layers x 256 experts + the MTP head, ~19 MB each at int4" per the README) live on disk as a roughly 370 GB int4 container and stream on demand, backed by a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free extra layer. The README's own arithmetic here does not fully reconcile: 75 x 256 is 19,200, not 21,504, while 84 x 256 is exactly 21,504; this page quotes the source's stated figures without resolving the discrepancy, per this KB's policy of flagging rather than silently picking a side when a primary source is internally inconsistent.

Two mechanisms carry most of the design. MLA attention with a compressed KV cache stores 576 floats per token instead of 32,768 uncompressed (a 57x reduction, since GLM-5.2 has 64 attention heads and no grouped-query attention), with weight absorption at decode time so the query absorbs kv_b and context is reconstructed after attention rather than materializing per-token keys and values. A DeepSeek-V3-style sigmoid router ("noaux_tc", a routed_scaling_factor, a shared expert, and the first three layers left dense) selects the top-k experts per token; the auxiliary-loss-free bias pattern it follows traces to the DeepSeek-V3 technical report. GLM-5.2's own lightning indexer (DSA, Dynamic Sparse Attention) additionally restricts causal key selection to the top 2,048 keys per query on most layers.

Colibri also implements native MTP (multi-token prediction) speculative decoding using GLM-5.2's own draft head at layer 78, grammar-forced speculative drafts for constrained JSON/NDJSON output, int8/int4/int2 AVX2 integer-dot kernels with per-row scales, a byte-level BPE tokenizer, and a single always-on coli serve process exposing an OpenAI-compatible HTTP API. It runs on Linux, WSL2, macOS (with an opt-in Metal backend), and native Windows 11 via MinGW-w64 (no WSL required, since pull request #131), plus an opt-in CUDA backend for pinning hot experts in VRAM.

The offline converter (c/tools/convert_fp8_to_int4.py) downloads the official 756 GB GLM-5.2-FP8 checkpoint one shard at a time (about 5 GB each), dequantizes, requantizes into colibri's container, and deletes the shard, so the full FP8 checkpoint never has to exist on disk at once; it is resumable and needs roughly 400 GB of free space for the int4 output.

Why use it

  • A 744B frontier-class MoE on hardware that costs less than a server fan. The README's own framing: this is "a 744B frontier-class model answering correctly on a machine that costs less than one H100 fan." The dev machine it was built and measured on is a 12-core WSL2 box with 25 GB of RAM.
  • RAM is a speed dial by design, not a correctness dial. "Insufficient fast memory may reduce speed, but the default policy never silently changes model precision or router semantics." The --policy quality and --policy balanced modes preserve checkpoint quantization and router decisions unless you explicitly pass --topk or --topp, and those lossy overrides print a warning before proceeding. The expert cache auto-sizes from MemAvailable at startup specifically so the kernel OOM-killer does not fire.
  • It gets faster the more you use it. A .coli_usage histogram records which experts your real workload routes to and pins the hottest ones in spare RAM at the next startup; --repin N additionally lets a decaying session heat map replace cold pinned experts with hot streamed ones at safe turn boundaries.
  • Sessions resume with zero re-prefill. coli chat appends the compressed MLA KV cache to .coli_kv after every turn (about 182 KB/token, crash-safe); closing and reopening a conversation the next day still has the full history, validated byte-identical to an uninterrupted session.
  • A single engineer's project you can actually read. The whole inference path is one file with no framework underneath it, which is also why its correctness caveats (next section) are unusually explicit compared to a project built on a larger dependency stack.

When to use it (and when not)

Use it when:

  • You want to run GLM-5.2 privately on a single consumer machine (25 GB RAM is the proven floor; the project explicitly asks for more RAM, cores, and faster NVMe as datapoints) and you accept sub-1 to a few tokens/second depending on disk and RAM, not interactive-chat-service latency.
  • You want to inspect or modify a small, dependency-free reference implementation of MLA attention, a sigmoid MoE router, and MTP speculative decoding rather than reading a framework's abstraction layers.
  • You can tolerate the documented non-determinism: greedy output is not guaranteed byte-identical across kernel families, batch sizes, or GPU tiers.

Do not use it when:

  • You need throughput or multi-tenant serving. coli serve keeps one model process and one mutable KV context; concurrent HTTP requests queue behind a bounded FIFO (default depth 8) rather than being batched. For a datacenter GLM-5.2 deployment with tensor parallelism and real concurrency, use vLLM's GLM-5.2 recipe instead.
  • You need a stable, versioned dependency. There are no tagged releases; the pinned commit above had crash-path and memory-safety fixes land the same day this page was written.
  • You need bit-exact reproducibility without extra flags. MTP, the CUDA expert tier, and batched prefill each round slightly differently from the single-token CPU path, and the project's own issue tracker documents this flipping greedy output on 3 of 5 test prompts with MTP and 2 of 5 with the CUDA tier, with zero speculation in the MTP case. DRAFT=0 removes speculation; add IDOT=0 COLI_CUDA=0 for full kernel-family and GPU independence.
  • You need arbitrary model support. Like ds4, this is a one-model engine; it does not load arbitrary GGUF or safetensors checkpoints, only its own converted GLM-5.2 container.

Architecture

flowchart TB
  subgraph DISK["Disk: int4 container, ~370 GB"]
    EXP["21,504 routed experts<br/>~19 MB each, int4"]
    KVD[".coli_kv<br/>compressed MLA KV, crash-safe append"]
  end
  subgraph RAM["Resident RAM: int4 dense skeleton, ~9.9 GB"]
    ATT["MLA attention + shared expert<br/>+ embeddings (~17B params)"]
    RTR["Sigmoid router (noaux_tc)<br/>top-k expert selection"]
    EC["Per-layer LRU expert cache<br/>+ learning pin (.coli_usage)"]
    MTP["MTP head, layer 78<br/>must be int8 or acceptance collapses"]
    KV["Live compressed KV<br/>576 floats/token"]
  end
  subgraph OPT["Optional pinned tiers (opt-in)"]
    CUDA["CUDA: resident + hot-pinned experts"]
    METAL["Metal: batched expert SwiGLU + fused attention"]
  end
  RTR -->|"top-8 expert ids"| EC
  EC -->|"cache miss: read from disk"| EXP
  EC -.->|"hottest experts"| CUDA
  EC -.->|"hottest experts"| METAL
  MTP -->|"draft tokens, verified in one batched forward"| RTR
  KV <-->|"append every turn / resume at startup, zero re-prefill"| KVD
  CLI["coli chat / coli run"] --> RTR
  SRV["coli serve<br/>OpenAI-compatible, FIFO queue, kv-slots"] --> RTR

The design bet mirrors ds4's: precision follows load-bearing structure. Every token passes through attention, the router, and the shared expert, so those stay resident at int4 with the rest of the dense skeleton; each token only touches 8 of 256 routed experts per layer, so those absorb the streaming cost. The difference from ds4 is where the correctness edge sits: ds4's risk is 2-bit expert quantization behind an 8-bit skeleton, while colibri keeps everything at int4/int8 but documents a distinct hazard, that its integer kernels are shape-dependent, so a batched or GPU forward can round differently from the single-token CPU path and flip an argmax tie near the router's decision boundary.

How it works (validated core mechanics)

The mechanism validated below is the sigmoid MoE router: given per-token logits over 256 experts and a small per-expert bias (the auxiliary-loss-free load-balancing term colibri's README attributes to the DeepSeek-V3-style "noaux_tc" pattern), select the top-8 experts and renormalize their raw sigmoid affinities into gate weights. A second, independently-written brute-force implementation validates the fast path, and three adversarial cases probe the boundary this router shares with colibri's own documented non-determinism: an exact tie at the cutoff, a rounding-scale perturbation at a near-tie (the same class of instability the project's issue tracker reports for batched and GPU forwards), and a corrupted (non-finite) input that must be rejected rather than silently ranked.

import numpy as np


def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-x))


def route_topk(logits, bias, top_k):
    """DeepSeek-V3/GLM-5.2-style sigmoid router ('noaux_tc'): the bias is an
    auxiliary-loss-free load-balancing term added only to RANK experts for
    top-k selection. The combination weight uses the raw (unbiased) sigmoid
    affinity, renormalized over the selected set and scaled by
    routed_scaling_factor downstream (omitted here: it is a constant
    post-multiply and does not affect which experts are chosen).
    Fast path: argpartition for the top-k cut, then a full order on just
    those k candidates with a deterministic tie-break (lower expert id wins
    a tied score). Raises on non-finite input instead of ranking garbage."""
    if not np.all(np.isfinite(logits)) or not np.all(np.isfinite(bias)):
        raise ValueError("non-finite router logits/bias: corrupted expert metadata")
    gate = sigmoid(logits)
    rank_score = gate + bias
    n, E = logits.shape
    assert 1 <= top_k <= E
    cut = np.argpartition(-rank_score, top_k - 1, axis=1)[:, :top_k]
    ordered = np.empty_like(cut)
    for i in range(n):
        ordered[i] = sorted(cut[i], key=lambda e: (-rank_score[i, e], e))
    w = np.take_along_axis(gate, ordered, axis=1)
    w = w / w.sum(axis=1, keepdims=True)
    return ordered, w


def route_topk_bruteforce(logits, bias, top_k):
    """Reference: full sort of every expert per token, same tie-break rule,
    no argpartition shortcut. Independent code path for the equivalence check."""
    if not np.all(np.isfinite(logits)) or not np.all(np.isfinite(bias)):
        raise ValueError("non-finite router logits/bias: corrupted expert metadata")
    gate = sigmoid(logits)
    rank_score = gate + bias
    n, E = logits.shape
    ordered = np.empty((n, top_k), dtype=np.int64)
    for i in range(n):
        full_order = sorted(range(E), key=lambda e: (-rank_score[i, e], e))
        ordered[i] = full_order[:top_k]
    w = np.take_along_axis(gate, ordered, axis=1)
    w = w / w.sum(axis=1, keepdims=True)
    return ordered, w


# --- equivalence: fast (argpartition) path must match the brute-force path ---
rng = np.random.default_rng(5)
n, E, top_k = 64, 256, 8   # one GLM-5.2 layer: 256 routed experts, top-8
logits = rng.normal(size=(n, E))
bias = rng.normal(scale=0.05, size=E)   # small load-balancing bias, shared across tokens

fast_idx, fast_w = route_topk(logits, bias, top_k)
ref_idx, ref_w = route_topk_bruteforce(logits, bias, top_k)
assert np.array_equal(fast_idx, ref_idx), "argpartition path disagrees with brute force"
assert np.allclose(fast_w, ref_w), "gate weights disagree between the two code paths"
assert np.allclose(fast_w.sum(axis=1), 1.0), "renormalized gate weights must sum to 1"
print(f"equivalence: {n} tokens x {E} experts, top-{top_k}: fast path == brute force. "
      f"OK ({n}/{n} tokens matched)")

# --- adversarial 1: exact tie at the top-k cutoff must be broken deterministically ---
tied_logits = np.zeros((1, 8))
tied_bias = np.zeros(8)
# experts 3 and 4 tie exactly for the last (8th vs 9th... here just the top-4) slot
tied_logits[0, :4] = 1.0     # four clear winners
tied_logits[0, 4] = 0.3      # exact tie for the last spot
tied_logits[0, 5] = 0.3      # exact tie for the last spot
tied_logits[0, 6:] = -1.0
idx_a, _ = route_topk(tied_logits, tied_bias, top_k=5)
idx_b, _ = route_topk_bruteforce(tied_logits, tied_bias, top_k=5)
assert np.array_equal(idx_a, idx_b), "tie-break must be identical across code paths"
assert list(idx_a[0][-1:]) == [4], "tie-break must pick the LOWER expert id (4, not 5)"
print(f"tie-break: experts 4 and 5 tied exactly, deterministic rule picked "
      f"expert {int(idx_a[0][-1])} (lower id). Selected set: {sorted(idx_a[0].tolist())}")

# --- adversarial 2: a rounding-scale nudge at a near-tie flips the winner ---
# This is the numeric root cause the colibri README documents (issue #100): a
# batched/kernel-family forward rounds slightly differently from the single-token
# path, and int4 GLM-5.2 sits close enough to argmax ties that the rounding change
# flips a token. Reproduced here as a bias perturbation far smaller than any
# score gap elsewhere in the row.
near_tie = np.zeros((1, 8))
near_tie[0, :4] = 1.0
near_tie[0, 4] = 0.30000
near_tie[0, 5] = 0.29999   # 1e-5 below expert 4: same "near tie" scale as int4 rounding noise
near_tie[0, 6:] = -1.0
before, _ = route_topk(near_tie, tied_bias, top_k=5)
nudged = near_tie.copy()
nudged[0, 5] += 2e-5       # a rounding-magnitude nudge, smaller than any other gap in the row
after, _ = route_topk(nudged, tied_bias, top_k=5)
assert before[0][-1] == 4 and after[0][-1] == 5, "the nudge must flip the 5th selected expert"
print(f"near-tie sensitivity: a 2e-5 nudge (rounding-noise scale) at a near-tied boundary "
      f"flips the 5th selected expert from {int(before[0][-1])} to {int(after[0][-1])} "
      f"-- this is the same class of instability behind the engine's own documented "
      f"non-determinism (greedy output differs across batched/kernel-family forwards).")

# --- adversarial 3: corrupted / non-finite router input must be rejected, not ranked ---
corrupt = logits.copy()
corrupt[3, 17] = np.nan   # simulate a corrupted or partially-read expert-bias file
try:
    route_topk(corrupt, bias, top_k)
    raised = False
except ValueError as e:
    raised = True
    msg = str(e)
assert raised, "a NaN in router logits must raise, not silently sort into a wrong-but-plausible result"
print(f"corruption check: NaN router logit at token 3 raised ValueError('{msg}') "
      f"instead of silently ranking (Python's sort with NaN present is inconsistent "
      f"and would otherwise produce a nondeterministic, unflagged wrong selection).")

print("colibri sigmoid router: all asserts passed")

Executed output:

equivalence: 64 tokens x 256 experts, top-8: fast path == brute force. OK (64/64 tokens matched)
tie-break: experts 4 and 5 tied exactly, deterministic rule picked expert 4 (lower id). Selected set: [0, 1, 2, 3, 4]
near-tie sensitivity: a 2e-5 nudge (rounding-noise scale) at a near-tied boundary flips the 5th selected expert from 4 to 5 -- this is the same class of instability behind the engine's own documented non-determinism (greedy output differs across batched/kernel-family forwards).
corruption check: NaN router logit at token 3 raised ValueError('non-finite router logits/bias: corrupted expert metadata') instead of silently ranking (Python's sort with NaN present is inconsistent and would otherwise produce a nondeterministic, unflagged wrong selection).
colibri sigmoid router: all asserts passed

The point the second adversarial case makes is not that colibri's real router has a bug: it is that any top-k router run through mixed-precision, shape-dependent kernels sits near ties often enough that a rounding difference of this magnitude changes the selected expert set, which is exactly the mechanism colibri's own maintainers point to for issue #100 (MTP and the CUDA tier both changing greedy output with zero speculation involved). The third case is a defensive property this toy implementation adds, not a claim about what colibri's C code does on a corrupted disk read; it demonstrates why a real streaming-from-disk engine needs an explicit non-finite check rather than trusting sort semantics on NaN.

How to use it

Build the engine (executed here)

git clone https://github.com/JustVugg/colibri && cd colibri
git checkout 37ffb616747bd77ea6e0b28fd8e1147e88882162   # pinned commit; no tagged release exists
cd c && ./setup.sh                # checks gcc/OpenMP, builds (make glm ARCH=native), self-tests

This build was executed on 2026-07-17 on this page's authoring host (18 cores, 94 GiB RAM, no GPU, gcc 13, Linux). setup.sh output, elided to the load-bearing lines:

  gcc: 13 · 18 core
  OpenMP: ok
  building (ARCH=native)…
glm.c: In function ‘hwinfo_emit’:
glm.c:3526:10: warning: unused variable ‘c’ [-Wunused-variable]
  RAM: 99 GB   (more RAM = more cached experts = faster inference)

The single warning is expected at this exact commit (upstream silenced it the same afternoon in e12a4295, after the pin) and doubles as a checksum that you built the pinned tree. The result is a 252,872-byte glm binary. The engine has no --version flag; run bare or with --help it prints only its usage line, SNAP=<dir>, after an [OMP] hot-thread tuning: re-exec once notice (both behaviors captured here). Version identity therefore comes from git rev-parse HEAD of the checkout, one more consequence of the no-releases posture. setup.sh skips its self-test step when the tiny oracle model does not exist yet; the test is run explicitly below.

Get the model (names and sizes verified upstream; not downloaded here)

Three Hugging Face repositories matter. All three names were verified to exist against the HF API (https://huggingface.co/api/models/<org>/<name>) on 2026-07-17, with total sizes computed from the API's per-file blob metadata the same day:

Repository Size (HF API, 2026-07-17) Role
mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp 383.8 GB, 150 files pre-converted int4 container with int8 MTP heads: the one to use
jlnsrk/GLM-5.2-colibri-int4 378.8 GB, 150 files original mirror, int4 MTP heads: 0% draft acceptance (issues #8, #102)
zai-org/GLM-5.2-FP8 755.7 GB, 141 weight shards official FP8 checkpoint, the converter's default --repo source

Both mirrors ship the same layout the engine expects: 141 main shards (out-00000.safetensors through out-00140.safetensors), 3 MTP shards (out-mtp-00000/1/2.safetensors), plus config.json, generation_config.json, tokenizer.json, and tokenizer_config.json. tokenizer.json is a hard gate: coli exits with tokenizer.json is missing from <dir> before launching the engine (check at c/coli lines 138-139; glm.c loads it again itself). The MTP-head byte sizes the README tells you to verify are exactly what the HF API reports for each mirror (re-verified 2026-07-17): int8, correct, is 3527131672 / 5366238584 / 1065950496; int4, 0% acceptance, is 1765523544 / 2686077736 / 536747200.

Download invocation (not executed here; this host has neither the disk nor a use for a 384 GB download):

pip install "huggingface_hub[cli]"   # 1.23.0 at the time of writing; upstream pins no version
hf download mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp --local-dir /nvme/glm52_i4
ls -l /nvme/glm52_i4/out-mtp-*       # must be 3527131672 / 5366238584 / 1065950496

Or convert from FP8 yourself (not executed here)

./coli convert --model /nvme/glm52_i4 needs about 400 GB free and runs two resumable passes; the defaults below are read from the coli argparse and c/tools/convert_fp8_to_int4.py at the pinned commit, not from the README. Pass 1 converts the 78 main layers at --ebits 4 --io-bits 8 from --repo zai-org/GLM-5.2-FP8, one roughly 5 GB shard at a time (download, dequantize, requantize, delete, with a --min-free-gb 20 guard), so the full 756 GB checkpoint never exists on disk. Pass 2 re-runs with --mtp and the bits forced to max(8, ebits), so the MTP head lands at int8 without being asked (--no-mtp skips it). Upstream pins no converter dependency versions anywhere: the pinned commit contains no requirements.txt or pyproject.toml, and the README says only pip install torch safetensors huggingface_hub numpy. The versions that actually worked on this host for the adjacent oracle workflow below were Python 3.12.3, torch 2.13.0+cpu, transformers 5.14.1, safetensors 0.8.0, huggingface_hub 1.23.0, numpy 2.5.1; record the set you used next to the model directory, because the container format is unversioned too.

Prove the binary before committing 400 GB (executed here)

The repo's own oracle harness runs the real engine end to end on a 2.4 MB random-weight model with the true glm_moe_dsa architecture (MLA, DSA indexer, sigmoid router, shared expert). Executed here on 2026-07-17, oracle generated with transformers 5.14.1; the C engine itself needs no Python:

python3 tools/make_glm_oracle.py       # writes c/glm_tiny/ (2.4 MB) + ref_glm.json
SNAP=./glm_tiny TF=1 ./glm 64 16 16    # teacher-force the C engine vs the transformers oracle

[DSA] indexer active: top-4096 sparse attention beyond 4096 context tokens
[MTP] absent (draft=0)
[RAM_GB=60.0 auto] cap=64 ok (projected peak 3.7 GB)
== GLM C engine (glm_moe_dsa), cache=64 experts/layer | experts@16-bit dense@16-bit | idot: avx2 ==
loaded in 0.00s | resident dense: 1.57 MB | layers=5 experts=8 | MTP absent (draft=0)
PREFILL (teacher-forcing) C vs oracle: 32/32 positions | 5.7 pos/s

32/32 is the same pass criterion setup.sh checks for.

Preflight the placement (both commands executed here against the oracle)

Inspect the planned Disk/RAM/VRAM placement before loading anything (coli plan reads only safetensors headers; coli doctor is a read-only readiness check; neither starts the engine or touches tensor payloads):

COLI_MODEL=/nvme/glm52_i4 ./coli plan --ram 25 --json
COLI_MODEL=/nvme/glm52_i4 ./coli doctor --ram 25 --json
COLI_MODEL=/nvme/glm52_i4 ./coli chat --auto-tier   # apply the plan; interactive chat

Both ran here against the tiny oracle model: plan --json emitted its versioned plan ("version": 2, policy quality, per-tier byte budgets, cache_slots_per_layer), and doctor printed ten [ ok]/[skip] checks ending in result ok, exit code 0, with [skip] accelerator.cuda no NVIDIA GPU detected; CPU path is available on this GPU-less host.

Serve (complete invocation; executed here against the oracle only)

The gateway's full flag surface at the pinned commit, from the c/openai_server.py argparse (environment fallback in parentheses): --model (COLI_MODEL), --engine (path to the glm binary), --host (default 127.0.0.1), --port (default 8000), --model-id (COLI_MODEL_ID, default glm-5.2-colibri), --api-key (COLI_API_KEY), --cors-origin (repeatable), --cap (default 8 cache slots/layer), --max-tokens (default 1024), --max-queue (COLI_MAX_QUEUE, default 8), --queue-timeout (COLI_QUEUE_TIMEOUT, default 300 seconds), --kv-slots (COLI_KV_SLOTS, default 1, maximum 16).

COLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve \
  --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri \
  --max-queue 8 --queue-timeout 300 --kv-slots 1

That shape was executed here on 2026-07-17 with COLI_MODEL=./glm_tiny (the real 384 GB container does not fit this host), with one accommodation stated plainly: make_glm_oracle.py does not emit a tokenizer.json, so a minimal 256-token byte-level BPE tokenizer.json (the GPT-2 byte-to-unicode mapping, id equals byte, zero merges: the structure c/tok.h parses) was constructed locally to pass the tokenizer gate. The gateway ran under system Python 3.12.3 with no third-party packages installed, confirming the stdlib-only claim. Startup log, then one real round trip:

[stop] 1 stop tokens: 1
OpenAI-compatible API listening on http://127.0.0.1:8399/v1

curl -s http://127.0.0.1:8399/v1/chat/completions \
  -H 'Authorization: Bearer audit-secret' -H 'Content-Type: application/json' \
  -d '{"model": "glm-tiny-oracle", "messages": [{"role": "user", "content": "Hello"}],
       "max_tokens": 8, "stream": false}'

{"id": "chatcmpl-258ffb88ce01421ab661c32ef541c1b5", "object": "chat.completion",
 "model": "glm-tiny-oracle",
 "choices": [{"index": 0, "message": {"role": "assistant",
    "content": "\ufffd\f`\u0013\ufffd\u0007\ufffd\ufffd", "refusal": null},
    "logprobs": null, "finish_reason": "stop"}],
 "usage": {"prompt_tokens": 53, "completion_tokens": 8, "total_tokens": 61}}

The 8-token content is garbage bytes by construction (random weights, byte-level vocab); what this run validates is the serving plumbing, each item observed for real: HTTP 200 with the documented x-colibri-queue-wait-ms: 0 response header, GET /v1/models answering behind the bearer token, an HTTP 401 invalid_api_key error body without it, /health counters advancing from admitted: 0, completed: 0 to admitted: 1, completed: 1 across the request, and the engine appending a 67,480-byte .coli_kv into the model directory (so the model directory must be writable unless KVSAVE=0). A serving run of the real GLM-5.2 container was not executed here.

Useful runtime knobs: --temp T and nucleus --topp (defaults 0.7/0.90, tuned for int4; the official 1.0/0.95 samples quantization noise from the tail), --ngen N max tokens per answer, DRAFT=n MTP draft depth (DRAFT=0 disables speculation for reproducibility), GRAMMAR=g.gbnf grammar-forced drafts for constrained JSON/NDJSON output, THINK=1 enables GLM-5.2's reasoning block, PIN=stats.txt PIN_GB=N pins the hottest measured experts in spare RAM, and TF=1 runs teacher-forcing validation against the reference oracle.

How to develop with it

The development loop is anchored on the same kind of validation artifacts ds4 uses:

  • Official-oracle tests: TF=1 ./glm.exe 64 16 16 (or the Linux/macOS equivalent) teacher-forces a tiny random model with the real glm_moe_dsa architecture against a transformers oracle generated by tools/make_glm_oracle.py; the README's own reported result is 32/32 teacher-forcing positions and 20/20 greedy generation matches, with the same exactness re-validated when weight absorption is forced everywhere. The 32/32 teacher-forcing half was reproduced for real on this page's host (transcript in "How to use it").
  • Quality regression: coli bench runs HellaSwag, ARC-Challenge, and MMLU (0-shot log-likelihood, EleutherAI-harness style) against the real converted model, auto-downloading datasets on first use. The one published datapoint at the pinned commit is 62.5% mean acc_norm (n=40 per task, so roughly plus or minus 14 percentage points) against 85-95% published for full-precision GLM-5.2; the README is explicit that this gap is not yet attributable to quantization, because 0-shot log-likelihood scoring underserves a reasoning model that never gets to think, and the decisive experiment (an OLMoE fp16-vs-int4 A/B under the same harness, small enough to run both precisions) had not been run as of this commit.
  • Disk benchmarking: iobench measures parallel 19 MB random reads the way the engine actually reads experts. The project's own documented caveat (issue #86) matters here: a buffered read on a big-RAM box measures the OS page cache, not the disk, so use the O_DIRECT argument on a shard you have not touched this session for a real cold-read number; macOS has no O_DIRECT, and its F_NOCACHE fallback cannot evict pages a prior buffered run already resident-mapped.
  • Backend correctness: make test-c, make test-python, make metal-test (Metal kernels vs the CPU reference), and make cuda-test CUDA=1 (int8/int4/int2/f32 kernel correctness) are the per-backend gates; a reproducible CUDA A/B without downloading the full checkpoint exists via tools/make_glm_bench_model.py, which generates a deterministic 313M-parameter fixture with the real MLA/MoE/streaming shapes and random (non-language-model) weights.
  • Debugging: --trace-equivalent visibility comes from STATS=stats.txt (routing frequency capture, feeding PIN) and the COLI_DEBUG levels the openai_server gateway exposes (1 for output only, 2 for both request and response sides).

How to run it in production

"Production" here means a personal daily driver or a single trusted-team box, the same envelope ds4 targets, not a multi-tenant fleet:

  • Capacity model: coli serve holds one mutable KV context and executes one sequence at a time; concurrent HTTP requests queue behind a bounded FIFO admission queue (--max-queue, default 8; --queue-timeout, default 300 seconds), and a saturated or timed-out request gets an OpenAI-shaped HTTP 429 before any streaming headers are sent. --kv-slots N (up to 16) gives independent sequence contexts with their own token history, KV memory, and .coli_kv persistence file, but this establishes explicit ownership across slots, not real concurrent batching; the engine still runs one forward pass at a time.
  • Network posture: the default bind is localhost. COLI_API_KEY gates the HTTP API with a bearer token, and the pinned commit's history includes a same-day fix moving that comparison to hmac.compare_digest (a constant-time check, replacing whatever comparison preceded it) which is itself a sign of how recently the server's auth path has been hardened. There is no TLS termination built in; anything beyond loopback needs a reverse proxy, tunnel, or VPN in front of it, and --cors-origin should stay scoped to a specific origin rather than * outside a trusted local network.
  • Health and readiness: GET /health is the gateway's real health surface (the do_GET handler in c/openai_server.py at the pin). It is deliberately unauthenticated, as are /experts and the static web-dashboard files (the source comment marks static serving as "same trust level as /health"), so treat all three as public to anyone who can reach the port; everything under /v1/ requires the bearer token, and a missing key produced a real HTTP 401 invalid_api_key body in this page's executed run. The payload carries status, the scheduler counters (active, queued, capacity, max_queue, queue_timeout_seconds, admitted, completed, rejected, timed_out, cancelled), kv_slots, and, once the engine reports in, tiers (expert counts and GB per VRAM/RAM/disk tier) and hwinfo (cores, RAM, GPUs, CPU model); the counters advanced from admitted: 0, completed: 0 to 1, 1 across this page's executed oracle completion. One startup-order fact matters for probes, from serve() in the gateway source: the TCP port is bound before the engine child starts (upstream's own comment: a stale or occupied port must fail in milliseconds rather than after loading hundreds of GB), but Engine.__init__ blocks on the engine's READY sentinel before the gateway starts answering, so connections made during the skeleton load sit in the accept backlog instead of being refused. A /health probe during startup therefore hangs rather than erroring: give probes a timeout, treat the first 200 as readiness, and gate real traffic on one successful completion rather than on /health alone.
  • Supervision: upstream ships no unit file. The template below (not executed here; this host ran the gateway in the foreground) encodes what the sources actually require: the 65,536 fd limit the coli wrapper raises for itself because the engine mmaps 144-plus shards, a start timeout that covers the dense-skeleton load given the READY gating above, and a writable model directory for .coli_kv (observed written there in this page's executed run; set KVSAVE=0 to keep the model directory read-only instead).

# /etc/systemd/system/colibri.service  (template, not executed here; adjust paths)
[Unit]
Description=colibri GLM-5.2 OpenAI-compatible server
After=network-online.target

[Service]
User=coli
WorkingDirectory=/opt/colibri/current/c
EnvironmentFile=/etc/colibri/env   # COLI_MODEL=/nvme/glm52_i4  COLI_API_KEY=...
ExecStart=/usr/bin/python3 /opt/colibri/current/c/coli serve --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri
Restart=on-failure
RestartSec=10
TimeoutStartSec=900                # port binds early, but answers only after the skeleton loads
LimitNOFILE=65536                  # matches the rlimit the coli wrapper raises for itself

[Install]
WantedBy=multi-user.target
- Disk requirement: the model directory must sit on local ext4 (Linux) or NTFS (Windows), never a network or 9p mount; decode speed is bounded almost entirely by random-read latency to that disk, so this is not a soft recommendation. - RAM budget: the expert cache auto-sizes from MemAvailable, but the project's own history contains a real regression here (issue #12, fixed 2026-07-10): before that fix, the cache capped at 8 experts/layer regardless of available RAM, so a 128 GB machine ran with the same tiny cache as a 16 GB one. Any colibri benchmark from before that date should be treated as invalid and rerun. - Realistic throughput range: on the maintainer's own 25 GB WSL2 dev box, cold decode measures roughly 0.05 to 0.1 tokens/second, with a separate community datapoint of 0.11 tokens/second on a different 24 GB machine (Intel Core Ultra 7 270K); community-reported numbers on better hardware range up to about 2.06 tokens/second on an Apple M5 Max with the Metal backend warm, and up to 6.28 to 6.84 tokens/second on the project's own documented 6x RTX 5090 experiment with the full model resident in VRAM (see the GPU experiment write-up in References), well above the roughly 1.0 to 1.2 tokens/second single-GPU, CPU-streaming configurations. These are single-run, community-contributed measurements from a project actively soliciting more of them, not a vendor-audited benchmark suite; do not plan capacity around any single row.

How to maintain it

  • Pin the commit. There is no tagged release; main receives same-day commits, and at the pinned commit above, commits landing the same day as this page include hardening five separate crash paths in glm.c (a buffer guard, out-of-memory-checked mallocs, null-pointer dereferences), fixing a silent short-read that left uninitialized memory in config and oracle readers, and fixing a UTF-8 decode crash on Windows. Rebuild from a commit you have actually read the diff for, not a moving main.
  • Upgrade: a pin bump with a re-verification gate, and there is already a concrete reason to take it. Four security-relevant fixes landed within 17 hours of the pin (all verified in the compare range on 2026-07-17): 8c18b7af (2026-07-14, path traversal in the gateway's web static serving; the pinned commit still carries the weaker str(target).startswith(str(base)) check at c/openai_server.py line 739 that this commit replaced with relative_to, one more reason the pinned gateway stays on loopback), 2bdf0546 (2026-07-14, kv_alloc double-free on KV re-allocation), 4268e00f (2026-07-15, hardens the safetensors and JSON parsers against malicious model files, which matters precisely because the recommended container is a third-party community download), and ff04b320 (2026-07-15, Windows: load coli_cuda.dll by absolute path, closing a CWD DLL hijack). The procedure: git fetch, read git log --oneline <old>..<new> and the diff of anything touching glm.c, st.h, tok.h, or openai_server.py; check the new commit out into a fresh directory next to the old one, never in place; ./setup.sh; regenerate the oracle and re-run TF=1 expecting 32/32; start coli serve on a spare port against the same model directory; require a 200 from /health and one successful completion; only then switch. At the observed cadence (117 commits in the two days after the pin), review the range, not each commit.
  • Rollback: a directory switch, provided the two shared state kinds stay unmutated. Keep each verified checkout, with the glm binary it built (the binary lives inside the checkout in the run-in-place layout), as its own directory, and point a current symlink (or the systemd unit's WorkingDirectory and ExecStart) at the active one; rollback is repointing the symlink and restarting, because the model directory is shared across versions. Two caveats keep the model directory "read-mostly" rather than read-only: first, never reconvert the model in place during an upgrade, because the container format is unversioned and already grew a group-scaled int4 variant after the pin (fmt=4, commit 498ab0c2) that an older binary cannot read, so a reconversion writes to a new directory and the old one stays until the new binary has passed the oracle gate and a real completion; second, .coli_kv and .coli_usage are written into the model directory by whichever version is running (observed in this page's executed serve run) and are equally unversioned, so if a rolled-back binary misbehaves on session resume, move them aside, at the cost of one re-prefill and a cold pin-learning curve.
  • Re-run the self-test and the oracle check after every pull. ./setup.sh reproduces the 32/32 architecture self-test; TF=1 teacher-forcing against the transformers oracle is the cheapest full-path regression check available without downloading the real checkpoint.
  • Treat the learning cache and pin files as workload-specific state. .coli_usage accumulates real routing history and is not decayed by session-local --repin adaptation; deleting it (or moving to new hardware) resets colibri back to a cold learning curve.
  • Re-check MTP head format after any model re-download. The single most common colibri support question on the tracker (issues #8 and #102) is MTP acceptance stuck at 0% because the downloaded mirror ships int4 MTP heads instead of int8; the fix is comparing the three out-mtp-* file sizes against the README's documented int8 values, not re-running the engine.
  • Expect Windows-native and Metal support to be younger than the Linux/CUDA path. The native Windows port (no WSL) merged in pull request #131 on 2026-07-13, one day before this page was written, and its two community datapoints in the README (issues #113, #128) were opened the same day; the Metal backend's own README section is marked experimental.

Failure modes

  • MTP stuck at 0% acceptance. Almost always an int4 MTP head instead of int8 (issues #8, #102); verify the three out-mtp-* file sizes, not just that speculation is enabled.
  • Greedy output differs between runs with no sampling involved. Documented in issue #100: MTP, the CUDA expert tier, and batched prefill each round slightly differently from the single-token CPU path, and int4 GLM-5.2 sits close enough to argmax ties that this can flip a token; every emitted token is still the argmax of a valid forward, so the continuation stays correct, but the stream is not byte-identical across configurations. Use DRAFT=0, and add IDOT=0 COLI_CUDA=0, for reproducibility.
  • Cold-cache speculative decoding is a net time loss. A verified MTP draft routes to extra experts (about 660 rising to about 1,100 expert loads per token on a cold cache per the README's own measurement), so speculation can cost more than it saves until the expert cache or pin has warmed up.
  • Silently capped expert cache on a big-RAM machine. Fixed 2026-07-10 (issue #12); any measurement from before that date on a high-RAM box understates real throughput because the cache was capped at 8 experts/layer regardless of available RAM.
  • iobench over-reporting disk speed. A buffered read on a box with enough RAM to cache the tested shard reports page-cache bandwidth, not disk bandwidth (issue #86); use the O_DIRECT argument and an untouched shard for a real cold number, and note macOS has no true O_DIRECT equivalent.
  • Wrong model directory. The engine loads only its own converted int4 container; arbitrary GLM-5.2 GGUF or safetensors checkpoints will not load, by design.
  • Network exposure. No TLS is built in and, until very recently, the API key comparison was not constant-time; bind to loopback and put anything beyond a trusted local network behind a proxy or VPN.
  • No stability guarantee. No tagged release exists at the time of writing; a commit that builds and passes setup.sh today can be superseded by a same-day crash fix tomorrow. Read the diff before moving to a newer commit, do not assume monotonic improvement.

References

  • colibri repository (JustVugg/colibri), pinned commit: https://github.com/JustVugg/colibri/blob/37ffb616747bd77ea6e0b28fd8e1147e88882162/README.md
  • colibri OpenAI-compatible gateway at the pin (serve flags, /health handler, auth, static-serving check): https://github.com/JustVugg/colibri/blob/37ffb616747bd77ea6e0b28fd8e1147e88882162/c/openai_server.py
  • colibri CLI wrapper at the pin (tokenizer gate, convert orchestration, install layout): https://github.com/JustVugg/colibri/blob/37ffb616747bd77ea6e0b28fd8e1147e88882162/c/coli
  • colibri FP8-to-int4 converter at the pin (defaults, int8 MTP forcing, per-shard resume): https://github.com/JustVugg/colibri/blob/37ffb616747bd77ea6e0b28fd8e1147e88882162/c/tools/convert_fp8_to_int4.py
  • Post-pin security fixes (path traversal in static serving): https://github.com/JustVugg/colibri/commit/8c18b7af6897
  • Post-pin security fixes (safetensors/JSON parser hardening): https://github.com/JustVugg/colibri/commit/4268e00fa141
  • colibri grammar-forced-drafts reference: https://github.com/JustVugg/colibri/blob/37ffb616747bd77ea6e0b28fd8e1147e88882162/docs/grammar-draft.md
  • colibri 6x RTX 5090 GPU experiment write-up: https://github.com/JustVugg/colibri/blob/37ffb616747bd77ea6e0b28fd8e1147e88882162/docs/experiments/glm52-6x5090-2026-07-12.md
  • Issue #8, MTP int4 head unusable, int8 fixes acceptance: https://github.com/JustVugg/colibri/issues/8
  • Issue #100, greedy decoding is not reproducible across MTP/CUDA tier: https://github.com/JustVugg/colibri/issues/100
  • Issue #102, MTP stuck at 0% across RAM/CUDA (a second int4-head report): https://github.com/JustVugg/colibri/issues/102
  • Issue #12, expert cache RAM-cap regression, fixed 2026-07-10: https://github.com/JustVugg/colibri/issues/12
  • Issue #86, iobench buffered-read output misleading on big-RAM boxes: https://github.com/JustVugg/colibri/issues/86
  • Issue #101, WSL2 VHDX disk-speed correction: https://github.com/JustVugg/colibri/issues/101
  • Pull request #131, native Windows port (CUDA via runtime DLL, AVX-VNNI, RAM detection, serve-mode pipe fix): https://github.com/JustVugg/colibri/pull/131
  • GLM-5.2 int4 model with int8 MTP heads (community conversion, use this one): https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp
  • GLM-5.2 int4 model, original mirror (int4 MTP heads, 0% acceptance, do not use for speculation): https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4
  • GLM-5.2-FP8 checkpoint (converter source, 755.7 GB per the HF API on 2026-07-17): https://huggingface.co/zai-org/GLM-5.2-FP8
  • GLM-5.2 model card (Z.ai, MIT): https://huggingface.co/zai-org/GLM-5.2
  • GLM-5.2 blog (DSA/IndexShare, MTP): https://huggingface.co/blog/zai-org/glm-52-blog
  • DeepSeek-V3 technical report (source of the auxiliary-loss-free sigmoid router pattern colibri follows): https://arxiv.org/abs/2412.19437

Related: DwarfStar (ds4): DeepSeek V4 local inference · vLLM: serve GLM-5.2 · Prima.cpp: heterogeneous home-cluster inference · Quantization for inference · Speculative decoding · Open-weight serving · Running local coding agents · KV cache management · Glossary