Markdown

Colibri (colibrì): GLM-5.2 local inference on 25 GB of RAM¶

Scope: colibri (JustVugg/colibri), a single-file, zero-dependency C inference engine that runs exactly one checkpoint, Z.ai's GLM-5.2 (a roughly 744B-parameter Mixture-of-Experts model), on a consumer machine with about 25 GB of RAM by keeping the dense skeleton resident at int4 and streaming the routed experts from local NVMe with a per-layer LRU cache. This page covers the memory-hierarchy design, the sigmoid MoE router, native MTP speculative decoding and its documented acceptance and determinism caveats, and the project's actual maturity as of the commit pinned below, thirteen days after repo creation with same-day commits still landing. It is the counterpart to DwarfStar (ds4): DeepSeek V4 local inference (same shape of problem, a different model family and quantization mix) and the small-box end of the spectrum from serving GLM-5.2 with vLLM (the datacenter, multi-GPU, multi-tenant deployment of the same model family). It extends open-weight serving and quantization for inference.

Shell commands below are reference templates pinned to commit 37ffb61 (2026-07-14T14:13:19Z) of the JustVugg/colibri repository; verify flags against the commit you actually build. As of that commit the repository is 13 days old (created 2026-07-01), has 11,704 stars, 928 forks, 28 open issues, is Apache-2.0 licensed, and ships no tagged releases (gh api repos/JustVugg/colibri/releases and .../tags both return empty). Commits are landing same day, including crash and memory-safety hardening (see "How to maintain it"). Treat every performance number here as the repository's or a community contributor's own single-run measurement, not independently reproduced. The numpy block is self-contained, executed, and asserted on this page; the real 744B-parameter engine was not run here (it needs roughly 400 GB of free disk and a 756 GB FP8 checkpoint download), so the block validates colibri's sigmoid MoE router mechanism instead, matching how DwarfStar (ds4) handles the same constraint for a similar project.

What it is¶

Colibri is a single C file, c/glm.c (238,244 bytes on disk at the pinned commit; the README describes it as "~2,400 lines," but a direct line count on that file at the same commit is 4,212 lines, 4,121 non-blank, so treat the README's line-count figure as stale rather than as this page's own measurement), plus small headers, with zero runtime dependencies: no BLAS, no Python at runtime, no GPU required. It is not a general GGUF or safetensors runner; it loads exactly one model family, GLM-5.2, converted into its own int4 container. Python appears only in the one-time offline FP8-to-int4 converter. Colibri's README states GLM-5.2 is "744B-parameter" with about 40B active per token; the vLLM serving page for the same checkpoint cites the vLLM recipe's roughly 743B total, roughly 39B active, with the Hugging Face safetensors metadata totaling 753B. Treat the small spread across these independent sources as a rounding or counting-convention gap, not something this page resolves.

The engine treats VRAM, RAM, and disk as one managed memory hierarchy, split by how often each part of the model changes per token:

The dense skeleton (attention, shared expert, embeddings, router, roughly 17B parameters) stays resident in RAM at int4, about 9.9 GB.
The 21,504 routed experts ("75 MoE layers x 256 experts + the MTP head, ~19 MB each at int4" per the README) live on disk as a roughly 370 GB int4 container and stream on demand, backed by a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free extra layer. The README's own arithmetic here does not fully reconcile: 75 x 256 is 19,200, not 21,504, while 84 x 256 is exactly 21,504; this page quotes the source's stated figures without resolving the discrepancy, per this KB's policy of flagging rather than silently picking a side when a primary source is internally inconsistent.

Two mechanisms carry most of the design. MLA attention with a compressed KV cache stores 576 floats per token instead of 32,768 uncompressed (a 57x reduction, since GLM-5.2 has 64 attention heads and no grouped-query attention), with weight absorption at decode time so the query absorbs kv_b and context is reconstructed after attention rather than materializing per-token keys and values. A DeepSeek-V3-style sigmoid router ("noaux_tc", a routed_scaling_factor, a shared expert, and the first three layers left dense) selects the top-k experts per token; the auxiliary-loss-free bias pattern it follows traces to the DeepSeek-V3 technical report. GLM-5.2's own lightning indexer (DSA, Dynamic Sparse Attention) additionally restricts causal key selection to the top 2,048 keys per query on most layers.

Colibri also implements native MTP (multi-token prediction) speculative decoding using GLM-5.2's own draft head at layer 78, grammar-forced speculative drafts for constrained JSON/NDJSON output, int8/int4/int2 AVX2 integer-dot kernels with per-row scales, a byte-level BPE tokenizer, and a single always-on coli serve process exposing an OpenAI-compatible HTTP API. It runs on Linux, WSL2, macOS (with an opt-in Metal backend), and native Windows 11 via MinGW-w64 (no WSL required, since pull request #131), plus an opt-in CUDA backend for pinning hot experts in VRAM.

The offline converter (c/tools/convert_fp8_to_int4.py) downloads the official 756 GB GLM-5.2-FP8 checkpoint one shard at a time (about 5 GB each), dequantizes, requantizes into colibri's container, and deletes the shard, so the full FP8 checkpoint never has to exist on disk at once; it is resumable and needs roughly 400 GB of free space for the int4 output.

Why use it¶

A 744B frontier-class MoE on hardware that costs less than a server fan. The README's own framing: this is "a 744B frontier-class model answering correctly on a machine that costs less than one H100 fan." The dev machine it was built and measured on is a 12-core WSL2 box with 25 GB of RAM.
RAM is a speed dial by design, not a correctness dial. "Insufficient fast memory may reduce speed, but the default policy never silently changes model precision or router semantics." The --policy quality and --policy balanced modes preserve checkpoint quantization and router decisions unless you explicitly pass --topk or --topp, and those lossy overrides print a warning before proceeding. The expert cache auto-sizes from MemAvailable at startup specifically so the kernel OOM-killer does not fire.
It gets faster the more you use it. A .coli_usage histogram records which experts your real workload routes to and pins the hottest ones in spare RAM at the next startup; --repin N additionally lets a decaying session heat map replace cold pinned experts with hot streamed ones at safe turn boundaries.
Sessions resume with zero re-prefill. coli chat appends the compressed MLA KV cache to .coli_kv after every turn (about 182 KB/token, crash-safe); closing and reopening a conversation the next day still has the full history, validated byte-identical to an uninterrupted session.
A single engineer's project you can actually read. The whole inference path is one file with no framework underneath it, which is also why its correctness caveats (next section) are unusually explicit compared to a project built on a larger dependency stack.

When to use it (and when not)¶

Use it when:

You want to run GLM-5.2 privately on a single consumer machine (25 GB RAM is the proven floor; the project explicitly asks for more RAM, cores, and faster NVMe as datapoints) and you accept sub-1 to a few tokens/second depending on disk and RAM, not interactive-chat-service latency.
You want to inspect or modify a small, dependency-free reference implementation of MLA attention, a sigmoid MoE router, and MTP speculative decoding rather than reading a framework's abstraction layers.
You can tolerate the documented non-determinism: greedy output is not guaranteed byte-identical across kernel families, batch sizes, or GPU tiers.

Do not use it when:

You need throughput or multi-tenant serving. coli serve keeps one model process and one mutable KV context; concurrent HTTP requests queue behind a bounded FIFO (default depth 8) rather than being batched. For a datacenter GLM-5.2 deployment with tensor parallelism and real concurrency, use vLLM's GLM-5.2 recipe instead.
You need a stable, versioned dependency. There are no tagged releases; the pinned commit above had crash-path and memory-safety fixes land the same day this page was written.
You need bit-exact reproducibility without extra flags. MTP, the CUDA expert tier, and batched prefill each round slightly differently from the single-token CPU path, and the project's own issue tracker documents this flipping greedy output on 3 of 5 test prompts with MTP and 2 of 5 with the CUDA tier, with zero speculation in the MTP case. DRAFT=0 removes speculation; add IDOT=0 COLI_CUDA=0 for full kernel-family and GPU independence.
You need arbitrary model support. Like ds4, this is a one-model engine; it does not load arbitrary GGUF or safetensors checkpoints, only its own converted GLM-5.2 container.

Architecture¶

flowchart TB
  subgraph DISK["Disk: int4 container, ~370 GB"]
    EXP["21,504 routed experts<br/>~19 MB each, int4"]
    KVD[".coli_kv<br/>compressed MLA KV, crash-safe append"]
  end
  subgraph RAM["Resident RAM: int4 dense skeleton, ~9.9 GB"]
    ATT["MLA attention + shared expert<br/>+ embeddings (~17B params)"]
    RTR["Sigmoid router (noaux_tc)<br/>top-k expert selection"]
    EC["Per-layer LRU expert cache<br/>+ learning pin (.coli_usage)"]
    MTP["MTP head, layer 78<br/>must be int8 or acceptance collapses"]
    KV["Live compressed KV<br/>576 floats/token"]
  end
  subgraph OPT["Optional pinned tiers (opt-in)"]
    CUDA["CUDA: resident + hot-pinned experts"]
    METAL["Metal: batched expert SwiGLU + fused attention"]
  end
  RTR -->|"top-8 expert ids"| EC
  EC -->|"cache miss: read from disk"| EXP
  EC -.->|"hottest experts"| CUDA
  EC -.->|"hottest experts"| METAL
  MTP -->|"draft tokens, verified in one batched forward"| RTR
  KV <-->|"append every turn / resume at startup, zero re-prefill"| KVD
  CLI["coli chat / coli run"] --> RTR
  SRV["coli serve<br/>OpenAI-compatible, FIFO queue, kv-slots"] --> RTR

The design bet mirrors ds4's: precision follows load-bearing structure. Every token passes through attention, the router, and the shared expert, so those stay resident at int4 with the rest of the dense skeleton; each token only touches 8 of 256 routed experts per layer, so those absorb the streaming cost. The difference from ds4 is where the correctness edge sits: ds4's risk is 2-bit expert quantization behind an 8-bit skeleton, while colibri keeps everything at int4/int8 but documents a distinct hazard, that its integer kernels are shape-dependent, so a batched or GPU forward can round differently from the single-token CPU path and flip an argmax tie near the router's decision boundary.

How it works (validated core mechanics)¶

The mechanism validated below is the sigmoid MoE router: given per-token logits over 256 experts and a small per-expert bias (the auxiliary-loss-free load-balancing term colibri's README attributes to the DeepSeek-V3-style "noaux_tc" pattern), select the top-8 experts and renormalize their raw sigmoid affinities into gate weights. A second, independently-written brute-force implementation validates the fast path, and three adversarial cases probe the boundary this router shares with colibri's own documented non-determinism: an exact tie at the cutoff, a rounding-scale perturbation at a near-tie (the same class of instability the project's issue tracker reports for batched and GPU forwards), and a corrupted (non-finite) input that must be rejected rather than silently ranked.

import numpy as np


def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-x))


def route_topk(logits, bias, top_k):
    """DeepSeek-V3/GLM-5.2-style sigmoid router ('noaux_tc'): the bias is an
    auxiliary-loss-free load-balancing term added only to RANK experts for
    top-k selection. The combination weight uses the raw (unbiased) sigmoid
    affinity, renormalized over the selected set and scaled by
    routed_scaling_factor downstream (omitted here: it is a constant
    post-multiply and does not affect which experts are chosen).
    Fast path: argpartition for the top-k cut, then a full order on just
    those k candidates with a deterministic tie-break (lower expert id wins
    a tied score). Raises on non-finite input instead of ranking garbage."""
    if not np.all(np.isfinite(logits)) or not np.all(np.isfinite(bias)):
        raise ValueError("non-finite router logits/bias: corrupted expert metadata")
    gate = sigmoid(logits)
    rank_score = gate + bias
    n, E = logits.shape
    assert 1 <= top_k <= E
    cut = np.argpartition(-rank_score, top_k - 1, axis=1)[:, :top_k]
    ordered = np.empty_like(cut)
    for i in range(n):
        ordered[i] = sorted(cut[i], key=lambda e: (-rank_score[i, e], e))
    w = np.take_along_axis(gate, ordered, axis=1)
    w = w / w.sum(axis=1, keepdims=True)
    return ordered, w


def route_topk_bruteforce(logits, bias, top_k):
    """Reference: full sort of every expert per token, same tie-break rule,
    no argpartition shortcut. Independent code path for the equivalence check."""
    if not np.all(np.isfinite(logits)) or not np.all(np.isfinite(bias)):
        raise ValueError("non-finite router logits/bias: corrupted expert metadata")
    gate = sigmoid(logits)
    rank_score = gate + bias
    n, E = logits.shape
    ordered = np.empty((n, top_k), dtype=np.int64)
    for i in range(n):
        full_order = sorted(range(E), key=lambda e: (-rank_score[i, e], e))
        ordered[i] = full_order[:top_k]
    w = np.take_along_axis(gate, ordered, axis=1)
    w = w / w.sum(axis=1, keepdims=True)
    return ordered, w


# --- equivalence: fast (argpartition) path must match the brute-force path ---
rng = np.random.default_rng(5)
n, E, top_k = 64, 256, 8   # one GLM-5.2 layer: 256 routed experts, top-8
logits = rng.normal(size=(n, E))
bias = rng.normal(scale=0.05, size=E)   # small load-balancing bias, shared across tokens

fast_idx, fast_w = route_topk(logits, bias, top_k)
ref_idx, ref_w = route_topk_bruteforce(logits, bias, top_k)
assert np.array_equal(fast_idx, ref_idx), "argpartition path disagrees with brute force"
assert np.allclose(fast_w, ref_w), "gate weights disagree between the two code paths"
assert np.allclose(fast_w.sum(axis=1), 1.0), "renormalized gate weights must sum to 1"
print(f"equivalence: {n} tokens x {E} experts, top-{top_k}: fast path == brute force. "
      f"OK ({n}/{n} tokens matched)")

# --- adversarial 1: exact tie at the top-k cutoff must be broken deterministically ---
tied_logits = np.zeros((1, 8))
tied_bias = np.zeros(8)
# experts 3 and 4 tie exactly for the last (8th vs 9th... here just the top-4) slot
tied_logits[0, :4] = 1.0     # four clear winners
tied_logits[0, 4] = 0.3      # exact tie for the last spot
tied_logits[0, 5] = 0.3      # exact tie for the last spot
tied_logits[0, 6:] = -1.0
idx_a, _ = route_topk(tied_logits, tied_bias, top_k=5)
idx_b, _ = route_topk_bruteforce(tied_logits, tied_bias, top_k=5)
assert np.array_equal(idx_a, idx_b), "tie-break must be identical across code paths"
assert list(idx_a[0][-1:]) == [4], "tie-break must pick the LOWER expert id (4, not 5)"
print(f"tie-break: experts 4 and 5 tied exactly, deterministic rule picked "
      f"expert {int(idx_a[0][-1])} (lower id). Selected set: {sorted(idx_a[0].tolist())}")

# --- adversarial 2: a rounding-scale nudge at a near-tie flips the winner ---
# This is the numeric root cause the colibri README documents (issue #100): a
# batched/kernel-family forward rounds slightly differently from the single-token
# path, and int4 GLM-5.2 sits close enough to argmax ties that the rounding change
# flips a token. Reproduced here as a bias perturbation far smaller than any
# score gap elsewhere in the row.
near_tie = np.zeros((1, 8))
near_tie[0, :4] = 1.0
near_tie[0, 4] = 0.30000
near_tie[0, 5] = 0.29999   # 1e-5 below expert 4: same "near tie" scale as int4 rounding noise
near_tie[0, 6:] = -1.0
before, _ = route_topk(near_tie, tied_bias, top_k=5)
nudged = near_tie.copy()
nudged[0, 5] += 2e-5       # a rounding-magnitude nudge, smaller than any other gap in the row
after, _ = route_topk(nudged, tied_bias, top_k=5)
assert before[0][-1] == 4 and after[0][-1] == 5, "the nudge must flip the 5th selected expert"
print(f"near-tie sensitivity: a 2e-5 nudge (rounding-noise scale) at a near-tied boundary "
      f"flips the 5th selected expert from {int(before[0][-1])} to {int(after[0][-1])} "
      f"-- this is the same class of instability behind the engine's own documented "
      f"non-determinism (greedy output differs across batched/kernel-family forwards).")

# --- adversarial 3: corrupted / non-finite router input must be rejected, not ranked ---
corrupt = logits.copy()
corrupt[3, 17] = np.nan   # simulate a corrupted or partially-read expert-bias file
try:
    route_topk(corrupt, bias, top_k)
    raised = False
except ValueError as e:
    raised = True
    msg = str(e)
assert raised, "a NaN in router logits must raise, not silently sort into a wrong-but-plausible result"
print(f"corruption check: NaN router logit at token 3 raised ValueError('{msg}') "
      f"instead of silently ranking (Python's sort with NaN present is inconsistent "
      f"and would otherwise produce a nondeterministic, unflagged wrong selection).")

print("colibri sigmoid router: all asserts passed")

Executed output:

equivalence: 64 tokens x 256 experts, top-8: fast path == brute force. OK (64/64 tokens matched)
tie-break: experts 4 and 5 tied exactly, deterministic rule picked expert 4 (lower id). Selected set: [0, 1, 2, 3, 4]
near-tie sensitivity: a 2e-5 nudge (rounding-noise scale) at a near-tied boundary flips the 5th selected expert from 4 to 5 -- this is the same class of instability behind the engine's own documented non-determinism (greedy output differs across batched/kernel-family forwards).
corruption check: NaN router logit at token 3 raised ValueError('non-finite router logits/bias: corrupted expert metadata') instead of silently ranking (Python's sort with NaN present is inconsistent and would otherwise produce a nondeterministic, unflagged wrong selection).
colibri sigmoid router: all asserts passed

The point the second adversarial case makes is not that colibri's real router has a bug: it is that any top-k router run through mixed-precision, shape-dependent kernels sits near ties often enough that a rounding difference of this magnitude changes the selected expert set, which is exactly the mechanism colibri's own maintainers point to for issue #100 (MTP and the CUDA tier both changing greedy output with zero speculation involved). The third case is a defensive property this toy implementation adds, not a claim about what colibri's C code does on a corrupted disk read; it demonstrates why a real streaming-from-disk engine needs an explicit non-finite check rather than trusting sort semantics on NaN.

How to use it¶

git clone https://github.com/JustVugg/colibri && cd colibri
git checkout 37ffb616747bd77ea6e0b28fd8e1147e88882162   # pinned commit; no tagged release exists
cd c && ./setup.sh                # checks gcc/OpenMP, builds, self-tests (expects "32/32" positions)

Skip the FP8-to-int4 conversion by downloading the pre-converted mirror with int8 MTP heads (the original mirror ships int4 MTP heads, which give 0% draft acceptance per issues #8 and #102; verify with ls -l <model>/out-mtp-* against the byte sizes the README documents for each variant):

# https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp
COLI_MODEL=/nvme/glm52_i4 ./coli chat

Or convert from the official FP8 checkpoint yourself; this needs pip install torch safetensors huggingface_hub numpy and about 400 GB of free space, and is resumable at any point since it processes one shard at a time and never keeps the full 756 GB checkpoint on disk:

./coli convert --model /nvme/glm52_i4

Inspect the planned Disk/RAM/VRAM placement before loading anything (coli plan reads only safetensors headers; coli doctor is a read-only readiness check; neither starts the engine or touches tensor payloads):

COLI_MODEL=/nvme/glm52_i4 ./coli plan --ram 25 --json
COLI_MODEL=/nvme/glm52_i4 ./coli doctor --ram 25 --json
COLI_MODEL=/nvme/glm52_i4 ./coli chat --auto-tier   # apply the plan; interactive chat

Useful runtime knobs: --temp T and nucleus --topp (defaults 0.7/0.90, tuned for int4; the official 1.0/0.95 samples quantization noise from the tail), --ngen N max tokens per answer, DRAFT=n MTP draft depth (DRAFT=0 disables speculation for reproducibility), GRAMMAR=g.gbnf grammar-forced drafts for constrained JSON/NDJSON output, THINK=1 enables GLM-5.2's reasoning block, PIN=stats.txt PIN_GB=N pins the hottest measured experts in spare RAM, and TF=1 runs teacher-forcing validation against the reference oracle.

How to develop with it¶

The development loop is anchored on the same kind of validation artifacts ds4 uses:

Official-oracle tests: TF=1 ./glm.exe 64 16 16 (or the Linux/macOS equivalent) teacher-forces a tiny random model with the real glm_moe_dsa architecture against a transformers oracle generated by tools/make_glm_oracle.py; the README's own reported result is 32/32 teacher-forcing positions and 20/20 greedy generation matches, with the same exactness re-validated when weight absorption is forced everywhere.
Quality regression: coli bench runs HellaSwag, ARC-Challenge, and MMLU (0-shot log-likelihood, EleutherAI-harness style) against the real converted model, auto-downloading datasets on first use. The one published datapoint at the pinned commit is 62.5% mean acc_norm (n=40 per task, so roughly plus or minus 14 percentage points) against 85-95% published for full-precision GLM-5.2; the README is explicit that this gap is not yet attributable to quantization, because 0-shot log-likelihood scoring underserves a reasoning model that never gets to think, and the decisive experiment (an OLMoE fp16-vs-int4 A/B under the same harness, small enough to run both precisions) had not been run as of this commit.
Disk benchmarking: iobench measures parallel 19 MB random reads the way the engine actually reads experts. The project's own documented caveat (issue #86) matters here: a buffered read on a big-RAM box measures the OS page cache, not the disk, so use the O_DIRECT argument on a shard you have not touched this session for a real cold-read number; macOS has no O_DIRECT, and its F_NOCACHE fallback cannot evict pages a prior buffered run already resident-mapped.
Backend correctness: make test-c, make test-python, make metal-test (Metal kernels vs the CPU reference), and make cuda-test CUDA=1 (int8/int4/int2/f32 kernel correctness) are the per-backend gates; a reproducible CUDA A/B without downloading the full checkpoint exists via tools/make_glm_bench_model.py, which generates a deterministic 313M-parameter fixture with the real MLA/MoE/streaming shapes and random (non-language-model) weights.
Debugging: --trace-equivalent visibility comes from STATS=stats.txt (routing frequency capture, feeding PIN) and the COLI_DEBUG levels the openai_server gateway exposes (1 for output only, 2 for both request and response sides).

How to run it in production¶

"Production" here means a personal daily driver or a single trusted-team box, the same envelope ds4 targets, not a multi-tenant fleet:

Capacity model: coli serve holds one mutable KV context and executes one sequence at a time; concurrent HTTP requests queue behind a bounded FIFO admission queue (--max-queue, default 8; --queue-timeout, default 300 seconds), and a saturated or timed-out request gets an OpenAI-shaped HTTP 429 before any streaming headers are sent. --kv-slots N (up to 16) gives independent sequence contexts with their own token history, KV memory, and .coli_kv persistence file, but this establishes explicit ownership across slots, not real concurrent batching; the engine still runs one forward pass at a time.
Network posture: the default bind is localhost. COLI_API_KEY gates the HTTP API with a bearer token, and the pinned commit's history includes a same-day fix moving that comparison to hmac.compare_digest (a constant-time check, replacing whatever comparison preceded it) which is itself a sign of how recently the server's auth path has been hardened. There is no TLS termination built in; anything beyond loopback needs a reverse proxy, tunnel, or VPN in front of it, and --cors-origin should stay scoped to a specific origin rather than * outside a trusted local network.
Disk requirement: the model directory must sit on local ext4 (Linux) or NTFS (Windows), never a network or 9p mount; decode speed is bounded almost entirely by random-read latency to that disk, so this is not a soft recommendation.
RAM budget: the expert cache auto-sizes from MemAvailable, but the project's own history contains a real regression here (issue #12, fixed 2026-07-10): before that fix, the cache capped at 8 experts/layer regardless of available RAM, so a 128 GB machine ran with the same tiny cache as a 16 GB one. Any colibri benchmark from before that date should be treated as invalid and rerun.
Realistic throughput range: on the maintainer's own 25 GB WSL2 dev box, cold decode measures roughly 0.05 to 0.1 tokens/second, with a separate community datapoint of 0.11 tokens/second on a different 24 GB machine (Intel Core Ultra 7 270K); community-reported numbers on better hardware range up to about 2.06 tokens/second on an Apple M5 Max with the Metal backend warm, and up to 6.28 to 6.84 tokens/second on the project's own documented 6x RTX 5090 experiment with the full model resident in VRAM (see the GPU experiment write-up in References), well above the roughly 1.0 to 1.2 tokens/second single-GPU, CPU-streaming configurations. These are single-run, community-contributed measurements from a project actively soliciting more of them, not a vendor-audited benchmark suite; do not plan capacity around any single row.

How to maintain it¶

Pin the commit. There is no tagged release; main receives same-day commits, and at the pinned commit above, commits landing the same day as this page include hardening five separate crash paths in glm.c (a buffer guard, out-of-memory-checked mallocs, null-pointer dereferences), fixing a silent short-read that left uninitialized memory in config and oracle readers, and fixing a UTF-8 decode crash on Windows. Rebuild from a commit you have actually read the diff for, not a moving main.
Re-run the self-test and the oracle check after every pull. ./setup.sh reproduces the 32/32 architecture self-test; TF=1 teacher-forcing against the transformers oracle is the cheapest full-path regression check available without downloading the real checkpoint.
Treat the learning cache and pin files as workload-specific state. .coli_usage accumulates real routing history and is not decayed by session-local --repin adaptation; deleting it (or moving to new hardware) resets colibri back to a cold learning curve.
Re-check MTP head format after any model re-download. The single most common colibri support question on the tracker (issues #8 and #102) is MTP acceptance stuck at 0% because the downloaded mirror ships int4 MTP heads instead of int8; the fix is comparing the three out-mtp-* file sizes against the README's documented int8 values, not re-running the engine.
Expect Windows-native and Metal support to be younger than the Linux/CUDA path. The native Windows port (no WSL) merged in pull request #131 on 2026-07-13, one day before this page was written, and its two community datapoints in the README (issues #113, #128) were opened the same day; the Metal backend's own README section is marked experimental.

Failure modes¶

MTP stuck at 0% acceptance. Almost always an int4 MTP head instead of int8 (issues #8, #102); verify the three out-mtp-* file sizes, not just that speculation is enabled.
Greedy output differs between runs with no sampling involved. Documented in issue #100: MTP, the CUDA expert tier, and batched prefill each round slightly differently from the single-token CPU path, and int4 GLM-5.2 sits close enough to argmax ties that this can flip a token; every emitted token is still the argmax of a valid forward, so the continuation stays correct, but the stream is not byte-identical across configurations. Use DRAFT=0, and add IDOT=0 COLI_CUDA=0, for reproducibility.
Cold-cache speculative decoding is a net time loss. A verified MTP draft routes to extra experts (about 660 rising to about 1,100 expert loads per token on a cold cache per the README's own measurement), so speculation can cost more than it saves until the expert cache or pin has warmed up.
Silently capped expert cache on a big-RAM machine. Fixed 2026-07-10 (issue #12); any measurement from before that date on a high-RAM box understates real throughput because the cache was capped at 8 experts/layer regardless of available RAM.
iobench over-reporting disk speed. A buffered read on a box with enough RAM to cache the tested shard reports page-cache bandwidth, not disk bandwidth (issue #86); use the O_DIRECT argument and an untouched shard for a real cold number, and note macOS has no true O_DIRECT equivalent.
Wrong model directory. The engine loads only its own converted int4 container; arbitrary GLM-5.2 GGUF or safetensors checkpoints will not load, by design.
Network exposure. No TLS is built in and, until very recently, the API key comparison was not constant-time; bind to loopback and put anything beyond a trusted local network behind a proxy or VPN.
No stability guarantee. No tagged release exists at the time of writing; a commit that builds and passes setup.sh today can be superseded by a same-day crash fix tomorrow. Read the diff before moving to a newer commit, do not assume monotonic improvement.

References¶

colibri repository (JustVugg/colibri), pinned commit: https://github.com/JustVugg/colibri/blob/37ffb616747bd77ea6e0b28fd8e1147e88882162/README.md
colibri grammar-forced-drafts reference: https://github.com/JustVugg/colibri/blob/37ffb616747bd77ea6e0b28fd8e1147e88882162/docs/grammar-draft.md
colibri 6x RTX 5090 GPU experiment write-up: https://github.com/JustVugg/colibri/blob/37ffb616747bd77ea6e0b28fd8e1147e88882162/docs/experiments/glm52-6x5090-2026-07-12.md
Issue #8, MTP int4 head unusable, int8 fixes acceptance: https://github.com/JustVugg/colibri/issues/8
Issue #100, greedy decoding is not reproducible across MTP/CUDA tier: https://github.com/JustVugg/colibri/issues/100
Issue #102, MTP stuck at 0% across RAM/CUDA (a second int4-head report): https://github.com/JustVugg/colibri/issues/102
Issue #12, expert cache RAM-cap regression, fixed 2026-07-10: https://github.com/JustVugg/colibri/issues/12
Issue #86, iobench buffered-read output misleading on big-RAM boxes: https://github.com/JustVugg/colibri/issues/86
Issue #101, WSL2 VHDX disk-speed correction: https://github.com/JustVugg/colibri/issues/101
Pull request #131, native Windows port (CUDA via runtime DLL, AVX-VNNI, RAM detection, serve-mode pipe fix): https://github.com/JustVugg/colibri/pull/131
GLM-5.2 int4 model with int8 MTP heads (community conversion, use this one): https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp
GLM-5.2 int4 model, original mirror (int4 MTP heads, 0% acceptance, do not use for speculation): https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4
GLM-5.2 model card (Z.ai, MIT): https://huggingface.co/zai-org/GLM-5.2
GLM-5.2 blog (DSA/IndexShare, MTP): https://huggingface.co/blog/zai-org/glm-52-blog
DeepSeek-V3 technical report (source of the auxiliary-loss-free sigmoid router pattern colibri follows): https://arxiv.org/abs/2412.19437