Skip to content
Markdown

GLM Infra Agent and dense feedback for inference optimisation

Scope: Z.ai's account of how a GLM-5.3-powered agent helped bring GLM-5.3-Flash inference to production on a cluster of more than 100,000 domestic accelerators in under two weeks; covers the dense-feedback loop, three worked cases (a context-parallel precision bug, a GIL-blocked KV transfer, kernel optimisation skeletons), the serving stack, and a runnable emulation of the precision fix, with a record of what the post does not substantiate.

Overview

The post (Z.ai, published around 17 September 2026 by the asset date) describes an "Infra Agent" that carried out much of the work of standing up GLM-5.3-Flash inference: a 1M-token context window, multimodal input, a new architecture, limited memory capacity and bandwidth, and immature kernels. Z.ai reports about 3x end-to-end throughput from baseline to launch, production readiness in under two weeks, and per-token cost and hardware utilisation "comparable to mainstream NVIDIA GPUs". The accelerator vendor is not named and no absolute throughput, latency, or cost figure is given, so the comparison cannot be checked.

The transferable idea is "dense feedback": give an agent feedback that is local, cheap, and objectively checkable, so each experiment answers one question, instead of a single end-to-end metric that says only that things got worse.

flowchart LR
  ENG["Engineers: objectives, constraints, review"] --> AG["Infra Agent: hypothesis and code change"]
  AG --> FB["Layered feedback"]
  FB --> C["Kernel correctness: partitioned vs unpartitioned"]
  FB --> T["Traces and events: where time goes"]
  FB --> M["Microbenchmarks: which approach, under what shape"]
  C --> GATE["End-to-end acceptance"]
  T --> GATE
  M --> GATE
  GATE --> AG

Core knowledge

Dense feedback: three properties

Feedback must be sufficiently local, inexpensive and timely to obtain, and support objective verification. Correctness feedback answers whether the computation is correct, system-behaviour feedback shows where time goes, and performance feedback says which approach wins and under what conditions. Local checks discard bad changes early; end-to-end load tests confirm that local gains survive real traffic. Engineers define goals and boundaries, and review changes touching numerical semantics, asynchronous concurrency, architecture, and production risk.

Serving stack reported

Intra-node tensor parallelism for linear attention and the LM head, ReplaySSM, W8A8 quantisation, mixed-precision cache quantisation (INT8, FP8, BF16), Layer Split, and an Encode-Prefill-Decode disaggregated architecture. Z.ai credits these together with roughly 3x end-to-end serving gain. The post separately says the agent-driven loop tripled throughput relative to the initial baseline; these read as the same 3x described two ways, not two compounding gains.1 For disaggregation background see disaggregated inference and KV cache transfer with NIXL.

Case 1: context-parallel precision bug (correctness feedback)

The agent mapped each parallelism configuration to the kernels it exercises and compared partitioned against unpartitioned paths within numerical tolerance. This exposed a precision problem in the context-parallel (CP) path of the KDA kernel. CP merges state across context shards with two operations: M = tl.dot(M_chunk, M) and S_next = tl.dot(M, S) + H. Triton's tl.dot defaulted to TF32 for FP32 inputs, and error accumulated over merges, worse at long context. The fix sets input_precision="tf32x3" on both, which combines three TF32 tensor-core operations for near-FP32 accuracy. Z.ai states the fix merged upstream in Flash Linear Attention PR #1180. Independently checked: the PR is titled "[CP] use tf32x3 affine chain in kcp", was merged on 27 August 2026, and its description states the same default-TF32 cause and the tf32x3 remedy across three kernels.

Case 2: KV transfer blocked by the GIL (system-behaviour feedback)

Engineers defined scenarios (prefill alone, prefill plus KV transfer, decode alone) with an acceptance rule that prefill plus KV transfer stays within 5 percent of prefill alone. The agent measured a gap above 20 percent in some scenarios. The timeline showed Python-side Mooncake transfer never overlapping DeepEP dispatch and combine intervals. In DeepEP v1.2.1, intranode_dispatch and intranode_combine did not release the Python GIL (and dispatch waited on the CPU for the GPU to return the received-token count), so the transfer thread could not get the lock. The same section names internode_dispatch. Releasing the GIL across those C++ sections brought the gap below 1 percent. Z.ai does not say whether the fix was contributed to DeepEP or Mooncake.

Case 3: kernel optimisation skeletons (performance feedback)

The agent distilled techniques from SGLang, Flash Linear Attention, and DeepGEMM kernels into "optimization skeletons" holding applicability conditions, transformation method, resource limits, and validation evidence, then re-tuned tiling, memory access, and resource allocation per kernel. For a KDA decode kernel: ReplaySSM, which trades compute for memory, first raised kernel time (v0 to v1); a division optimisation cut v1 time by 9.6 percent; then merging V-dimension tiles into one thread block, keeping shared FP32 normalisation and gating results in registers, and using one warp-level reduction removed four-fold redundant computation for a 1.71x speedup over v2. A kernel given more resources can slow the whole pipeline by starving the concurrent KV-transfer kernel, so kernels must be judged in the engine's real execution context. See agentic kernel generation harness for the general pattern.

Production result claimed

GLM-5.3-Flash was tested anonymously as "Ox-Alpha" on OpenCode and OpenRouter, and Z.ai reports it became the most-used model on both within a week, with more than 62 trillion tokens in six days. The Ornn paper (open-weight inference economics) independently lists GLM-5.3-Flash as a 320B total, 18B active MIT-licensed model released 26 August 2026 at $0.15 / $0.50 per million tokens and an Intelligence Index of 57 (v4.1.1).

Don't-miss checklist

  • Write the acceptance criterion before the experiment (for example a 5 percent bound between two named scenarios).
  • Test every parallelism configuration at kernel level against the unpartitioned reference, not only the unpartitioned kernel.
  • Check FP32 matmuls for silent TF32 defaults in Triton and PyTorch before attributing drift to the algorithm.
  • Confirm that any native-extension call on a hot path releases the GIL when another Python thread must progress.
  • Validate microbenchmark wins end to end, and keep a human on numerical-semantics and concurrency changes.

Failure modes

  • A single end-to-end metric drop with no attributable layer; the agent cannot choose its next test.
  • Profiling with incomplete coverage leading to wrong attribution; Z.ai warns of this explicitly.
  • A microbenchmark gain that does not survive end-to-end load.
  • Asynchronous transfer support at the lower layer defeated by blocked submission in the layer above (the GIL case).
  • Precision loss that grows with context length and is invisible in short tests.

Open questions & validation

The script emulates the precision fix on numpy: TF32 is emulated by truncating the FP32 mantissa to 10 bits (hardware rounds, so this is a pessimistic emulation), and tf32x3 splits each operand into high and low TF32 parts and sums three products. It shows that a chained merge like the CP update accumulates error with shard count under TF32 and stays orders of magnitude lower under tf32x3. This validates the mechanism on synthetic matrices; it does not reproduce KDA or any GLM number. Executed.

import numpy as np

def to_tf32(x: np.ndarray) -> np.ndarray:
    b = x.astype(np.float32).view(np.uint32) & np.uint32(0xFFFFE000)
    return b.view(np.float32)

def dot_tf32(a, b):
    return to_tf32(a).astype(np.float64) @ to_tf32(b).astype(np.float64)

def dot_tf32x3(a, b):
    a_hi, b_hi = to_tf32(a), to_tf32(b)
    a_lo, b_lo = to_tf32(a - a_hi), to_tf32(b - b_hi)
    f = lambda x, y: x.astype(np.float64) @ y.astype(np.float64)
    return f(a_hi, b_hi) + f(a_hi, b_lo) + f(a_lo, b_hi)

def rel(x, ref):
    return float(np.linalg.norm(x - ref) / np.linalg.norm(ref))

rng = np.random.default_rng(0)
a = rng.standard_normal((64, 64)).astype(np.float32)
b = rng.standard_normal((64, 64)).astype(np.float32)
ref = a.astype(np.float64) @ b.astype(np.float64)
e1, e3 = rel(dot_tf32(a, b), ref), rel(dot_tf32x3(a, b), ref)
print(f"single matmul  tf32 {e1:.2e}  tf32x3 {e3:.2e}")
assert e3 < e1 / 100

def chain(dot, n):
    rng = np.random.default_rng(1)
    Mc = (np.eye(32) + 0.02 * rng.standard_normal((32, 32))).astype(np.float32)
    M, Mr = np.eye(32, dtype=np.float32), np.eye(32)
    for _ in range(n):
        M = dot(Mc, M).astype(np.float32)
        Mr = Mc.astype(np.float64) @ Mr
    return rel(M.astype(np.float64), Mr)

prev = 0
for n in (4, 16, 64):
    c1, c3 = chain(dot_tf32, n), chain(dot_tf32x3, n)
    print(f"{n:3d} shards  tf32 {c1:.2e}  tf32x3 {c3:.2e}")
    assert c3 < c1 / 10
    assert c1 > prev
    prev = c1
i = np.eye(8, dtype=np.float32)
assert np.array_equal(dot_tf32(i, i), dot_tf32x3(i, i))
print("ok")

Executed output:

single matmul  tf32 7.75e-04  tf32x3 3.59e-07
  4 shards  tf32 2.83e-03  tf32x3 1.55e-06
 16 shards  tf32 1.32e-02  tf32x3 5.69e-06
 64 shards  tf32 5.11e-02  tf32x3 2.01e-05
ok

Not verifiable from the post: the accelerator vendor and model; absolute throughput, latency, and cost; the claim of parity with mainstream NVIDIA GPUs; the 62 trillion token and most-used-model claims; the Figure 1 and Figure 4 data (figures were not viewable in the extracted text); the DeepEP and Mooncake changes (no link given); and the agent's architecture, prompts, and human-review effort. The post is a vendor account of its own result.

References

  • Z.ai, "Toward Recursive Self-Improvement: How GLM Built Its Own Inference Infrastructure": https://z.ai/blog/glm-built-its-inference-infrastructure (URL returned HTTP 200; article text recovered from the site's bundle, not the rendered page).
  • Flash Linear Attention PR #1180, "[CP] use tf32x3 affine chain in kcp": https://github.com/fla-org/flash-linear-attention/pull/1180 (state and merge date confirmed through the GitHub API).
  • DeepEP: https://github.com/deepseek-ai/DeepEP
  • Mooncake: https://github.com/kvcache-ai/Mooncake
  • Triton: https://github.com/triton-lang/triton
  • Ornn Data, "The Economics of Open-Weight Inference": https://data.ornn.com/the-economics-of-open-weight-inference.pdf

Related: open-weight inference economics · disaggregated inference · KV cache transfer with NIXL · agentic kernel generation harness · self-improving agent harness · vLLM GLM-5.2 cookbook

  1. The post attributes "roughly 3x" end-to-end improvement to the stack of techniques, and separately says the agent loop took GLM-5.3-Flash to production readiness in under two weeks, "ultimately tripling end-to-end throughput relative to the initial baseline". The text does not state whether these are one gain or two; this page treats them as one. Similarly, the Figure 4 prose says a 9.6 percent cut on v1 and a 1.71x gain "over v2" without defining v2, and the figure data was not readable. The page was read from the site's JavaScript bundle because the HTML is a client-rendered shell; the figures' image files were not inspected. ↩