Skip to content
Markdown

How RAG paradigms scale

Scope: what happens to lexical, dense, graph-based, and agentic retrieval as the corpus grows while the workload stays fixed, and what that implies for sizing a retrieval tier on real infrastructure. This page covers the controlled ladder that makes the comparison meaningful, where the accuracy crossover actually sits and how firmly, the construction cost that decides whether a graph index is buildable at all, the retrieval-primitive control that separates agency from substrate, and the resolution floor below which none of the orderings survive. The mechanism-level comparison of retrieval against cached context is RAG versus CAG; the agent-side view of the same file-search loop is the filesystem as agent memory; the serving-side cost of long prompts is prompt caching.

Primary source: Wang, Xu, Wang, Zeng, Wu, Zhang, Zhang, "BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms", arXiv:2607.26497v2 [cs.CL], 30 July 2026 (USTC, Metastone Technology, Beijing Academy of Agriculture and Forestry Sciences). Benchmark: EnterpriseRAG-Bench (arXiv:2605.05253).

What this page adds. The Python block is executed and asserted (Python 3.11, numpy 2.x). It reproduces the paper's four headline build-cost extrapolations from their fitted exponents, verifies the Pareto claim across the stated amortization range, converts the crossover tier from documents to corpus tokens, and derives a cross-session resolution floor that the paper's own numbers supply but do not state. No retrieval system was run and no benchmark number was reproduced by re-execution; every score quoted is the paper's.

One reader, one judge, one corpus. Qwen3.6-27B at temperature zero served by vLLM reads for every paradigm, and the File-System Agent uses the same model as its policy. Conclusions are about paradigms under a fixed reader on an enterprise-shaped corpus, not about retrieval in general.

What it is

The study fixes everything except corpus size. It builds a ladder of 28 strictly nested tiers, each 1.25x the last, running from 1,144 to 511,959 documents (1.7M to 601M tokens), and verifies nesting through manifest checksums so an incremental index builder can extend from one tier to the next.

The smallest tier, the bedrock, is constructed to contain every document that could matter: the 722 gold documents for the 500 questions, 326 mined traps, 99 lures, and two organisational scaffold pages, deduplicated to 1,144. Traps are documents about the same entity or topic as a gold document that report the wrong version, date, or decision. Lures serve the 20 unanswerable questions so that "not found" cannot be solved by the mere absence of retrieved text.

Because all relevant and all adversarial material lives in the bedrock, every larger tier adds only background documents. Accuracy changes across tiers therefore measure exactly one thing: the difficulty of finding fixed evidence inside a growing haystack.

Seven pipelines run on that ladder: BM25 over an inverted index, DenseRAG over chunk embeddings, HippoRAG 2, MS-GraphRAG, LightRAG, LinearRAG, and a File-System Agent that has no index at all and explores the raw per-source file tree with read-only list, search, and read tools under a budget of 80 LLM calls per question. That last one is the paradigm most readers of this knowledge base actually operate: it is the same tool loop that coding agents use.

Why use it

Because the usual comparison is uninformative. Each paradigm is normally evaluated on its own benchmark at one corpus size, so the published record cannot answer the only question a platform team has: at our corpus size and growth rate, which retrieval tier should we build?

The controlled answer is a scale-dependent crossover, not a winner.

Documents 1,144 2,254 6,980 21,614 42,587 131,876 511,959
BM25 74.7 71.4 70.1 64.9 61.2 55.2 50.5
File-System Agent 77.4 75.4 69.9 62.6 58.9 50.9 30.7
DenseRAG 58.1 55.7 51.0 44.2 40.7 36.0 29.9
HippoRAG 2 66.2 63.1 58.6 53.8 50.5 41.0 not built
LinearRAG 46.2 44.1 38.8 34.3 31.3 29.8 not built
MS-GraphRAG 45.9 44.0 38.4 not built not built not built not built
LightRAG 48.0 42.5 not built not built not built not built not built

Two things in that table matter more than the ordering. The agentic column loses 46.7 points across the ladder while the lexical column loses 24.2. And four of the seven pipelines stop producing rows, not because they scored badly but because their index never finished building.

When to use it (and when not)

Default to BM25 for enterprise-shaped corpora. It anchors the cheap end of the accuracy-cost frontier, needs no LLM at construction time, and its query cost is flat in corpus size because the reader prompt dominates. The paper's practical guidance is exactly this, and the executed check below confirms BM25 is undominated at every amortization horizon from 10 to 10,000 queries.

Use agentic search after ranked discovery, not instead of it. The paper's cleanest experiment holds the model, harness, budget, answer constraints, question set, and judge fixed, and swaps only the retrieval primitive from raw-file tools to ranked lexical search plus chunk reading. At full scale that single change takes the agent from 36.9 to 69.4, past one-shot BM25's 54.8, while cutting calls from 36.1 to 5.8 and tokens per question from 895K to 101K.

Use raw-file agency only on genuinely small corpora, and know what you are paying. At the bedrock the File-System Agent has the best point estimate, and it leads BM25 on intra-document, project-related, completeness, and conflicting-information questions at N = 42,587 (56 against 27 on completeness). It costs 39x BM25's query tokens to do it, rising to 60x by N = 21,614.

Do not build an LLM-extracted graph index at 10^5 to 10^6 documents unless construction is near-linear and relational questions dominate. The construction arithmetic below is the binding constraint, and the accuracy at the tiers that do complete is below BM25 anyway.

Do not read these results as generic. Enterprise questions carry precise lexical anchors (ticket IDs, product names, dates) and the traps are semantically similar but factually wrong, which is close to the ideal case for exact matching and close to the worst case for embedding similarity. The paper tests this with a paraphrase control, where the File-System Agent leads BM25 (73.3 against 63.9 at the bedrock), so wording sensitivity is real and asymmetric.

Architecture

flowchart TB
  subgraph LADDER["Nested corpus ladder, 28 tiers, 1.25x per rung"]
    BED["Bedrock 1,144 docs<br/>722 gold + 326 traps<br/>+ 99 lures + 2 scaffolds"]
    BG["+ stratified background prefix<br/>up to 511,959 docs / 601M tokens"]
    BED --> BG
  end
  subgraph OFFLINE["Construction cost (paid once, scales with corpus)"]
    NOIDX["BM25 inverted index<br/>File-System raw tree<br/>0 LLM tokens"]
    EMB["DenseRAG / LinearRAG<br/>embedding only"]
    GEN["HippoRAG 2 / MS-GraphRAG / LightRAG<br/>generative extraction per chunk"]
  end
  subgraph ONLINE["Query cost (paid per question)"]
    FLAT["one-shot pipelines<br/>4.9K to 10.4K tok/q, flat in N"]
    SEQ["File-System Agent<br/>226K to 343K tok/q, grows with N"]
  end
  LADDER --> OFFLINE --> ONLINE
  ONLINE --> READER["Shared reader: Qwen3.6-27B, T=0, top-5 chunks<br/>official judge, combined score"]

The diagram is the whole thesis: the paradigms differ in where the cost lands, and corpus growth moves the accuracy advantage toward whichever paradigm amortizes global ranking into an index.

How to use it

Convert the crossover from documents to your own units before applying it. The ladder is reported in documents, the finding in tokens. At the tier where BM25 first leads (6,980 documents), corpus size is between 8.2M and 10.4M tokens depending on which end of the corpus density you use, which brackets the stated 10M. If your documents are longer or shorter than this corpus's 1,174 to 1,486 tokens each, translate through tokens, not document counts.

Meter build and query tokens separately, per paradigm, per question. The study's metering layer intercepts every LLM call and attributes prompt and completion tokens to build or query, counting embedding calls separately because they carry no prompt template. Without that split the graph paradigms look competitive: their per-question cost is 6.5K to 10.4K, in line with BM25's 5.8K. The entire difference is offline.

Budget the agent loop by calls and by exhaustion rate, not by average tokens. Median LLM calls rise from 5 to 8 by N = 42,587 while budget exhaustion goes from below 7% to 15% at N = 131,876 and 31% at full scale. Accuracy also falls among questions that finished inside budget, so raising the call cap does not recover the loss.

Treat the top-k as a lever you can afford. The depth control shows BM25 at 83.0 with top-10 against 74.7 at top-5 on the bedrock, and DenseRAG at 70.0 against 58.1. Retrieval depth is a cheap accuracy knob relative to any of the architectural changes on this page, bounded by reader context and by prompt caching economics.

Executed audit

import numpy as np

DOCS = np.array([1144, 2254, 6980, 21614, 42587, 131876, 511959])
BM25 = np.array([74.7, 71.4, 70.1, 64.9, 61.2, 55.2, 50.5])
FSA  = np.array([77.4, 75.4, 69.9, 62.6, 58.9, 50.9, 30.7])
QTOK = {"BM25": 5.8e3, "File-System Agent": 226e3, "DenseRAG": 4.9e3, "HippoRAG 2": 6.5e3,
        "LinearRAG": 5.8e3, "MS-GraphRAG": 10.4e3, "LightRAG": 8.5e3}
BUILD = {"BM25": 0.0, "File-System Agent": 0.0, "DenseRAG": 0.0, "LinearRAG": 0.0,
         "HippoRAG 2": 7.5e6, "MS-GraphRAG": 35.1e6, "LightRAG": 34.6e6}
BED = {"BM25": 74.7, "File-System Agent": 77.4, "DenseRAG": 58.1, "HippoRAG 2": 66.2,
       "LinearRAG": 46.2, "MS-GraphRAG": 45.9, "LightRAG": 48.0}

# 1. Crossover tier, expressed in corpus tokens rather than documents.
delta = BM25 - FSA
first = int(np.argmax(delta > 0))
assert DOCS[first] == 6980 and delta[first - 1] < 0
LO, HI = 600.8e6 / 511959, 1.7e6 / 1144       # tokens/doc at full scale and at the bedrock
assert DOCS[first] * LO < 10e6 < DOCS[first] * HI
assert 19.0 < delta[-1] < 20.0                # "a margin approaching 20 points at full scale"

# 2. Amortized cost over the 500-question workload reproduces the paper's frontier points.
tot = {k: BUILD[k] + 500 * QTOK[k] for k in QTOK}
assert abs(tot["BM25"] - 2.9e6) < 5e4                 # paper: 74.7 at 2.9M
assert abs(tot["File-System Agent"] - 112.8e6) < 3e5  # paper: 77.4 at 112.8M
assert abs(tot["HippoRAG 2"] - 10.7e6) < 1e5          # paper: 66.2 at 10.7M

# 3. BM25 is undominated at every horizon the paper claims.
def dominators(target, horizon):
    cost = lambda k: BUILD[k] + horizon * QTOK[k]
    return [k for k in BED if k != target and BED[k] >= BED[target] and cost(k) < cost(target)]
for h in (10, 100, 1000, 10000):
    assert not dominators("BM25", h)

# 4. Build-cost power laws C(x) = a*x^b extrapolated to the 600.8M-token corpus.
FULL = 600.8e6
def extrapolate(measured, at_x, b):
    return measured * (FULL / at_x) ** b
assert abs(extrapolate(724e6, 154.7e6, 1.01) / 1e9 - 2.9) < 0.1     # HippoRAG 2
assert abs(extrapolate(190e6, 10.6e6, 0.92) / 1e9 - 7.9) < 0.2      # MS-GraphRAG
assert abs(extrapolate(73e6, 3.0e6, 1.36) / 1e9 - 102) < 5          # LightRAG
assert abs(extrapolate(73e6, 3.0e6, 1.20) / 1e9 - 42) < 2           # optimistic exponent
# and the wall-clock claims, from measured single-instance throughput.
assert abs(extrapolate(724e6, 154.7e6, 1.01) / 11.9e3 / 86400 - 3) < 0.3     # ~3 days
assert abs(extrapolate(190e6, 10.6e6, 0.92) / 1.8e3 / 86400 - 50) < 3        # ~50 days
assert abs(extrapolate(73e6, 3.0e6, 1.36) / 0.8e3 / 86400 / 365 - 4) < 0.3   # ~4 years

# 5. Coverage: what fraction of the corpus each graph builder ever indexed.
assert 8750 / 511959 < 0.02 and 2254 / 511959 < 0.005

# 6. Resolution floor. The same cell, scored in two independent sessions.
t1, t3 = 77.4, 76.181     # File-System Agent at the bedrock over all 500 questions
floor = max(abs(t1 - t3), 4.60, 3.56)   # plus the paper's own protocol- and judge-swap ranges
assert abs(delta[2]) < floor            # the crossover tier's 0.2-point margin is inside it
assert abs(delta[-1]) > 4 * floor       # the full-scale result is far outside it
inside = [int(n) for n, g in zip(DOCS, np.abs(delta)) if g < floor]
assert len(inside) == 6 and 511959 not in inside

# 7. Retrieval-primitive control at full scale (150 questions, one shared rejudge).
native_bm25, file_system, agent_bm25 = 54.8, 36.9, 69.4
assert agent_bm25 - file_system > 32     # paper: +32.52 [24.07, 40.90]
assert agent_bm25 - native_bm25 > 14     # paper: +14.56 [9.22, 20.12]
assert abs(895e3 / 101e3 - 8.86) < 0.1   # "roughly one ninth" of the query tokens
assert 36.12 / 5.79 > 6                  # and a sixth of the LLM calls

print("all assertions passed")

Five results follow.

Every headline extrapolation reproduces. From the fitted exponent, the largest measured build, and the measured single-instance throughput: HippoRAG 2 reaches 2.9B generative tokens and about 3 instance-days at full scale; MS-GraphRAG 7.8B and about 50 instance-days; LightRAG 98.5B and about 3.9 instance-years, falling to 42B under the paper's own optimistic b = 1.2. Parallelism buys calendar time, not total work.

The construction wall is a coverage problem, not an accuracy problem. HippoRAG 2 and LinearRAG index 25.8% of the corpus before stopping, MS-GraphRAG 1.71%, LightRAG 0.44%. Comparing their scores against BM25's is comparing quality conditional on completion. The paper handles this correctly by leaving unsupported tiers missing in the main curves and scoring them zero only in a separately labelled coverage-adjusted summary, which is the reporting convention worth copying.

BM25 is Pareto-undominated across the whole stated amortization range. No pipeline scores at least as well at the bedrock for fewer total tokens, at horizons of 10, 100, 1,000, or 10,000 queries. The check is trivial arithmetic, but it is the claim that actually justifies "default to BM25", and it holds because BM25's construction cost is zero LLM tokens.

The crossover location is soft; the full-scale result is not. Scoring the same cell (File-System Agent, bedrock, all 500 questions) in two independent sessions gives 77.4 and 76.181, 1.22 points apart. The paper's own robustness work moves cells further: binary rescoring shifts official correctness by -4.60 to +0.69, and an independent judge by -3.56 to +1.18. Taking the largest of those as a floor, six of the seven shared tiers have a BM25-versus-agent gap inside it, including the 0.2-point margin at the crossover tier itself. Only the 19.8-point full-scale gap clears it by a wide margin. This agrees with the paper, whose paired bootstrap at the bedrock is [-11.57, +0.15] and at full scale [+9.68, +26.14]; it is stated here because the seven-column table invites reading a smooth curve where the study supports two endpoints and a direction.

The mechanism is candidate discovery, and it is fixable without abandoning agency. At full scale the File-System Agent's any-gold hit rate collapses to 39.0% against BM25's 71.6%, yet on the queries where it does find a gold document it scores 85.9 against BM25's 73.8. It reasons better and finds worse. Giving it ranked lexical search raises the hit rate to 78.0% and the score to 69.4, above both. Sequential local exploration is not a substitute for global ranking; it is a good consumer of it.

How to develop with it

Harness quality is a first-order variable and it is measurable. On the same 150 questions, same policy model, same raw corpus, and same judge, three file-agent harnesses score 86.3, 82.3, and 43.9. A 42-point spread between harnesses on an identical substrate is larger than every paradigm difference on the ladder except the full-scale collapse. Any claim of the form "agentic search underperforms" is a claim about a specific harness until proven otherwise. The design space is covered in agent harness architecture.

The same index supports very different outcomes under an agent than under its native ranker. Exposing each graph index through typed read-only tools (semantic search, neighborhood expansion, Personalized PageRank, chunk reading) and running one fixed tool-calling harness over it changes native scores by +22.2 to +29.9 points for LightRAG, -0.6 to +14.5 for HippoRAG 2, and -6.7 to +0.4 for MS-GraphRAG. Substrate and access policy are separable, and the win is not uniform.

Nested tiers are the reusable artifact. The construction is simple (fix the bedrock, order the background once with a seeded source-and-noise-stratified permutation, take prefixes) and it makes incremental index builders measurable, because the marginal build cost from tier to tier is exactly what you meter. If you run a retrieval tier in production, this is the cheapest honest way to answer "what happens when the corpus doubles".

How to run it in production

Size the retrieval tier by where cost lands, not by paradigm name. One-shot pipelines cost 4.9K to 10.4K tokens per question dominated by the shared reader prompt, and that is flat in corpus size. An agentic tier costs 226K to 895K per question and grows. On a fleet, that is the difference between a retrieval service and a second inference cluster: at 895K tokens per question, 1,000 questions per hour is 895M tokens per hour of generation.

Set an explicit call budget and alert on the exhaustion rate. Budget exhaustion crossing from single digits to 31% is the leading indicator that a raw-file agent has outgrown its corpus, and it arrives before the accuracy collapse is obvious in aggregate scores.

Prefer incremental builders and record marginal build cost per tier. A near-linear builder is operable at scale; a super-linear one is not, and the exponent is measurable long before the corpus reaches production size. LightRAG's b = 1.36 comes from repeated entity-merge rewrites, and no amount of throughput fixes an exponent.

Non-generative construction is the affordable middle. DenseRAG completes the full corpus with 659.4M embedding tokens and zero generative build calls; LinearRAG builds an entity co-occurrence graph with a local NER plus the shared embedding model and no generative calls, finishing within 1.8 points of MS-GraphRAG and LightRAG at the bedrock. That last comparison is the paper's most useful negative result: on this corpus, LLM extraction adds cost faster than it adds signal.

Watch the abstention behaviour on unanswerable questions. One-shot readers decline when nothing supports an answer; iterative exploration can commit to an unsupported one. If your workload contains genuinely unanswerable queries, that difference shows up as confident wrong answers rather than as a lower score, which is the more expensive failure. See LLM judge reliability.

How to maintain it

Re-run the ladder when the corpus composition changes, not on a schedule. The result that matters is the direction of the accuracy curve as background documents accumulate, and that direction depends on how lexically anchored your questions are. Keep the bedrock fixed across re-runs, or the comparison is meaningless.

Score everything in one session per comparison, and never mix sessions into one table. The 1.22-point drift on a fixed cell between sessions is small but it is the same order as several published orderings.

Keep a paraphrase control in the suite. BM25's advantage on this corpus comes partly from exact lexical anchors; the paraphrase control is the cheapest early warning that your question distribution has drifted away from that regime.

Failure modes

  • Comparing paradigms at one corpus size. The crossover is invisible at any single tier, and the tier you happen to test decides the winner.
  • Reading a graph system's score without its coverage. A missing tier is an unbuilt index, not an observed answer failure, and averaging over completed tiers only flatters the builders that failed earliest.
  • Budgeting an agentic retriever by average tokens. The distribution has a hard cap; the interesting number is the fraction of questions hitting it, which grows with the corpus.
  • Assuming more calls fixes the collapse. Accuracy falls among within-budget questions too, so truncation is a symptom rather than the cause.
  • Treating a 2-to-4-point gap at one tier as a result. Protocol and judge swaps move cells by up to 4.6 points in this study's own robustness checks.
  • Deploying a super-linear index builder because it looked fine on a sample. b = 1.36 is invisible at 3M tokens and fatal at 600M.
  • Letting the agent replace ranking rather than consume it. Sequential local exploration degrades as the search space grows; global ranking does not.

Open questions and validation

  • One corpus, one shape. EnterpriseRAG-Bench models a fictional company serving LLM inference, with 7.7% of documents natively marked as filing noise. Corpora without precise lexical anchors, or with heavy paraphrase between question and evidence, are the untested case, and the paraphrase control already reverses the ordering at small scale.
  • One reader, one policy model. The File-System Agent uses the same 27B model as the reader. Whether a stronger policy model changes the slope of the agentic curve, rather than its intercept, is unmeasured.
  • Hybrid retrieval is absent. No sparse-plus-dense fusion, no reranker, no learned sparse retriever. The comparison is between paradigms, not between the best available pipeline.
  • Agent+BM25 is a two-point control, not a ladder. It is measured at the bedrock and at full scale only, so the shape of its curve between them is unknown.
  • Single-stream latency. All latency figures are single-stream on an idle server, which says nothing about a retrieval tier under concurrency.

References

Related: RAG vs CAG · The filesystem as agent memory · Agent harness architecture · Agentic context management · Agent context and memory · Prompt caching · LLM judge reliability · Agent evaluation · Agent tools and function calling · Hierarchical agent decomposition · Agent loop economics · Glossary