Skip to content
Markdown

How RAG paradigms scale

Scope: what happens to lexical, dense, graph-based, and agentic retrieval as the corpus grows while the workload stays fixed, and what that implies for sizing a retrieval tier on real infrastructure. This page covers the controlled ladder that makes the comparison meaningful, where the accuracy crossover is observed, the construction cost that decides whether a graph index is buildable at all, and the retrieval control that changes both the search interface and its tool instructions. The mechanism-level comparison of retrieval against cached context is RAG versus CAG; the agent-side view of the same file-search loop is the filesystem as agent memory; the serving-side cost of long prompts is prompt caching.

Primary source: Wang, Xu, Wang, Zeng, Wu, Zhang, Zhang, "BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms", arXiv:2607.26497v2 [cs.CL], 30 July 2026 (USTC, Metastone Technology, Beijing Academy of Agriculture and Forestry Sciences). Benchmark: EnterpriseRAG-Bench (arXiv:2605.05253).

What this page adds. The Python block is executed and asserted (Python 3.11, numpy 2.x). It checks the shared-tier crossover, four evaluated amortization horizons, endpoint-anchored build estimates, construction coverage, and the full-scale retrieval control. The endpoint estimates are not reproductions of the fitted power laws because the paper does not publish their fitted coefficients. No retrieval system was run and no benchmark number was reproduced by re-execution; every score quoted is the paper's.

One reader, one judge, one corpus. Qwen3.6-27B at temperature zero served by vLLM reads for every paradigm, and the File-System Agent uses the same model as its policy. Conclusions are about paradigms under a fixed reader on an enterprise-shaped corpus, not about retrieval in general.

What it is

The study fixes everything except corpus size. It builds a ladder of 28 strictly nested tiers, spaced at approximately 1.25x, running from 1,144 to 511,959 documents (1.7M to 601M tokens), and verifies nesting through manifest checksums so an incremental index builder can extend from one tier to the next.1

The smallest tier, the bedrock, is constructed to contain the benchmark's 722 annotated gold documents for the 500 questions, 326 mined traps, 99 lures, and two organisational scaffold pages, deduplicated to 1,144. Traps are documents about the same entity or topic as a gold document that report the wrong version, date, or decision. Lures serve the 20 unanswerable questions so that "not found" cannot be solved by the mere absence of retrieved text. EnterpriseRAG-Bench treats gold sets as revisable hypotheses because exhaustive evidence annotation is infeasible.2

Every larger tier adds only background documents to that fixed annotated bedrock. Accuracy changes therefore measure retrieval under a growing haystack against the benchmark's current evidence labels; an unannotated relevant background document can still exist.

Seven pipelines run on that ladder: BM25 over an inverted index, DenseRAG over chunk embeddings, HippoRAG 2, MS-GraphRAG, LightRAG, LinearRAG, and a File-System Agent that has no index at all and explores the raw per-source file tree with read-only list, search, and read tools under a budget of 80 LLM calls per question. Each main-ladder system runs once per question; the reported intervals bootstrap questions and do not measure stochastic reruns. The file loop is the same basic retrieval pattern used by coding agents.

Why use it

Because the usual comparison is uninformative. Each paradigm is normally evaluated on its own benchmark at one corpus size, so the published record cannot answer the only question a platform team has: at this corpus size and growth rate, which retrieval tier should the team build?

The controlled answer is a scale-dependent crossover, not a winner.

Documents 1,144 2,254 6,980 21,614 42,587 66,932 131,876 511,959
BM25 74.7 71.4 70.1 64.9 61.2 59.5 55.2 50.5
File-System Agent 77.4 75.4 69.9 62.6 58.9 56.7 50.9 30.7
DenseRAG 58.1 55.7 51.0 44.2 40.7 38.1 36.0 29.9
HippoRAG 2 66.2 63.1 58.6 53.8 50.5 not evaluated 41.0 not evaluated
LinearRAG 46.2 44.1 38.8 34.3 31.3 30.9 29.8 not evaluated
MS-GraphRAG 45.9 44.0 38.4 not evaluated not evaluated not evaluated not evaluated not evaluated
LightRAG 48.0 42.5 not evaluated not evaluated not evaluated not evaluated not evaluated not evaluated

Two things in that table matter more than the ordering. The agentic column loses 46.7 points across the ladder while the lexical column loses 24.2. Four pipelines also lose rows. In the source table, a dash means that the index was not built or the tier was not evaluated, so the table alone cannot distinguish those cases.

When to use it (and when not)

Default to BM25 for enterprise-shaped corpora. It anchors the cheap end of the reported accuracy-cost frontier, needs no generative model calls at construction time, and its query cost is nearly flat in corpus size because the reader prompt dominates. The executed check below confirms that BM25 is undominated in generative model tokens at the four evaluated horizons: 10, 100, 1,000, and 10,000 queries. It is not a claim about embedding cost, latency, hardware, or horizons that the paper did not test.

Use agentic search after ranked discovery, not instead of it. A full-scale control changes raw-file tools to ranked lexical search plus chunk reading. It also supplies tool-specific instructions for the new interface, so it does not isolate a single primitive. Under that combined change, the score rises from 36.9 to 69.4, past one-shot BM25's 54.8, while calls fall from 36.1 to 5.8 and model tokens per question from 895K to 101K. The main text describes adapted instructions, while Appendix Table 13 says the prompt was held fixed; the source is internally inconsistent on this point.3

Use raw-file agency only on genuinely small corpora, and know what you are paying. At the bedrock the File-System Agent has the best point estimate, and it leads BM25 on intra-document, project-related, completeness, and conflicting-information questions at N = 42,587 (56 against 27 on completeness). It costs 39x BM25's query tokens to do it, rising to 60x by N = 21,614.

Treat LLM-extracted graph construction at 10^5 to 10^6 documents as a capacity-planning project. Near-linear measured growth is necessary but does not establish operability: throughput, parallel efficiency, checkpointing, rebuild cadence, and service limits still decide whether the build completes. In this study, the completed graph tiers also score below BM25.

Do not read these results as generic. Enterprise questions carry precise lexical anchors (ticket IDs, product names, dates) and the traps are semantically similar but factually wrong, which favors exact matching over embedding similarity. In the bedrock paraphrase control, the File-System Agent already leads BM25 before paraphrasing; paraphrasing widens the lead to 73.3 against 63.9 rather than reversing the ordering.

Architecture

flowchart TB
  subgraph LADDER["Nested corpus ladder, 28 tiers, 1.25x per rung"]
    BED["Bedrock 1,144 docs<br/>722 gold + 326 traps<br/>+ 99 lures + 2 scaffolds"]
    BG["+ stratified background prefix<br/>up to 511,959 docs / 601M tokens"]
    BED --> BG
  end
  subgraph OFFLINE["Construction cost (paid once, scales with corpus)"]
    NOIDX["BM25 inverted index<br/>File-System raw tree<br/>0 LLM tokens"]
    EMB["DenseRAG: embeddings<br/>LinearRAG: local NER + embeddings"]
    GEN["HippoRAG 2 / MS-GraphRAG / LightRAG<br/>generative extraction per chunk"]
  end
  subgraph ONLINE["Query cost (paid per question)"]
    FLAT["BM25, DenseRAG, HippoRAG 2<br/>near-flat model tokens per query"]
    SEQ["File-System Agent<br/>226K to 343K main-ladder model tok/q"]
  end
  LADDER --> OFFLINE --> ONLINE
  ONLINE --> READER["Shared reader: Qwen3.6-27B, T=0<br/>top-5 where the paradigm returns ranked chunks"]

The diagram is the whole thesis: the paradigms differ in where the cost lands, and corpus growth moves the accuracy advantage toward whichever paradigm amortizes global ranking into an index.

How to use it

Convert the measured tier to your own units before applying it. BM25 first leads at 6,980 documents, which the paper reports directly as 8.5M corpus tokens; the preceding tier has 2,254 documents and 3.0M tokens. The abstract's "roughly 10M tokens" is a rounded regime marker, not a fitted crossover threshold. Across the ladder the mean document length ranges from about 1,173 to 1,473 tokens, so transfer through token count and corpus shape rather than document count alone.

Meter build and query tokens separately, per paradigm, per question. The study's metering layer intercepts every LLM call and attributes prompt and completion tokens to build or query, counting embedding calls separately because they carry no prompt template. Without that split the graph paradigms look competitive: their per-question cost is 6.5K to 10.4K, in line with BM25's 5.8K. The entire difference is offline.

Budget the agent loop by calls and by exhaustion rate, not by average tokens. Median LLM calls rise from 5 to 8 by N = 42,587 while budget exhaustion goes from below 7% to 15% at N = 131,876 and 31% at full scale. Accuracy also falls among questions that finish inside budget, so truncation alone does not explain the loss. The paper does not test a higher call cap.

Treat top-k as a lever that needs its own cost measurement. On 148 to 150 sampled questions, the bedrock depth control reports BM25 at 83.0 with top-10 against 74.7 at top-5, and DenseRAG at 70.0 against 58.1. The incremental reader cost and latency are not reported, so the control supports an accuracy gain, not a claim that it is cheap. Reader context and prompt caching economics still bound the choice.

Executed audit

import numpy as np

DOCS = np.array([1144, 2254, 6980, 21614, 42587, 66932, 131876, 511959])
TOKENS = np.array([1.7e6, 3.0e6, 8.5e6, 25.6e6, 50.2e6, 78.6e6, 154.7e6, 600.8e6])
BM25 = np.array([74.7, 71.4, 70.1, 64.9, 61.2, 59.5, 55.2, 50.5])
FSA = np.array([77.4, 75.4, 69.9, 62.6, 58.9, 56.7, 50.9, 30.7])
QTOK = {"BM25": 5.8e3, "File-System Agent": 226e3, "DenseRAG": 4.9e3, "HippoRAG 2": 6.5e3,
        "LinearRAG": 5.8e3, "MS-GraphRAG": 10.4e3, "LightRAG": 8.5e3}
GEN_BUILD = {"BM25": 0.0, "File-System Agent": 0.0, "DenseRAG": 0.0, "LinearRAG": 0.0,
             "HippoRAG 2": 7.5e6, "MS-GraphRAG": 35.1e6, "LightRAG": 34.6e6}
BED = {"BM25": 74.7, "File-System Agent": 77.4, "DenseRAG": 58.1, "HippoRAG 2": 66.2,
       "LinearRAG": 46.2, "MS-GraphRAG": 45.9, "LightRAG": 48.0}

# 1. The first measured shared tier where the point estimates cross is 8.5M tokens.
delta = BM25 - FSA
first = int(np.argmax(delta > 0))
assert (DOCS[first], TOKENS[first]) == (6980, 8.5e6) and delta[first - 1] < 0
assert 19.0 < delta[-1] < 20.0

# 2. Bedrock generative model-token totals over the 500-question workload.
tot = {k: GEN_BUILD[k] + 500 * QTOK[k] for k in QTOK}
assert abs(tot["BM25"] - 2.9e6) < 5e4
assert abs(tot["File-System Agent"] - 112.8e6) < 3e5
assert abs(tot["HippoRAG 2"] - 10.75e6) < 5e4
# DenseRAG and LinearRAG also incur separately metered embedding or local-NER work.

# 3. BM25 is undominated in model tokens at the four horizons the paper evaluates.
def dominators(target, horizon):
    def cost(system):
        return GEN_BUILD[system] + horizon * QTOK[system]
    return [
        system for system in BED
        if system != target
        and cost(system) <= cost(target)
        and BED[system] >= BED[target]
        and (cost(system) < cost(target) or BED[system] > BED[target])
    ]
for h in (10, 100, 1000, 10000):
    assert not dominators("BM25", h)

# 4. Endpoint-anchored estimates use the reported exponent but not the unpublished fitted a.
FULL = 600.8e6
def endpoint_estimate(measured, at_x, b):
    return measured * (FULL / at_x) ** b
estimates = np.array([
    endpoint_estimate(724e6, 154.7e6, 1.01),
    endpoint_estimate(190e6, 10.6e6, 0.92),
    endpoint_estimate(73e6, 3.0e6, 1.36),
    endpoint_estimate(73e6, 3.0e6, 1.20),
]) / 1e9
np.testing.assert_allclose(estimates, [2.8502, 7.7966, 98.5169, 42.1942], rtol=2e-5)
assert abs(estimates[0] * 1e9 / 11.9e3 / 86400 - 2.77) < 0.01
assert abs(estimates[1] * 1e9 / 1.8e3 / 86400 - 50.13) < 0.01
assert abs(estimates[2] * 1e9 / 0.8e3 / 86400 / 365 - 3.90) < 0.01

# 5. Maximum evaluated construction coverage.
coverage = {"HippoRAG 2": 131876, "LinearRAG": 131876,
            "MS-GraphRAG": 8750, "LightRAG": 2254}
assert 0.25 < coverage["HippoRAG 2"] / 511959 < 0.26
assert coverage["MS-GraphRAG"] / 511959 < 0.02
assert coverage["LightRAG"] / 511959 < 0.005

# 6. Paired bootstrap intervals support no bedrock difference and a full-scale lead.
bedrock_ci = np.array([-11.57, 0.15])
full_scale_ci = np.array([9.68, 26.14])
assert bedrock_ci[0] < 0 < bedrock_ci[1]
assert np.all(full_scale_ci > 0)

# 7. Full-scale retrieval-and-instruction control (150 questions, one shared rejudge).
native_bm25, file_system, agent_bm25 = 54.8, 36.9, 69.4
assert agent_bm25 - file_system > 32
assert agent_bm25 - native_bm25 > 14
assert abs(895e3 / 101e3 - 8.86) < 0.1
assert 36.12 / 5.79 > 6

print("all assertions passed")

Five results follow.

Endpoint arithmetic approximates, but does not reproduce, the fitted projections. Anchoring each reported exponent at the largest rounded measurement gives 2.85B generative tokens for HippoRAG 2, 7.80B for MS-GraphRAG, and 98.52B for LightRAG. The paper reports fitted estimates of 2.9B, 7.9B, and about 102B. The difference is expected because the fitted coefficient and unrounded measurements are not published. These projections extend as far as about 57x for MS-GraphRAG and 200x for LightRAG beyond their largest measured corpus, with no fit-parameter uncertainty reported. At the measured single-instance throughput, the endpoint estimates correspond to about 2.8 days, 50 days, and 3.9 years. Parallelism can shorten calendar time but its efficiency is unmeasured.

Construction coverage is a separate deployability result. HippoRAG 2 and LinearRAG are evaluated through 25.8% of the corpus, MS-GraphRAG through 1.71%, and LightRAG through 0.44%. Scores at those tiers measure answer accuracy conditional on a usable index; they do not measure construction coverage. The paper leaves unsupported tiers missing in the main curves and assigns zero only in a separately labelled coverage-adjusted summary.

BM25 is Pareto-undominated at the four evaluated model-token horizons. No reported pipeline has at least its bedrock score for no more generative model tokens at 10, 100, 1,000, or 10,000 queries. DenseRAG and LinearRAG have zero generative construction tokens but nonzero embedding or local-NER work; the paper's cross-scale plot applies a model-size weighting to embedding tokens. The check therefore supports a model-token frontier at four horizons, not a total-infrastructure frontier.

The crossover is an observed neighboring-tier change, not a precise threshold. BM25 trails by 4.0 points at 3.0M tokens and leads by 0.2 at 8.5M. The paired File-System Agent versus BM25 interval includes zero at the bedrock, [-11.57, 0.15], while the full-scale interval, [9.68, 26.14] when expressed as the BM25 lead, is wholly positive. Judge and protocol robustness ranges describe shifts in individual cells under different scoring procedures; they are not a minimum resolvable pairwise gap and should not be converted into one.

Candidate discovery is the leading observed mechanism. At full scale the File-System Agent's any-gold hit rate is 39.0% against BM25's 71.6%. Conditional on finding a gold document, their scores are 85.9 and 73.8, but that selected subset does not establish that the agent reasons better. The combined ranked-search interface and instruction change raises hit rate to 78.0% and score to 69.4. This supports ranked discovery as a useful input to agency, while the control does not isolate search from its tool-specific instructions.

How to develop with it

Harness choice is a first-order measured variable. In a separate same-session 150-question control, three file-agent harnesses using the same policy model, raw corpus, and judge score 86.3, 82.3, and 43.9. These values are descriptive for that control and must not be mixed numerically with the main-ladder cells from another scoring session. They still show that a claim about agentic search applies to a specific harness. The design space is covered in agent harness architecture.

The same index supports very different outcomes under an agent than under its native ranker. Exposing each graph index through typed read-only tools (semantic search, neighborhood expansion, Personalized PageRank, chunk reading) and running one fixed tool-calling harness over it changes native scores by +22.2 to +29.9 points for LightRAG, -0.6 to +14.5 for HippoRAG 2, and -6.7 to +0.4 for MS-GraphRAG. Substrate and access policy are separable, and the win is not uniform.

Nested tiers are the reusable artifact. The construction is simple (fix the bedrock, order the background once with a seeded source-and-noise-stratified permutation, take prefixes) and it makes incremental index builders measurable, because the marginal build cost from tier to tier is exactly what you meter. If you run a retrieval tier in production, this is the cheapest honest way to answer "what happens when the corpus doubles".

How to run it in production

Size the retrieval tier by where cost lands, not by paradigm name. BM25, DenseRAG, and HippoRAG 2 use 4.9K to 6.5K model tokens per question and stay nearly flat with corpus size. The main File-System Agent ladder rises from 226K to 343K. The 895K figure belongs to a separate 150-question full-scale resweep that includes failed and retried attempts, not the 500-question main ladder. These counts include prompt and completion model tokens; they are not generation-only throughput or a hardware sizing result. The Agent+BM25 control uses 101K in that same resweep.

Set an explicit call budget and alert on the exhaustion rate. Budget exhaustion crossing from single digits to 31% is the leading indicator that a raw-file agent has outgrown its corpus, and it arrives before the accuracy collapse is obvious in aggregate scores.

Prefer incremental builders and record marginal build cost per tier. Fit construction work across several measured tiers and retain the coefficient, uncertainty, throughput, parallel efficiency, checkpoint recovery time, and rebuild cadence. LightRAG's fitted b = 1.36 is already visible across its measured tiers and the paper attributes it to repeated entity-merge rewrites. An exponent alone does not determine production feasibility.

Non-generative construction is the lower-model-cost middle. DenseRAG completes the full corpus with 659.4M embedding tokens and zero generative build calls. LinearRAG builds an entity co-occurrence graph with local NER plus the shared embedding model and no generative calls, finishing within 1.8 points of MS-GraphRAG and LightRAG at the bedrock. On this corpus, that result suggests the measured generative extraction cost grows faster than the answer-score benefit.

Watch abstention on unanswerable questions. One-shot readers decline when nothing supports an answer; iterative exploration can still produce an unsupported answer. The paper establishes unsupported outputs, not their confidence. Track abstention and evidence support separately from aggregate score. See LLM judge reliability.

How to maintain it

Re-run the ladder when the corpus composition changes, not on a schedule. The result that matters is the direction of the accuracy curve as background documents accumulate, and that direction depends on how lexically anchored your questions are. Keep the bedrock fixed across re-runs, or the comparison is meaningless.

Score everything in one session per comparison, and never mix sessions into one table. The 1.22-point drift on a fixed cell between sessions is small but it is the same order as several published orderings.

Keep a paraphrase control in the suite. BM25's advantage on this corpus comes partly from exact lexical anchors; the paraphrase control is the cheapest early warning that your question distribution has drifted away from that regime.

Open questions and validation

  • One synthetic corpus, one shape. EnterpriseRAG-Bench models a fictional company serving LLM inference, with 7.7% of documents natively marked as filing noise. Corpora without precise lexical anchors, or with heavy paraphrase between question and evidence, are untested. The small-scale paraphrase control widens an existing File-System Agent lead but does not establish the full-ladder behavior.
  • One reader, one policy model. The File-System Agent uses the same 27B model as the reader. Whether a stronger policy model changes the slope of the agentic curve, rather than its intercept, is unmeasured.
  • Hybrid retrieval is absent. No sparse-plus-dense fusion, no reranker, no learned sparse retriever. The comparison is between paradigms, not between the best available pipeline.
  • Agent+BM25 is a two-point control, not a ladder. It is measured at the bedrock and at full scale only, so the shape of its curve between them is unknown.
  • Single-stream latency. All latency figures are single-stream on an idle server, which says nothing about a retrieval tier under concurrency.
  • Gold labels are incomplete by construction. EnterpriseRAG-Bench says exhaustive annotation is infeasible and treats its pooled gold sets as revisable hypotheses. Added background can contain unlabelled useful evidence, so exact document recall is recall against the current label set.
  • Negative-document proposals are partly lexical. BM25 supplies candidates to the trap and lure mining pools, while dense retrieval reranks BM25-prefiltered pools. A direct full-corpus DenseRAG audit over 90 questions reduces, but does not remove, the possibility that benchmark construction favors lexical retrieval.
  • One run per system and question. Question-level bootstrapping estimates sampling variability over the benchmark questions, not variability from repeated model runs.

Failure modes

  • Comparing paradigms at one corpus size. A single tier cannot show a crossover; the selected tier can decide the point-estimate winner.
  • Reading a graph system's score without construction coverage. A dash means unbuilt or unevaluated, not an observed answer failure. Completed-tier accuracy and usable-index coverage are separate metrics.
  • Budgeting an agentic retriever by average tokens. The interesting companion metric is the fraction of questions hitting the call cap, which grows with the corpus.
  • Assuming more calls fixes the collapse. Accuracy falls among within-budget questions too, but the study does not test a higher cap.
  • Turning protocol shifts into a significance threshold. Judge and scoring-protocol shifts are not a paired minimum-detectable effect. Use the reported paired bootstrap interval for a pairwise claim.
  • Extrapolating a builder from one endpoint. The reported power laws use multiple tiers; endpoint arithmetic does not recover the unpublished fitted coefficient or its uncertainty.
  • Treating non-generative construction as free. DenseRAG and LinearRAG still pay for embeddings, local NER, CPU, storage, and indexing.
  • Letting the agent replace ranking rather than consume it. Sequential local exploration loses more accuracy than BM25 as this search space grows; BM25 also loses 24.2 points, so global ranking is more resistant rather than invariant.

References

Related: RAG vs CAG · The filesystem as agent memory · Agent harness architecture · Agentic context management · Agent context and memory · Prompt caching · LLM judge reliability · Agent evaluation · Agent tools and function calling · Hierarchical agent decomposition · Agent loop economics · Glossary

  1. The scaling paper states that the source corpus has 511,957 documents and adds two scaffold pages for 511,959. EnterpriseRAG-Bench's own Table 9 reports 511,962. The sources do not reconcile the three-document difference. ↩

  2. EnterpriseRAG-Bench documents the annotation as iterative evidence discovery rather than an exhaustive relevance judgment. This limits claims that every relevant document is fixed in the bedrock. ↩

  3. Section 5.1 and Appendix C.3 describe an adapted BM25 tool interface and its system message. Appendix Table 13 instead lists the prompt as fixed. The page treats the intervention as interface plus instructions because that is the more conservative interpretation. ↩