Markdown

RAG vs CAG: dynamic knowledge injection architecture¶

Scope: the two ways to give an LLM knowledge it was not trained with. RAG (retrieval-augmented generation) looks up relevant passages fresh, per query, from a retriever over an external corpus. CAG (cache-augmented generation) precomputes the KV cache for a whole knowledge base once and reuses it across queries with no retrieval step at all. Distinct from prompt caching and KV cache management (the systems-level prefix-caching mechanics CAG's cache reuse rides on, covered there in depth, not re-derived here) and from agent context memory (the narrower, agent-specific case of paging a vector store in and out of a single agent's working context). If a cache built for CAG is shared across multiple tenants, tenant cache isolation covers the confidentiality and fairness question that raises.

Vendor docs, paper results, and benchmark numbers below are quoted or paraphrased from the cited sources at research time; verify current status against the linked paper or repository before relying on it. The Python blocks are runnable and self-checking.

What it is¶

RAG: query -> retriever (vector DB, bi-encoder embeddings) -> optional cross-encoder re-ranker -> retrieved context -> LLM. Dynamic: every query re-searches the corpus, so the answer reflects whatever is in the index right now, and only the corpus's relevant slice ever reaches the model's context.
CAG: corpus -> one prefill pass over the whole corpus -> persisted KV cache -> query + cached state -> LLM, with the retrieval step removed entirely. Chan et al. describe the mechanism precisely: "preloading all relevant resources, especially when the documents or knowledge for retrieval are of a limited and manageable size, into the LLM's extended context and caching its runtime parameters," then answer subsequent queries "without additional retrieval steps."¹ Static: the knowledge baked into the cache is exactly what was there at build time, and staying current means rebuilding the cache, not re-indexing a document store.
Hybrid: a stable core (product documentation, policy text, a system prompt's worth of durable background) stays in a warm CAG-style cache; volatile, per-user, or per-session data (a live account balance, a just-uploaded document) is retrieved fresh via RAG and unified into one prompt alongside the cached core. Neither half tries to do the other's job.

Why it matters¶

Long context windows have not made retrieval obsolete, contrary to a common assumption. NVIDIA's own study found that "retrieval can significantly improve the performance of LLMs regardless of their extended context window sizes," and their retrieval-augmented Llama2-70B with a 32K window outperformed GPT-3.5-turbo-16k and Davinci003 on nine long-context tasks while also beating its own non-retrieval 32K baseline, and running faster at generation because it conditions on far fewer tokens per query.² A separate evaluation sharpens when each side wins, by query type rather than by how evidence happens to be spread through the corpus: long context generally wins on Wikipedia-based and narrative-topic questions, summarization-based retrieval matches it, chunk-based retrieval lags behind, and RAG holds the advantage on dialogue-based and general-question queries.³

CAG's own reported numbers are more nuanced than "no retrieval step" implies, and worth taking at face value rather than the marketing framing built up around the paper. On HotPotQA at their largest tested size (an 85k-token preloaded corpus), CAG's own measured total generation time was 2.2631 seconds, slower than sparse retrieval's combined 0.6675 seconds (0.0008s retrieval plus 0.6667s generation) and slower than dense retrieval's combined 1.3454 seconds.¹ Zero retrieval latency does not mean zero cost: conditioning generation on a much larger cached context is itself not free, and at that corpus size RAG's narrower per-query context won on raw latency in their own experiment. CAG's real advantage shows up where the paper frames it: "for certain applications, particularly those with a constrained knowledge base," where the whole corpus is small enough to preload cheaply and gets queried enough times to amortize that one-time cost.¹

When to use it (and when not)¶

Use RAG when:

The knowledge base is large, changes frequently, or both, so re-indexing new documents is cheaper than rebuilding a whole-corpus cache on every update.
The typical query is dialogue-based or a general question rather than a Wikipedia-style factual lookup, the query types where retrieval-augmented generation measurably wins over long-context alone.³
Per-query cost must stay low even at large corpus scale, since RAG's context (and therefore its generation cost) stays bounded to the retrieved top-k regardless of total corpus size.

Use CAG when:

The knowledge base is small and stable enough to preload affordably (the paper's own results are strongest at smaller preloaded sizes, not the 85k-token case above), and it does not change often enough to make cache rebuilds a constant tax.
The same cached corpus serves a high volume of repeat queries, so the one-time cache-build cost amortizes across many cheap, retrieval-free queries.
Retrieval failure modes (an embedding model that misses a paraphrase, a missing re-rank stage letting noisy chunks through) are a bigger risk to answer quality than the cost of preloading everything.

Use the hybrid pattern when a stable core and volatile, per-user data both belong in context and neither RAG nor CAG alone fits both: cache the core, retrieve the volatile part, and unify them in one prompt.

Skip both when the answer is already reliably in the model's own parametric knowledge and freshness or attribution do not matter: added retrieval or caching infrastructure is complexity with no corresponding benefit.

Architecture¶

flowchart TB
  subgraph RAG_PATH["RAG (dynamic)"]
    Q["User query"] --> EMB["Bi-encoder: embed query"]
    EMB --> VDB["Vector DB (FAISS / pgvector / Milvus)<br/>top-k by cosine similarity"]
    VDB --> RERANK["Cross-encoder re-rank<br/>(e.g. BGE reranker)"]
    RERANK --> CTX_R["Retrieved context"]
  end
  subgraph CAG_PATH["CAG (static, precomputed)"]
    CORPUS["Whole corpus<br/>(bounded, stable)"] --> PREFILL["One-time prefill pass"]
    PREFILL --> KVCACHE["Persisted KV cache"]
    KVCACHE -->|"reused every query,<br/>no retrieval step"| CTX_C["Cached context"]
  end
  CTX_R --> UNIFY["Unified prompt"]
  CTX_C --> UNIFY
  UNIFY --> LLM["LLM generates"]

The hybrid pattern is both subgraphs feeding the same unified prompt: the stable core arrives pre-cached with zero retrieval cost, the volatile slice arrives freshly retrieved, and only the volatile slice pays the retrieval-latency and staleness costs the RAG path incurs on its own.

How to build a RAG pipeline: retrieve, then rerank¶

A production retriever is two stages, not one, because the fast stage and the accurate stage want different architectures. A bi-encoder embeds the query and every document independently, so document embeddings are precomputed once and the query side is the only work done at request time; a vector index (FAISS is the reference implementation, an open-source library of indexing algorithms for vectors "of any size, up to ones that possibly do not fit in RAM," with GPU-accelerated search kernels⁶) makes that top-k lookup fast at corpus scale. A cross-encoder reads the query and a candidate document together and scores the pair jointly, capturing interaction terms an independent embedding cannot represent, at the cost of being too slow to run over an entire corpus. The house pattern is retrieve a wider top-k cheaply with the bi-encoder, then rerank only that shortlist with the cross-encoder. BAAI's bge-reranker-v2-m3 is a current, open-weight cross-encoder built on the bge-m3 base at 0.6B parameters, described as taking "question and document as input and directly output[ting] similarity instead of embedding," with "strong multilingual capabilities" evaluated on the MIRACL benchmark.⁷ Embedding-model choice for the first stage should not be by name recognition: MTEB benchmarks embeddings across "8 embedding tasks covering a total of 58 datasets and 112 languages," and its own headline finding is that "no particular text embedding method dominates across all tasks," so pick the model that wins on tasks resembling yours, not the model that tops an unrelated leaderboard.⁵

The property that justifies running two stages instead of one is that reranking can promote a document the bi-encoder under-ranked, because the bi-encoder's independent embeddings cannot see the interaction the cross-encoder can. This block builds a small, illustrative embedding space (not real trained vectors) with a topical near-miss that a bi-encoder ranks first and a truly relevant document it ranks second, then reranks with a joint score and confirms the correction, while checking reranking never reaches outside what retrieval already returned:

# retrieve_rerank.py -- validated: why a two-stage retrieve-then-rerank pipeline
# exists. A bi-encoder scores query and document independently (fast, precomputable,
# used for the first-pass top-k), so it cannot capture query-document interaction
# terms; a cross-encoder scores the pair jointly (slow, not precomputable, used only
# on the small top-k) and can recover a relevant document the bi-encoder under-ranks.
# Illustrative toy vectors, not real trained embeddings. numpy only.
import numpy as np

def cosine(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def bi_encoder_topk(query_vec, doc_vecs, k):
    """Bi-encoder retrieval: independent embeddings, cosine similarity, no
    query-document interaction term. This is what a precomputed vector index
    (FAISS, pgvector, ...) serves cheaply at query time."""
    scores = {doc_id: cosine(query_vec, v) for doc_id, v in doc_vecs.items()}
    return sorted(scores, key=scores.get, reverse=True)[:k], scores

rng = np.random.default_rng(0)
dim = 16
query_vec = rng.standard_normal(dim)

# Three candidate documents. "doc_decoy" is built to sit close to the query in
# embedding space (a topical near-miss) without truly answering it; "doc_true"
# is the genuinely relevant document but sits farther in embedding space because
# its true relevance depends on an interaction the bi-encoder cannot see.
doc_vecs = {
    "doc_decoy": query_vec + rng.normal(scale=0.05, size=dim),   # near-identical embedding
    "doc_true":  query_vec + rng.normal(scale=0.35, size=dim),   # farther, but truly relevant
    "doc_other": rng.standard_normal(dim),                        # unrelated
}

# An interaction term only a joint (query, doc) reader could assign: doc_true
# answers a specific clause in the query that pure embedding similarity misses;
# doc_decoy is topically close but does not actually answer it (a penalty).
INTERACTION_BONUS = {("q1", "doc_true"): 0.30, ("q1", "doc_decoy"): -0.20}

topk, bi_scores = bi_encoder_topk(query_vec, doc_vecs, k=3)

# Bi-encoder stage: the topical near-miss ranks first, the truly relevant
# document ranks behind it purely on embedding distance.
assert topk[0] == "doc_decoy", f"expected the bi-encoder to rank doc_decoy first, got {topk}"
assert "doc_true" in topk and "doc_decoy" in topk

# Cross-encoder rerank: joint (query, doc) scoring corrects the order using the
# interaction term the bi-encoder's independent embeddings could not express.
reranked = sorted(topk, key=lambda d: bi_scores[d] + INTERACTION_BONUS.get(("q1", d), 0.0),
                   reverse=True)
assert reranked[0] == "doc_true", f"reranking must promote the truly relevant doc, got {reranked}"

# Adversarial: reranking must be a permutation of the bi-encoder's own top-k, not
# a way to smuggle in a document the first stage never retrieved.
assert set(reranked) == set(topk), "rerank must only reorder the candidates retrieval already returned"

# Boundary: with no interaction terms at all, cross-encoder rerank reduces to the
# bi-encoder's own order (rerank must not invent structure that is not there).
no_interaction_rerank = sorted(topk, key=lambda d: bi_scores[d], reverse=True)
assert no_interaction_rerank == topk

print(f"OK: bi-encoder ranks {topk} (decoy first); cross-encoder rerank promotes "
      f"the truly relevant doc to {reranked}; rerank stays within the retrieved "
      f"set; with zero interaction terms rerank order matches bi-encoder order")

Run output:

OK: bi-encoder ranks ['doc_decoy', 'doc_true', 'doc_other'] (decoy first); cross-encoder rerank promotes the truly relevant doc to ['doc_true', 'doc_decoy', 'doc_other']; rerank stays within the retrieved set; with zero interaction terms rerank order matches bi-encoder order

For a shipping reference architecture rather than assembling one from parts, NVIDIA's RAG Blueprint packages an indexing pipeline (chunker, embedding NIM, vector store) and a query pipeline (retriever, an optional NeMo Retriever reranking NIM, generation); the reranker "reorders the retrieved passages, ensuring the most relevant chunks are selected to ground the response."⁸

How to build a CAG pipeline: precompute once, reuse the cache¶

CAG is an application pattern riding on the prefix-caching mechanics KV cache management and prompt caching already cover in depth: run one prefill pass over the whole corpus, persist the resulting KV state, and let every subsequent query hit that warm cache instead of re-encoding the corpus or running a retrieval step. Whether that persisted cache lives in HBM, host RAM, or a remote tier is exactly the tiering question those pages answer; this page's job is the decision to build the cache in the first place, not its storage mechanics.

The number that actually decides whether CAG pays off is not "does it skip retrieval," it is whether the one-time cache-build cost amortizes across enough repeat queries before RAG's flat per-query cost would have caught up. Part 1 below reproduces the paper's own HotPotQA-Large numbers exactly, confirming CAG is not the fastest option at that corpus size once generation time is counted. Part 2 is this page's own illustrative amortization model, not the paper's data, showing when a one-time build cost recovers itself and when it structurally cannot:

# rag_vs_cag_cost.py -- validated in two parts. Part 1 reproduces the actual
# reported per-query latencies from Chan et al., "Don't Do RAG: When
# Cache-Augmented Generation is All You Need for Knowledge Tasks" (arXiv
# 2412.15605), HotPotQA-Large (85k preloaded tokens): CAG is NOT universally
# faster than RAG once retrieval AND generation are both counted, contradicting
# the naive "CAG has zero retrieval step so it must be faster" assumption. Part 2
# is an ILLUSTRATIVE (not the paper's own numbers) amortization model: CAG pays a
# one-time cache-build cost separate from its per-query cost, so the question is
# how many repeat queries against the SAME cached corpus it takes to break even
# against RAG's flat per-query cost. Standard library only.

# --- Part 1: the paper's own reported numbers, HotPotQA-Large (seconds) ---
sparse_top3 = {"retrieval": 0.0008, "generation": 0.6667}
dense_top3  = {"retrieval": 0.4123, "generation": 0.9331}
cag         = {"retrieval": 0.0,    "generation": 2.2631}

def total_latency(method):
    return method["retrieval"] + method["generation"]

totals = {
    "sparse_top3": total_latency(sparse_top3),
    "dense_top3": total_latency(dense_top3),
    "cag": total_latency(cag),
}

# CAG has zero retrieval overhead, by construction.
assert cag["retrieval"] == 0.0

# But at this (large, 85k-token) corpus size, CAG is NOT the fastest option once
# generation time is counted: sparse retrieval's tiny retrieval cost plus a much
# shorter generation pass (over ~top-3 chunks, not the full 85k-token cache) beats
# CAG's total. This is the paper's own data, not a claim invented for this page.
fastest = min(totals, key=totals.get)
assert fastest == "sparse_top3", f"expected sparse_top3 fastest per the paper's own numbers, got {fastest}"
assert totals["cag"] > totals["sparse_top3"], "CAG's zero retrieval step does not make it fastest overall here"
assert totals["cag"] > totals["dense_top3"], "CAG is slower than dense RAG too at this corpus size"

# --- Part 2: illustrative amortization model (NOT the paper's numbers) ---
# CAG pays a one-time cache-build cost B (prefilling the whole corpus once);
# after that, each subsequent query against the SAME cached corpus costs only
# the per-query generation time above. RAG pays its flat per-query cost every
# time, with no separate build step. Break-even is the query count N where
# CAG's amortized total first drops below RAG's.
def cag_amortized_total(build_cost, per_query_cost, n_queries):
    return build_cost + per_query_cost * n_queries

def rag_flat_total(per_query_cost, n_queries):
    return per_query_cost * n_queries

def break_even_queries(build_cost, cag_per_query, rag_per_query, max_n=100_000):
    """Smallest N where CAG's amortized total <= RAG's flat total, or None if it
    never crosses within max_n (happens when cag_per_query >= rag_per_query)."""
    if cag_per_query >= rag_per_query:
        return None    # CAG never catches up if its own per-query cost isn't cheaper
    for n in range(1, max_n + 1):
        if cag_amortized_total(build_cost, cag_per_query, n) <= rag_flat_total(rag_per_query, n):
            return n
    return None

# Illustrative: a one-time 8-second cache build, CAG per-query cheaper than RAG
# once the cache exists (unlike Part 1's large-corpus case; this models a SMALL,
# constrained knowledge base where the paper's own small-corpus results show CAG
# competitive or ahead on quality with no retrieval step to pay for).
be = break_even_queries(build_cost=8.0, cag_per_query=0.3, rag_per_query=0.6675)
assert be is not None
assert cag_amortized_total(8.0, 0.3, be) <= rag_flat_total(0.6675, be)
assert cag_amortized_total(8.0, 0.3, be - 1) > rag_flat_total(0.6675, be - 1)  # not yet broken even one step earlier

# Adversarial: if CAG's own per-query cost is not cheaper than RAG's (Part 1's
# large-corpus regime), no amount of amortization ever recovers it.
assert break_even_queries(build_cost=8.0, cag_per_query=2.2631, rag_per_query=0.6675) is None

# Monotonicity: a larger one-time build cost can only push the break-even point
# later (more queries needed), never earlier.
be_cheap_build = break_even_queries(build_cost=2.0, cag_per_query=0.3, rag_per_query=0.6675)
be_expensive_build = break_even_queries(build_cost=20.0, cag_per_query=0.3, rag_per_query=0.6675)
assert be_cheap_build < be_expensive_build

print(f"OK Part 1 (paper's real numbers): totals={totals}; fastest={fastest}, "
      f"CAG is NOT fastest at this large-corpus size despite zero retrieval step. "
      f"OK Part 2 (illustrative): break-even at {be} queries when CAG's own "
      f"per-query cost is cheaper; never breaks even when it is not "
      f"(large-corpus regime); break-even query count grows with build cost "
      f"({be_cheap_build} -> {be_expensive_build})")

Run output:

OK Part 1 (paper's real numbers): totals={'sparse_top3': 0.6675, 'dense_top3': 1.3454000000000002, 'cag': 2.2631}; fastest=sparse_top3, CAG is NOT fastest at this large-corpus size despite zero retrieval step. OK Part 2 (illustrative): break-even at 22 queries when CAG's own per-query cost is cheaper; never breaks even when it is not (large-corpus regime); break-even query count grows with build cost (6 -> 55)

Re-derive Part 2's break-even against your own measured build cost and per-query costs before trusting it; the numbers here are illustrative, chosen only to show the shape of the trade-off.

How to build the hybrid¶

Cache the stable core, retrieve the volatile part, and unify both into one prompt rather than forcing either pattern to cover ground it is not suited for. This fuses RAG and long-context conditioning at the prompt level, both present in every request; Li et al.'s Self-Route hybridizes at the query level instead: they find long-context LLMs generally beat RAG on quality when adequately resourced, but RAG remains far cheaper, so Self-Route routes each query wholesale to either RAG or LC based on the model's own self-reflection, trading a small quality gap for most of RAG's cost savings rather than paying for both on every request.⁴ The two hybrid shapes solve different problems and are not substitutes for each other: this page's cache-plus-retrieve pattern is for when a request genuinely needs both a stable core and fresh volatile data at once, Self-Route's per-query routing is for when a single request would be well served by either RAG or long-context alone and the choice is about cost, not content coverage. In this page's pattern the "long-context" half is the warm CAG cache holding the stable core, and the "retrieval" half is an ordinary RAG lookup over whatever changes too often or is too user-specific to bake into a shared cache. Watch for the two halves surfacing the same fact twice, wasting context budget on duplication, and for the retrieved slice silently contradicting something the cached core already asserts; neither failure is caught by either half alone, only by treating the unified prompt as one artifact to validate.

Failure modes¶

Retrieval failure modes going undiagnosed. An embedding model that misses a paraphrase, or a missing re-rank stage letting a topically-close but wrong chunk reach the LLM, both degrade answer quality silently; MTEB's own finding that no embedding model dominates every task type means the default choice is not automatically the right one for your domain.⁵
Assuming CAG is always faster because it skips retrieval. The paper's own large-corpus numbers show the opposite; validate the actual break-even for your corpus size and query volume, per the Part 1/Part 2 model above, rather than trusting "no retrieval step" as a proxy for "faster."
Cache staleness in CAG. The knowledge baked into a persisted cache is frozen at build time; rebuilding on every minor update erases the amortization benefit the whole pattern depends on, so track a staleness window bounded to how fast the underlying corpus actually changes.
Duplicate or contradictory content in the hybrid. The cached core and the freshly retrieved slice both asserting (or worse, disagreeing about) the same fact, invisible unless the unified prompt is validated as one artifact.
A CAG cache shared across tenants without deciding to. A whole-corpus cache built once and reused is deliberate cross-request sharing; if different tenants query it, that is exactly the trust-group sharing case tenant cache isolation covers, decide it on purpose rather than by accident.

Open questions & validation¶

Whether your corpus size and query volume actually sit in the regime where CAG wins, tested against your own measured latencies the way this page tested against the paper's, not assumed from the "no retrieval step" framing alone.
Whether your embedding model and reranker were chosen against a held-out, domain-representative benchmark, MTEB or your own, rather than by name recognition.
Whether a hybrid deployment's cached core and retrieved layer are ever audited for duplicate or contradictory content reaching the same prompt.
Whether a CAG cache's staleness window is tracked and bounded, and whether anyone would notice if it silently exceeded that bound.

References¶

Chan, Chen, Cheng, and Huang, "Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks": https://arxiv.org/abs/2412.15605
Xu, Ping, Wu, McAfee, Zhu, Liu, Subramanian, Bakhturina, Shoeybi, and Catanzaro (NVIDIA), "Retrieval meets Long Context Large Language Models," ICLR 2024: https://arxiv.org/abs/2310.03025
"Long Context vs. RAG for LLMs: An Evaluation and Revisits": https://arxiv.org/abs/2501.01880
"Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach": https://arxiv.org/abs/2407.16833
Muennighoff, Tazi, Magne, and Reimers, "MTEB: Massive Text Embedding Benchmark": https://arxiv.org/abs/2210.07316
Douze et al., "The Faiss Library": https://arxiv.org/abs/2401.08281
Johnson, Douze, and Jégou, "Billion-scale similarity search with GPUs": https://arxiv.org/abs/1702.08734
BAAI, bge-reranker-v2-m3 model card: https://huggingface.co/BAAI/bge-reranker-v2-m3
NVIDIA, RAG Blueprint (NeMo Retriever reference architecture): https://github.com/NVIDIA-AI-Blueprints/rag

Chan, Chen, Cheng, and Huang, "Don't Do RAG": CAG preloads all relevant resources into an LLM's extended context and caches the resulting runtime state, answering queries with no additional retrieval step; on HotPotQA-Large (85k preloaded tokens) CAG's own reported total generation time was 2.2631s, versus sparse retrieval top-3's combined 0.6675s (0.0008s retrieval, 0.6667s generation) and dense retrieval top-3's combined 1.3454s (0.4123s retrieval, 0.9331s generation); the paper frames CAG's advantage as strongest for constrained, bounded knowledge bases. https://arxiv.org/abs/2412.15605 ↩↩↩
Xu et al. (NVIDIA), "Retrieval meets Long Context Large Language Models" (ICLR 2024): retrieval improves LLM performance regardless of extended context window size; a 4K-context LLM with simple retrieval augmentation matched a 16K positional-interpolation fine-tuned LLM at much less compute, and a retrieval-augmented Llama2-70B-32k outperformed GPT-3.5-turbo-16k and Davinci003 on nine long-context tasks while beating its own non-retrieval 32k baseline and generating faster. https://arxiv.org/abs/2310.03025 ↩
Li, Cao, Ma, and Sun, "Long Context vs. RAG for LLMs: An Evaluation and Revisits": long context generally outperforms RAG in question-answering, especially for Wikipedia-based questions; summarization-based retrieval performs comparably to long context, chunk-based retrieval lags behind, and RAG holds an advantage on dialogue-based and general-question queries, a split by query and task type rather than by how evidence is spread through the corpus. https://arxiv.org/abs/2501.01880 ↩↩
Li, Li, Zhang, Mei, and Bendersky (Google DeepMind, University of Michigan), "Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach": when adequately resourced, long-context LLMs consistently outperform RAG on average, but RAG remains far cheaper; the paper's proposed Self-Route routes each query wholesale to either RAG or long-context based on the model's own self-reflection, reducing computation cost substantially while keeping performance comparable to long-context alone, a per-query routing hybrid, not a per-prompt fusion of both. https://arxiv.org/abs/2407.16833 ↩
Muennighoff, Tazi, Magne, and Reimers, "MTEB: Massive Text Embedding Benchmark": spans 8 embedding tasks across 58 datasets and 112 languages; its own headline finding is that no particular text embedding method dominates across all tasks, so embedding-model choice should be benchmarked against tasks resembling the target use case. https://arxiv.org/abs/2210.07316 ↩↩
Douze et al., "The Faiss Library": FAISS is a toolbox of indexing methods and related primitives for search, clustering, compression, and transformation of vectors, written in C++ with Python/C wrappers and GPU-accelerated kernels for some of its most useful algorithms (building on Johnson, Douze, and Jégou's "Billion-scale similarity search with GPUs"); the project's own GitHub README describes it as searching "sets of vectors of any size, up to ones that possibly do not fit in RAM." https://arxiv.org/abs/2401.08281 ; https://arxiv.org/abs/1702.08734 ; https://github.com/facebookresearch/faiss ↩
BAAI, bge-reranker-v2-m3 model card: a cross-encoder reranker built on the bge-m3 base at 0.6B parameters that takes a query and document as input and outputs a similarity score directly rather than an embedding, with strong multilingual capabilities evaluated on the MIRACL benchmark. https://huggingface.co/BAAI/bge-reranker-v2-m3 ↩
NVIDIA, RAG Blueprint / NeMo Retriever: a reference RAG pipeline (chunker, embedding NIM, vector store on the indexing side; retriever, an optional NeMo Retriever reranking NIM, and generation on the query side), with the reranker sitting between ANN retrieval and the LLM to reorder retrieved passages for precision after the retrieval stage optimizes for recall. https://github.com/NVIDIA-AI-Blueprints/rag ↩