Retrieval, RAG and computational memory

Understand embeddings and vector retrieval, hybrid/dense retrieval and reranking, RAG and agentic RAG (the agent decides what to retrieve), GraphRAG, and computational memory for agents (episodic/semantic, what persists and the risks of persistence) — and where retrieval helps versus where it launders unverified content into context.

Track

Research

Module

9 · Agent cognition & reasoning

Prerequisites

The Transformer and the residual stream

A language model carries a great deal of knowledge in its weights, but that knowledge is frozen at training time, has no provenance, and cannot be revised one fact at a time. Retrieval is the standard answer: keep the knowledge outside the model, in a store the system can search, and at query time fetch the relevant pieces and place them in the context window. The model stops being the sole source of truth and becomes a reader of evidence it was handed.

This changes the problem in a useful way. Instead of asking "does the model remember this fact", we ask "did the system retrieve the right evidence, and did the reader use it faithfully". Those are separable, measurable questions. But it also introduces a new failure surface that this lesson keeps in view: retrieval decides what the generator can possibly cite, so an error in retrieval cannot be fixed downstream — what is not retrieved cannot be quoted, and what is wrongly retrieved arrives looking exactly as trustworthy as what is correct.

An embedding maps a text to a point in a vector space where geometric similarity is meant to approximate relevance. A bi-encoder computes one vector per text — the document once, offline; the query online — and scores a pair by their dot product or cosine. Search is then a nearest-neighbour problem, served by approximate indices. The whole point is precomputation: everything a future query could ask about a document must fit into those few hundred numbers.

Dense (bi-encoder) retrieval

One fixed-length vector per text; cheap to index and search, but a single dot product cannot express conjunctions or negations, and rare tokens (identifiers, codes, proper names) blur into the average.

Sparse learned retrieval

A high-dimensional vector over the vocabulary with a few non-zero weights, including learned synonyms. It keeps the strengths of lexical matching (exact rare terms, an inverted index, interpretability) while adding expansion.

Late interaction (multi-vector)

One vector per token; the score sums, over query terms, the best match in the document. This represents conjunctions well — a rare query term with no match is penalised even if the rest agrees — at the cost of storing a vector per token.

Cross-encoder (reranker)

Query and document are read together by one Transformer that can model any interaction, so it cannot precompute anything. It is used to rerank a short candidate list, not to search the whole corpus.

The geometry has honest, PROVEN limits. A single fixed-dimension vector can only realise a bounded family of top-k orderings: a recent combinatorial result shows that for any dimension d there exist relevance patterns no d-dimensional embedding can reproduce exactly, and the number of documents at which this bites grows with the diversity of queries. Measured consequences follow — single-vector encoders order negation ("incidents NOT resolved by a restart") close to chance, because a mean-pooled vector cannot represent an operator that inverts meaning. The lesson: a similarity space is an approximation with known blind spots, not an oracle.

Because no single representation wins everywhere — reported benchmarks consistently find lexical scoring (BM25) a robust baseline out of domain, dense retrieval strong on paraphrase, and late interaction best on conjunctions — the dominant design is a cascade: a cheap, high-recall first stage that fuses lexical and dense candidates, then an expensive reranker over a short list. Fusion is usually either reciprocal-rank fusion or a tuned linear combination of the two scores.

1. 1Retrieve a candidate pool with a lexical index (exact terms, identifiers) and a dense index (paraphrase, semantics), in parallel.
2. 2Fuse the two ranked lists into one candidate set, so that neither a rare exact term nor a semantic match is lost.
3. 3Apply a structured filter for hard constraints — dates, identifiers, negations — outside the vector space, because the dot product cannot enforce them reliably.
4. 4Rerank the short fused-and-filtered list with a cross-encoder that reads query and document together.
5. 5Pass the top results to the generator, carrying each candidate's provenance so the answer can cite it and a later step can verify it.

synthetic — a hybrid retrieve-and-rerank pipeline sketch (illustrative, not a benchmark)
def retrieve(query, k_pool=100, k_final=8):
    lex   = bm25_index.search(query, k=k_pool)      # exact terms, IDs
    dense = vector_index.search(embed(query), k=k_pool)  # paraphrase

    pool  = rrf_fuse(lex, dense)                     # reciprocal-rank fusion
    pool  = [d for d in pool if hard_filter(query, d)]  # dates / IDs / negation

    # cross-encoder reads (query, doc) TOGETHER; expensive, so only on the pool
    ranked = sorted(pool, key=lambda d: cross_encoder.score(query, d),
                    reverse=True)

    # a score is a ranking signal, NOT a probability of truth
    return [with_provenance(d) for d in ranked[:k_final]]

A controlled comparison on synthetic collections (reported in the source material) isolates the effect: on mixed queries that combine a rare exact term with concepts expressed through synonyms, every single-vector system scores poorly while late interaction or a hybrid-then-rerank cascade recovers the correct document — and the cascade matches exhaustive late interaction at an order of magnitude less compute. The takeaway is MEASURED and modest: reranking buys accuracy on compositional queries; it does not turn a resemblance score into a guarantee.

Retrieval-augmented generation conditions the generator on documents fetched from an external memory. Its original formulation treats the retrieved document as a latent variable and marginalises over the top-k: the answer is a mixture whose quality depends both on how the retriever spreads probability mass and on what the generator does with each document. Reading that equation carefully dispels a common illusion — using only the single best document gives more confidence, not more accuracy, because the mixture is exactly what attenuates a retriever's mistake and makes disagreement between sources visible.

A single retrieval pass fails on multi-hop questions ("in what country is the company that employs Ana based?"), because the document naming the country never mentions Ana — the query that finds it can only be formed after the intermediate answer is known. Agentic RAG is the response: a controller decides, at run time, when to retrieve, with what query, and when to stop. It decomposes the question, retrieves per sub-question using each intermediate answer, judges whether the evidence is sufficient, and verifies support before answering or abstaining. The difference from a fixed pipeline is that the edges of the graph are decisions, each with a cost and an error rate.

Agentic RAG also makes abstention a first-class decision. If a correct answer is worth 1, a wrong one costs c, and abstaining is worth 0, it is worth answering only when the estimated probability of being right exceeds c/(1+c). That rule needs a calibrated probability (the subject of a later lesson) and a verifier that checks whether the retrieved context actually supports each claim — distinguishing an answer that is faithful to the evidence from one that is merely fluent.

A flat passage index answers "find the passage that resembles this query". It does not answer "what do these thousand documents have in common", because no single passage says so, nor "where is Ana's company based", because the relevant passage does not resemble the question. Structuring the memory as a graph of entities and relations, or as a hierarchy of summaries, changes which questions are easy. GraphRAG extracts an entity graph with a model, partitions it into communities, and summarises each; a global question is answered map-reduce style — a partial answer per community summary, then a combination. Reported comparisons are consistent and type-dependent: a flat index wins on single-hop, detail-oriented questions; graph and community methods win on multi-hop and global, sense-making questions. There is no global ordering of methods, only a dependence on the question.

For an agent, the memory is what distinguishes a stateful agent from a stateless function. Following the classical psychology of memory, we separate episodic memory (time-stamped events — "on Monday the user asked X"), semantic memory (general facts — "the user prefers X"), and procedural memory (how to do something, e.g. a growing library of reusable skills). Working memory is the context window; the parametric memory is the weights. The operations that turn a store into a memory are the hard part: consolidation, update, contradiction resolution, and forgetting.

Persistence is a double-edged property. A memory that only accumulates fails two ways: it retrieves old and new versions of the same fact and lets the reader pick (an internal contradiction), and it grows without bound so that irrelevant entries crowd out relevant ones. Simulated streams reported in the source show the effect sharply — accumulating without resolving contradictions leaves accuracy low because more than half the answers are stale values the memory still holds, while overwriting or retrieving the most recent value restores it. Forgetting by a usage-based activation (frequency and recency combined) beats forgetting by age alone, because it keeps the attributes that are actually queried.

And persistence is a security surface. A malicious entry written into memory persists and is retrieved in future sessions; without redundant provenance there is no way to adjudicate between a true fact and a planted one. Every edge, every stored fact, should carry its provenance — which passages assert it — because without provenance there is no citation, no way to resolve contradictions, and no way for what the agent remembers to justify what it does.

Retrieval helps most where the model's weights are weakest: unpopular, long-tail, or changing facts that scaling barely memorises. Putting the fact in the context also repairs structural blind spots — a model that fails to infer "B is A" from training on "A is B" will infer it when "A is B" is in the prompt, because retrieval turns a memory task into a reading task. These are real, MEASURED benefits.

But retrieval can also launder unverified content into context, and this is the failure the lab cares about most. Three mechanisms recur. Context-induced hallucination: when the true fact is absent but plausible entities of the right type are present, the reader picks one, and the context raises the probability of a confident, specific, wrong answer. Decorative citations: the answer cites a document that does not actually support the claim. Hostile content: a retrieved passage is untrusted input that may carry instructions or poisoned data. In every case the retrieved text arrives wearing the same authority as verified evidence, and fluency hides the difference.

• ▸A high similarity score means resemblance, not truth: it ranks candidates, it does not certify them.
• ▸A faithful answer to a wrong document is still wrong; a correct answer taken from the weights is neither faithful nor attributable. Faithfulness, attribution, and correctness are three independent properties.
• ▸Retrieved text is untrusted input. Treat it as an observation that updates belief, never as an instruction or an authorization.
• ▸Provenance is the unit of auditability: a fact without a source cannot justify an action, and an action taken on unverifiable evidence is UNKNOWN-risk, not SAFE.

In an agentic workflow, RAG is a perception channel with a loop: decide what to observe, observe, update what is believed, decide again. That makes agentic RAG structurally a read-only ReAct agent, and the lab's transversal thesis applies with full force — what the agent retrieves feeds its beliefs; the authority to act must come from an external policy that does not depend on what the generator happened to read. A verifier that confirms a plan respects constraints is not permission to execute it.

• ▸Retrieval errors are not recoverable downstream: what is not retrieved cannot be cited, so first-stage recall is where single-hop systems should invest.
• ▸In multi-hop chains, decomposition reliability dominates, because its error multiplies at every hop and a support verifier cannot catch a well-supported answer to the wrong sub-question.
• ▸Graph memory moves cost from query time to an offline indexing step and makes knowledge auditable, but the index ages and extraction omissions multiply over long paths.
• ▸Long-term memory needs explicit time, update, contradiction resolution and a forgetting policy chosen from the measured reuse pattern — not borrowed by analogy from psychology.
• ▸None of these mechanisms turn evidence into authorization. High recall and high confidence are not proof; a decision with effects passes through a control the model does not own.

Expected direction: A 0.94 cosine measures resemblance between the query and the passage, not the truth of the claim or the authority of its source; retrieval feeds belief, not authorization. The passage is an observation (and untrusted input that could be poisoned or a decorative/irrelevant match), not an order. The question is genuinely multi-hop — invoice to purchase order, order to approver, approver to signing limit for that amount — and at least the signing-limit link should be resolved by a structured query against the authority database, not text retrieval, because chain accuracy compounds (p^m). Each link needs provenance and per-claim support verification, and the decision to release payment must pass through an external authorization policy that does not depend on what the generator read. Until the signing authority is verified, the claim is UNKNOWN, not SAFE, and the agent should abstain or escalate rather than act.

Why is a single fixed-dimension embedding fundamentally limited, and give one measured symptom?

A single dot product over a fixed-dimension vector can only realise a bounded family of top-k orderings, so for any dimension there exist relevance patterns it cannot reproduce; it also cannot express conjunctions or negations. A measured symptom: single-vector encoders rank negation queries close to chance.

What does "a chain is only as reliable as the product of its links" mean for agentic RAG?

If each of m hops succeeds independently with probability p, end-to-end accuracy is about p^m, so even highly reliable per-hop steps compound to a much lower total — the compositionality gap. It implies investing in decomposition reliability and resolving demanding hops with structured queries rather than text retrieval.

Give three ways retrieval can launder unverified content into context.

Context-induced hallucination (plausible wrong entities present when the true fact is absent), decorative citations (citing a document that does not support the claim), and hostile content (retrieved text carrying injected instructions or poisoned data). In all three the content arrives with the same apparent authority as verified evidence.

State the lab's epistemic rule for retrieved evidence.

Retrieval feeds beliefs, not authorization. A relevant passage is an observation, not an order; a high similarity score is resemblance, not proof. What cannot be verified stays UNKNOWN, never SAFE, and decisions with effects must pass through an external policy.