Evidence, provenance and corroboration

Turn "a belief needs verifiable justification" into machinery: what counts as evidence for an agent's claim, provenance (where a datum came from and who vouches for it), corroboration across independent sources, and how to weigh evidence. Ground the rule that a finding exists only when a verifier confirms it with evidence, and expose the prompt-injection-poisoning chain in which an attacker manufactures evidence.

Track

Research

Module

11 · Securing & verifying agents

Prerequisites

The epistemology of agentic systems

A synthetic payments agent receives an email: "ACME has changed banks; the new account is …". Minutes later it holds four pieces that assert it — the email, a PDF attachment with the new account, a note the vendor portal recorded while processing the same email, and the summary its own extractor wrote to memory. A reasoner that counts confirmations concludes there are four agreeing sources and, with reasonable reliabilities, crosses the threshold a business would set for such a change. But the four pieces have a single origin: the last three derive from the email, and whoever wrote the email controls all four. This is the engine of most knowledge and memory poisoning — the attacker does not compromise many sources, only gets the system to count the one it controls many times.

What is missing is provenance: knowing where each piece came from and what depends on what. Evidence is not a pile of agreeing statements; it is a graph of derivations with responsible parties attached. Once that graph exists, corroboration stops being a count and becomes a calculation that discounts shared origins — and the attacker's cheap trick of fabricating agreement loses its force.

The W3C PROV data model gives a standard vocabulary for provenance. An entity is a thing with some fixed aspects; an activity acts on or with entities over an interval; an agent bears some responsibility for an activity happening. The core relations are generation (an entity was generated by an activity), use, derivation (one entity was derived from another), attribution (an entity is attributed to an agent) and delegation (one agent acted on behalf of another). Recording these makes "where did this come from, and who vouches for it?" a query rather than a guess.

Evidence

A record < proposition, value (asserts or denies), the entity that supports it, the instant it refers to, an optional expiry >. Its responsible party is the agent the entity is attributed to, followed up the delegation chain to a principal.

Origin

An entity with no prior derivation, reached by following derivation links backwards. Two pieces of evidence are independent only if their origins are disjoint and their responsible parties differ.

Attribution rule

A model-produced entity is attributed to whoever controls its input, NOT to the extractor. A zone-1 output inherits the trust of its input: the extractor is recorded by association (its version matters for the audit), but it adds no independence.

Resolution by authority

For propositions with a designated owner (the vendor master, an approval flow), the rule is not to combine evidence but to settle it: adopt the value from the most recent evidence whose responsible party is that authority, and treat contrary evidence as a security event.

Attributing the extractor's output to whoever controls its input, rather than to the extractor, is the perception-inherits-trust rule of the threat model expressed in PROV. It is also what stops the agent's own memory from laundering an untrusted origin into an apparently fresh, trusted source.

If reports are conditionally independent given the hypothesis, the log-likelihood-ratio evidence is a sum, and N agreeing sources make evidence grow linearly in N. That independence assumption is precisely what the attacker exploits. Model a group of reports with a common origin: each copies the origin's value with probability c, or is an independent observation with probability 1 − c. Then the per-copy contribution is positive but shrinks as c rises, equals the full value at c = 0, and vanishes at c = 1 — at c = 1 the whole group is worth exactly its origin, however many copies there are. It is cleaner to report the effective number of independent pieces of evidence rather than the raw count.

synthetic — an evidence-weighting sketch that discounts shared origins
def effective_independent(pieces, reliability, copy_prob):
    """How many INDEPENDENT pieces of evidence a group really carries.
    pieces sharing an origin count once in the limit of certain copying."""
    groups = group_by_origin(pieces)        # provenance decides the grouping
    log_lr = 0.0
    for g in groups:
        s, f = reliability[g.responsible]   # sensitivity, false-positive rate
        c = copy_prob(g)                     # from history, or conservative (~1)
        origin_term = log(s / f)                               # the origin itself
        copy_term   = log((c + (1 - c) * s) / (c + (1 - c) * f))
        log_lr += origin_term + (len(g) - 1) * copy_term       # copies add little
    return log_lr / log(reliability_baseline())  # as an EFFECTIVE count

# four pieces, one origin, high copy probability -> ~1.1 effective, not 4.
# For external sources with no history: assume c near 1 unless independence
# is PROVEN -> i.e. require distinct origins with distinct responsible parties.

Inside the system the copy structure is knowable: every component declares what its output derives from, and the log rejects derivations toward unknown entities. Outside the system it is not — a site that copies a press release without citing it looks independent — so undeclared copying must be estimated from shared errors, or, for sources with no history, assumed near-certain. Sharing true values proves little, since independent sources agree on the truth; sharing errors is improbable without copying. Changing formalism does not help: Dempster–Shafer and subjective logic also assume source independence. The dependence has to be modelled, not assumed away.

1. 1Record each datum in an append-only provenance log: who generated it, from where, when, derived from what — content stored as a hash, each entry chained to the previous so tampering breaks the chain.
2. 2Attribute each entity to whoever controls its content, not to the component that produced it; trace responsibility up the delegation chain to a principal.
3. 3Group evidence by shared origin and discount copies: N pieces from one origin are worth, in the limit, one. Report the effective independent count, not the raw count.
4. 4Age the evidence by how fast the world changes, and treat an absence as evidence only with the declared coverage of the source consulted.
5. 5For a proposition with a designated authority, resolve by authority rather than combining — and raise a security event if other evidence contradicts the authority.
6. 6Adopt only against a threshold derived from the costs of the action (not a fixed "p > 0.9"), and add structural requirements — minimum distinct origins, a minimum responsible-party zone, a maximum age — that hold even if the estimated reliabilities were biased.

The prompt-injection-poisoning chain is an attacker manufacturing evidence. A hostile document is read (injection); the agent writes a derived note to shared memory (stored injection); later retrieval re-serves that note as apparently independent corroboration; and a downstream step counts it. Retrieval-poisoning attacks in the public literature work exactly this way — inserting several texts for one target query, i.e. fabricating corroboration. Provenance breaks the chain by refusing to let derived pieces count as independent.

synthetic — a provenance record that defeats manufactured corroboration
# four pieces assert "vendor account changed"; provenance records derivation:
log.entity("email1",    attributed_to="external_sender",   derived_from=[])
log.entity("pdf1",      attributed_to="external_sender",   derived_from=["email1"])
log.entity("portal1",   attributed_to="external_sender",   derived_from=["email1"])
log.entity("memory1",   attributed_to="external_sender",   derived_from=["email1"])
#   ^ the agent's OWN memory note is attributed to the sender, not the agent:
#     zone-1 output inherits the trust of its input.

origins = log.origins_of(["email1","pdf1","portal1","memory1"])
# -> {"email1"}  : ONE origin, ONE responsible party -> NOT corroboration

# the only independent evidence has a different origin AND a zone-2 responsible:
log.entity("call1", attributed_to="treasury_operator", derived_from=[])  # out-of-band call

status = classify("account_changed", evidence=log.current(), authority="treasury")
# treasury has not vouched -> UNKNOWN; the account change is NOT adopted.
# If treasury's call CONTRADICTS the four pieces -> SECURITY EVENT.

The naive count would cross the threshold and the agent would propose the change with a solid-looking justification — four agreeing sources. With the provenance graph, the four collapse to one origin worth roughly one effective piece, far below threshold; and because an account change is owned by an authority, the proposition is resolved by authority, not by any amount of third-party evidence. When the authority's out-of-band call denies the change, the contradiction is the signal of an attempted fraud and is logged as a security event. What the design does not remove: an attacker who compromises the authority's channel, or deceives the operator — it moves the attack from an email anyone can send to an identified channel and a trained human, which is the goal, not the elimination of risk.

This is the concrete form of the lab's rule that a finding exists only when a verifier confirms it with evidence. When Raven assembles support for a claim, the pieces are grouped by origin before they are weighed, so Raven's own notes to memory — and any material derived from a single scraped source — cannot inflate into corroboration. A claim reaches PROVEN only when distinct origins with distinct responsible parties, or a designated authority over a channel the adversary does not control, back it; otherwise it is published, if at all, as NOT PROVEN or UNKNOWN with its provenance visible.

The guarantee is conditional and the lab states the conditions, because that is where the residual risk lives. If the provenance log is intact and attribution is correct, an adversary controlling the entities of m responsible parties cannot by itself get a proposition adopted when the requirement is m + 1 distinct origins, nor one resolved by an authority it does not control. Integrity of the log is protected by hash-chaining or signatures; correct attribution is an identity problem; and the guarantee says nothing about the truth of what a legitimate but mistaken authority asserts. This closes the loop the module opened: the threat model placed the boundaries, authority lives outside the model, epistemology set the honest statuses, and provenance turns those statuses into auditable data.

• ▸Counting documents is the core error: treating pieces that derive from one origin — including the ones the agent wrote to its own memory — as corroboration is exactly what the attacker wants.
• ▸Attribution to the extractor instead of to whoever controls its content launders an untrusted origin into an apparently trusted source; the zone-1-inherits-trust rule must be enforced in the log.
• ▸Silence as negation: taking an absence of results as negative evidence without knowing the coverage of the source is unsound, and an adversary's silence is never evidence.
• ▸A threshold without costs ("p > 0.9" for everything) ignores that the threshold is a consequence of the cost structure and must be set per action; and the Bayesian calculation is only valid if the probability is calibrated — naive independence biases it toward overconfidence, the dangerous direction.
• ▸Undeclared copying between external sources can only be detected statistically and with history; a conservative copy probability pushes corroboration toward authority resolution and its slow, human out-of-band channels.
• ▸The provenance of a claim through a language model is uncertain by nature: when the model summarises many documents, the true derivation of each assertion is not observable, and citation systems attribute without guarantee. And a legitimate authority that is wrong or impersonated still yields an adopted belief that is false with impeccable provenance.

Expected direction: A provenance graph in which the PDF, portal note and memory summary all derive from the email and share its responsible party, collapsing four pieces to roughly one effective independent piece. A correct answer refuses the change both because the effective corroboration is far below threshold and because an account change is resolved by the designated authority, not by third-party evidence. The three assumptions should be log integrity (violated by tampering, countered by hash-chaining), correct attribution (violated by impersonating the authority or responsible party), and the authority being right and genuine (violated by a mistaken or compromised authority) — with the note that a contradiction against the authority is itself a security event.

Why are four pieces that agree not necessarily four pieces of evidence?

Because corroboration depends on independent origins, not on agreement. If the pieces share an origin — the PDF, portal note and memory summary all deriving from one email — they are one voice repeated; in the limit of certain copying, N copies are worth exactly the single origin they came from.

What does provenance record, and why is a model output attributed to the content's controller rather than to the extractor?

Provenance records who generated a datum, from where, when, and derived from what, with a responsible party traced up the delegation chain. A zone-1 output inherits the trust of its input, so the entity is attributed to whoever controls the content; the extractor is recorded by association but adds no independence, which stops memory from laundering an untrusted origin into a trusted-looking source.

How does the prompt-injection-poisoning chain manufacture evidence, and what breaks it?

A hostile document is read, the agent writes a derived note to shared memory, retrieval re-serves it as apparently independent corroboration, and a downstream step counts it — fabricating agreement from one origin. Provenance breaks the chain by grouping pieces by origin so derived copies cannot count as independent, and by resolving owned propositions through a designated authority.

Why must an adoption threshold be derived from costs and require calibration?

Because the threshold is a consequence of the costs of acting wrongly versus not acting, so it must be set per action, not fixed globally; and the posterior only minimises expected cost if it is calibrated. Treating dependent copies as independent biases the posterior toward overconfidence, which is the dangerous direction for a threshold.