The epistemology of agentic systems

Ask what it means for an agent "to know" something: the difference between a belief, a justified claim and a proven fact; the PROVEN / NOT PROVEN / UNKNOWN discipline; why "it looks vulnerable" must never become a reported fact; and how to calibrate claims honestly. This is the conceptual anchor the rest of the lab leans on.

Track

Research

Module

11 · Securing & verifying agents

Prerequisites

Reasoning is not authorizing

There is a well-known case in which a company's conversational assistant told a customer about a refund policy that did not exist. The sentence was plausible, well-formed, consistent with what such a company might offer — and false. Nothing in the sentence let the customer distinguish it from a true one, and the customer reasonably believed it. Now replace the customer with an agent that reads another system's answer, or a retrieved document, and uses it to decide. The agent is in the customer's position, with one difference: no one will grant that it "reasonably relied". What an agent believes, and with what right it believes it, stops being a philosophical question the moment that belief enables an action.

The interest here is not hallucination. It is indistinguishability. A statistical component that reads attacker-reachable content produces a plausible proposition; plausibility is how typical a statement is in a corpus, which correlates with truth but separates from it exactly where it matters — in rare claims, popular falsehoods, and content designed to look typical. So an agent does not know what its model says. It receives plausible testimony from a speaker with no beliefs, and the engineering question is what has to be added before that testimony is allowed to govern an action.

The classical analysis treats knowledge as justified true belief: an agent knows p if p is true, it believes p, and it is justified in believing it. Each condition excludes something — without truth you confuse knowing with being convinced, without belief with something merely being the case, without justification with getting it right by luck. Gettier showed in three pages that the three together still do not suffice: one can infer a true conclusion from a justified but false premise, so the belief is true and justified yet the link between justification and truth is accidental. For an engineer the lesson does not depend on which repair you prefer: a system can decide correctly for the wrong reasons, and an audit that checks only the outcome will not catch it. For security, the chain matters, not the hit.

Belief

A proposition the agent holds — here, typically the plausible output of a statistical component, carrying at most a first-order confidence.

Justified claim

A belief backed by evidence with provenance, the measured reliability of the source in its reference class, and currency — reasons that are independent of the belief itself and accessible to a third party.

Proven fact (operational)

A proposition true in every world compatible with reading an authoritative source — i.e. an authority the organisation designated can settle it, over a channel the adversary does not control. It does not solve knowledge in general; it settles the propositions whose authority can be named.

Reference class

The class in which a component's reliability is measured. A reliability figure without a reference class justifies nothing: the same extraction is "extraction by model M" (measured: high) and "extraction from input controlled by the adversary" (set by the adversary).

Reliabilism — a belief is justified if a reliable process produced it — is the natural epistemology for an agent that perceives with statistical models, because a reliability can be measured. But it inherits the generality problem: one episode instances many process types with different reliabilities, and reliabilism does not say which counts. In a system this is resolved by decision, not philosophy: the policy fixes the reference class, it must include the provenance of the input, and reliability is measured and applied per class.

Treat the model's output as testimony. The epistemology of testimony splits into reductionism — accept what you are told only with positive, independent reasons to trust it — and anti-reductionism, which takes testimony as a basic source, accepting by default what is presented as true and intelligible unless there are stronger reasons not to. That default-acceptance stance is a precise description of the vulnerability prompt injection exploits. An agent therefore cannot be anti-reductionist about untrusted content: it must be reductionist, accepting testimony only with positive reasons independent of the testimony itself — provenance, the measured reliability of the source in its class, and corroboration.

Two kinds of defeater matter. A rebutting defeater gives reasons for not-p. An undercutting defeater attacks the link between the reason and p without asserting not-p — knowing a wall is lit by red light undercuts the inference "it looks red, so it is red". Most evidence of manipulation is undercutting: that a PDF contains invisible text aimed at the model says nothing about the invoice amount, but it voids the reliability of the process that extracted it for that document. An agent that models only rebutting defeaters will keep believing a value extracted from a manipulated document as long as nothing contradicts it.

synthetic — the PROVEN / NOT PROVEN / UNKNOWN discipline
def classify(claim, evidence, authority=None):
    """Map a model-produced proposition to an honest epistemic status.
    The model's own confidence is NOT an input here."""
    if authority is not None:
        settled = authority.ruling(claim)          # channel the adversary can't forge
        if settled is None:
            return "UNKNOWN"                        # authority has not spoken
        return "PROVEN" if settled else "NOT_PROVEN"

    # no designated authority: can an independent channel separate the worlds?
    independent = [e for e in evidence
                   if e.channel_outside_adversary_control
                   and not e.undercut]              # drop undercut (manipulated) evidence
    if distinct_origins(independent) >= claim.required_origins:
        return "PROVEN"                             # corroborated across real sources
    if any(e.contradicts(claim) for e in evidence):
        return "NOT_PROVEN"
    return "UNKNOWN"                                # plausible, but not established

# "it looks vulnerable" -> at most UNKNOWN until a verifier confirms it.
# UNKNOWN is a first-class, reportable status, never silently upgraded.

1. 1Treat the model's output as perceived content: a plausible proposition with, at most, a first-order confidence — never yet a fact.
2. 2Record evidence for it: who produced it, from where, when, derived from what. Without a provenance record there is no justification to check.
3. 3Fix the reference class — including the input's provenance — and apply the measured reliability of the source for that class, not a benchmark number from a different class.
4. 4Separate first-order confidence (how much the component believes) from second-order support (how much the figure itself is backed): a source with ten irrelevant hits is not a source with years of history.
5. 5Decide the status per action: PROVEN (settled by a channel that separates the worlds), NOT PROVEN (contradicted or below requirement), or UNKNOWN. The same proposition can be adopted for a draft and refused for an irreversible action.
6. 6Treat a contradiction against a designated authority not as noise but as a security event — a sign that something manufactured the opposing evidence.

A synthetic payments agent only pays approved invoices. Its extractor reads an "APPROVED" stamp with a purchasing manager's signature on invoice 881, and the agent adopts p = approved(881). Three worlds produce the same perception: in w1 the invoice is approved and the stamp is genuine; in w3 a vendor inserted a fake stamp and the invoice is not approved; in w2 — the real world — the stamp is fake, but the manager did approve the invoice yesterday in the ERP, for an independent reason.

synthetic — belief vs safe belief vs knowledge in the three worlds
worlds (same perception; ranked by what the agent finds plausible):
  w1  rank 0   approved  &  genuine stamp
  w3  rank 1   NOT approved &  injected stamp
  w2  rank 2   approved (in ERP) & injected stamp     <- the REAL world

evaluated in w2 (the real world):
  p = approved(881)   TRUE          # the invoice really is approved
  B p  (belief)       TRUE          # the most plausible world, w1, has p
  S p  (safe belief)  FALSE         # w3 is at least as plausible and has not-p
  K p  (knowledge)    FALSE         # truth did not come from the justification

# This is a Gettier case: true, justified belief that is NOT knowledge.
# If the POLICY trusted the stamp, it would also pay in w3 -- indistinguishable.
# Consulting the ERP (an authority) is HARD information: it deletes w3,
# and in w2 the agent now KNOWS p.

The agent that pays in w2 gets it right. But if its policy rested on the stamp, it would also pay in w3, which it cannot tell apart. An audit that checks "was the paid invoice approved?" finds nothing wrong; an audit that checks "with what evidence was approval adopted?" finds the flaw. This is the general proposition in miniature: if a rule sees only the perception and the adversary can realise a world with the same perception where p is false, the rule adopts p there too. No better model, prompt or classifier changes that — only a channel whose value differs between the worlds and that the adversary does not control.

This is why the lab never lets Raven report "it looks vulnerable" as a finding. A model's impression of vulnerability is a plausible proposition in the same equivalence class the adversary can fabricate; it is UNKNOWN until a channel the adversary does not control — a reproducing verifier, an authoritative record, independent corroboration — separates the worlds. The lab's rule is that a finding exists only when a verifier confirms it with evidence; everything short of that is published, if at all, with its honest status, so that a reader is never handed a guess dressed as a fact.

The combination the lab adopts is externalist for measuring and internalist for recording: the reliability of each class of source is measured, not assumed, and the reasons for each belief are kept accessible to whoever audits, even though they were never accessible to the model. A belief that governs an action and lacks verifiable justification is an attack surface — whoever controls an input in its equivalence class controls the belief. The next lesson builds the machinery that turns "verifiable justification" into concrete data: a provenance record and a rule for weighing corroboration.

• ▸Verbal confidence is not probability: "I am sure", or a number the model emits, is not a calibrated probability of truth unless it was calibrated in the relevant reference class — and calibration degrades out of distribution, which is exactly where an adversary operates.
• ▸Reliability without a class justifies nothing: citing an extractor's benchmark accuracy as justification for its output on inputs of unknown provenance is the generality problem in disguise.
• ▸The default witness is the injection vulnerability: accepting what the model extracts unless something contradicts it is anti-reductionism applied to untrusted content.
• ▸Hard information from perception is the formal shape of memory poisoning: writing a model output in a way that deletes alternatives cannot be undone by later evidence. Content from the model should enter as soft, revisable information with its provenance.
• ▸Auditing the result, not the evidence, hides Gettier cases: a decision that was correct for the wrong reasons passes an outcome audit and fails the next time the coincidence does not hold.
• ▸Operational "knowledge" only settles propositions whose authority can be named. It does not make the authority infallible: a legitimate authority that is wrong, or whose credential was stolen, produces an adopted belief that is false with impeccable justification.

Expected direction: Three worlds producing the same perception, ranked so that the most plausible has p while a world at least as plausible has not-p (making safe belief fail); a demonstration that belief and truth hold in the real world but knowledge does not, because truth did not flow from the justification; and identification of an authoritative or independent channel whose value differs across the worlds and that the adversary cannot forge. The status assignment should be UNKNOWN (or NOT PROVEN) before the channel and PROVEN after, with the explicit note that the model's own confidence was never used and that the authority could still be wrong or impersonated.

What does it mean to say an agent does not "know" what its model outputs?

The model produces plausible testimony from a speaker with no beliefs; plausibility measures typicality in a corpus, not truth. The output is at most a belief, justified only if backed by provenance, measured reliability in the right reference class, and corroboration — reasons independent of the output itself.

Why must "it looks vulnerable" never become a reported fact?

Because it is a plausible proposition in the same equivalence class an adversary can fabricate; a rule that sees only the perception cannot tell the real world from the planted one. It stays UNKNOWN until a channel the adversary does not control — a verifier, an authority, independent corroboration — separates the worlds.

What is the generality problem and how is it resolved in a system?

A single belief-forming episode instances many process types with different reliabilities, and reliabilism does not say which one counts. A system resolves it by decision: the policy fixes a reference class that must include the input's provenance, and reliability is measured and applied per class — a reliability number with no class justifies nothing.

Why is auditing the outcome of a decision insufficient?

Because a Gettier-style decision can be correct for the wrong reasons: an outcome audit sees "the paid invoice was approved" and finds nothing wrong, while the belief rested on forgeable evidence and will fail the next time the coincidence does not hold. The audit must check with what evidence the belief was adopted.