Reasoning is not authorizing

State and defend the central principle of agent security: a model's reasoning or confidence is never authorization. Authority must live outside the model — in deterministic code, behind a frontier no decision branch can cross — and the protection principles that follow (least privilege, deny-by-default, separation of the proposer and the authorizer) are what make that frontier real.

Track

Research

Module

11 · Securing & verifying agents

Prerequisites

The agent threat model and its tactic taxonomy

Consider four fixes a team might apply to a synthetic payments agent, in order. First, the agent reads an invoice PDF, extracts the account number and calls a pay function with a service credential; a third party changed the account, and the payment leaves for the attacker. No model disobeyed anything — copying the account from the invoice is exactly what the task seemed to ask. Second, the team adds a reviewer model that inspects each call; it catches most odd account numbers, but not the ones the agent edits after review, nor the calls it still makes directly with the credential it still holds. Third, the reviewer is replaced by a deterministic policy that compares the account with the vendor master — but the policy reads the master from the invoice, so the policy is now perfect and empty. Each fix closed one path and left another.

The lesson behind all four is one sentence: reasoning is not authorizing. The model's chain of thought, its stated confidence, its "I am sure" — none of these is authority. Authority is the judgement, made by a component with power delegated by a principal, that this concrete proposal is permitted in this state. When the proposal and the decision are the same object — as they are in a direct model-to-tool call — there is no authorization at all; there is only the model, which the threat model already showed is confusable by design.

Three objects are fused in a single framework tool call and must be pulled apart. Keeping them separate is the whole discipline.

Proposal

What the agent wants to do. It is produced by the model and is, at best, a request plus a designation of an object. It carries no authority of its own.

Decision

The judgement, by a component with authority delegated from a principal, that a specific proposal is permitted in a specific trusted state. It is a deterministic function of a typed intent, the request, and state the model does not control.

Effect

The change on a real system. Only one component — the executor — holds the credentials that produce effects, and it acts only on a decision it can verify.

Authorization frontier

The line between proposing and deciding. No branch of the model's reasoning can cross it: the model's output arrives as a typed intent to be judged, never as an instruction to be executed.

Saltzer and Schroeder's protection principles, written in the 1970s for time-sharing operating systems, remain the best guide. Three carry most of the weight for agents. Complete mediation: every access to every object is checked against authority — in an agent, every tool invocation passes the decision point, not just the first of a session, and an approval is never reused for a different intent. Least privilege: each component runs with the minimum authority its task needs, so a mistake or a deception has a bounded blast radius. Separation of privilege: a mechanism that needs two independent keys is more robust than one that needs one — in an agent, the natural split is between the one who proposes and the one who authorizes.

Anderson's reference monitor gives the three requirements for the component that decides: it must be tamper-proof, always invoked, and small enough to be analysed and tested with confidence. A language model satisfies none of them — it is manipulable by content, bypassable if the agent holds its own credentials, and not analysable. So the decision cannot be the model. It must be a separate chain of components the model cannot steer.

the authority chain (schematic, after Clark–Wilson well-formed transactions)
agent (model, zone 1)  --emits-->  INTENT as text  (a proposal; untrusted)
        |
VALIDATOR (zone 2)   canonicalises or REJECTS: closed schema, no duplicate
        |            keys, resolved designations; one parser in the whole chain
POLICY ENGINE (zone 2)   decides on the canonical intent + the request +
        |                TRUSTED state the model cannot write:  allow / deny / escalate
FRONTIER             signs the decision, binding the HASH of the decided bytes,
        |            the versions of records read, a nonce and a short expiry
EXECUTOR (zone 3)    ONLY holder of credentials; acts iff signature valid,
                     decision = allow, hash matches, nonce unused, versions current

Clark and Wilson gave this its commercial form decades ago: data that matters changes only through certified well-formed transactions, unconstrained input becomes constrained data only by passing such a transaction, and separation of duty stops the proposer from being the approver. The translation is direct — the model's output is unconstrained input; the validator and policy engine are the transaction that turns it into a legitimate order; the executor is the only component that touches the real system; and the agent can never be both proposer and approver.

1. 1Define a closed schema of operations. The agent emits a typed intent (operation, typed arguments, per-argument provenance, the requesting principal), and object designations must resolve against a trusted registry — free-text names are not valid designations ("no designation without authority").
2. 2Validate by rejecting, not repairing: one parser for the whole chain, duplicate keys refused, unknown fields refused, amounts normalised and checked. Whatever travels onward is canonical bytes, never the original text.
3. 3Decide deterministically on the canonical intent, the request and trusted state — the vendor master, the ERP, roles, limits — read from zone 2, never from the content that produced the intent.
4. 4Return allow, deny, or escalate. Escalate is the honest answer for what the policy cannot decide alone (a high-impact or irreversible action), routing it to an authority or a human with its own design.
5. 5Bind the decision to the exact bytes with a signature over their hash, the versions of records read, a nonce and a short expiry — so the executor runs what was decided, once, before anything changed.
6. 6Give credentials to the executor alone. The agent process has no network path to them; it can only write to a proposal queue. A "credential just in case" re-opens every closed path.

A deny-by-default authorization gate is the smallest honest version of the policy engine. It enumerates permitted intents with their conditions and denies everything else, including any intent the validator could not type. The confidence the model attaches to its proposal appears nowhere in the decision.

synthetic — a deny-by-default gate where model confidence is never an input
def authorize(intent, request, trusted_state):
    # intent: {op, args, provenance, principal}   -- a PROPOSAL, not a command
    # NOTE: intent.get("model_confidence") is deliberately NEVER read.

    rule = POLICY.get(intent["op"])
    if rule is None:
        return deny("unknown operation")           # deny-by-default

    # authority is the REQUESTER's, not the agent's (no confused deputy)
    if not request.principal_may(intent["op"], intent["args"]):
        return deny("requester lacks the right")

    # decide on TRUSTED state, never on the content that produced the intent
    if intent["op"] == "pay":
        account = trusted_state.master[intent["args"]["vendor"]].account
        if intent["args"]["account"] != account:
            return security_event("account != master")   # not merely 'deny'
        if intent["args"]["amount"] > rule.limit:
            return escalate("over limit -> human / authority")

    # bind the decision to these exact bytes, once, with a short expiry
    return allow(sign(hash(canonical(intent)), versions=trusted_state.versions))

Three things make it work and are worth stating plainly. The gate never reads the model's confidence or its justification — those are proposal metadata, not authority. It checks the requester's rights, not the agent's broad service identity, which is what stops the confused deputy. And it decides against trusted state, so an injected instruction that changes the proposal cannot also change the yardstick it is measured against. A mismatch is not a quiet denial but a security event, because a benign agent would not have produced it.

When Raven proposes an action that touches a real system — opening a ticket, writing to a shared store, running a tool with side effects — the proposal is text, and the lab treats it as text. It crosses the authorization frontier into a deterministic gate that decides on typed state Raven cannot write, and only an executor Raven cannot reach holds the credentials. Raven's reasoning is logged for traceability and review, but it is never consulted as the authority for the effect. This is the architectural expression of the whole module's thesis: the threat model showed that the two roots meet at granted authority, and this lesson removes the model from the authority path entirely.

Deny-by-default is also why Raven's capabilities are scoped per task rather than per agent: a lab assistant that reads advisories does not need, and is not given, the ability to move money or change infrastructure, so an injection has nothing powerful to reach. What authority Raven does hold is a narrow, attenuated capability for the task at hand. The next lesson asks the complementary question — what it means for Raven "to know" the thing it is proposing to act on — because authorizing an action and justifying the belief behind it are two different disciplines that must both hold.

• ▸The chain guarantees that every effect was authorised; it does not guarantee that what was authorised is what the principal wanted. A correct policy over a correct state can still permit an unwise action.
• ▸The policy must be correct. Authority outside the model moves the risk to the policy and the trusted state it reads; a wrong rule is a deterministic hole that no amount of model robustness fixes.
• ▸If the policy reads the content that produced the intent, it is correct and empty — the attacker now controls the yardstick. Decide only on state the model cannot write.
• ▸Two parsers, or forwarding the original text after validating a copy, re-open the gate: the decision is made on one interpretation of the bytes and the effect on another. One canonical form, one parser.
• ▸A credential left with the agent "for debugging" bypasses the entire chain. Mediation is only as complete as the least-guarded path to the real system.
• ▸Least privilege per task needs you to know what a task requires before running it — and an agent exists because that is not always known. The honest options (a slightly loose capability, escalation, or deriving the capability from a validated plan) all cost something; none is free.

Expected direction: A gate that enumerates the two operations and denies everything else, reads vendor/approval state from a trusted registry rather than from the invoice, scopes authority to the requester, escalates the account-number change to an out-of-band authority, and explicitly excludes model confidence and justification from the decision. The four-fix analysis should map: direct call -> no mediation/separation (chain: validator + executor-only credentials); reviewer model -> same key twice + bypass (independent zone-2 decision + no agent credentials); policy reading the document -> decision not independent of content (decide on trusted state); and should note the residual risk that a correct policy over correct state can still permit an unwanted action.

Why is a model's reasoning or confidence never a source of authority?

Because the model is confusable by design — it cannot reliably separate data from instructions — so its output is at most a proposal. Authority must come from a principal and be evaluated by a tamper-proof, always-invoked, analysable component the model cannot steer; the model's confidence is proposal metadata, not a permission.

What are proposal, decision and effect, and why must they be separated?

A proposal is what the agent wants; a decision is a delegated judgement that the proposal is permitted in a trusted state; an effect is the change on a real system. In a direct model-to-tool call the proposal is the effect, so there is no authorization at all — separating them is what lets authority live outside the model.

Why is a second reviewer model not a valid "separation of privilege"?

Because it reads the same untrusted content as the first model, so it is the same key twice: it shares the vulnerability. Separation of privilege requires independent keys — the authorizer must decide on typed facts the model cannot influence.

What does the authority chain guarantee, and what does it explicitly not?

It guarantees that every effect on a real system was authorised by a deterministic decision against trusted state the model could not steer. It does not guarantee that what was authorised is what the principal actually wanted, nor that the policy or the trusted state is correct.