Identity, delegation, access control and information-flow control

Understand identity and delegation for agents, how privilege escalation and the confused deputy reappear at the agent layer, the access-control models that decide what an agent may do (capabilities, deny-by-default gates, intersection of principal/agent/task), and how information-flow control and isolation bound what an agent can reach — stopping exfiltration and the lethal trifecta regardless of what the model decides.

Track

Research

Module

11 · Securing & verifying agents

Prerequisites

Reasoning is not authorizing

An agent reads untrusted text — a document, a web page, a tool result, a message from another agent — and then decides which tools to call. Nothing in a language model's architecture separates the instructions it was given from the data it ingests, so in the worst case any content it reads can influence any action it takes. That is not a claim that the model will obey an injected instruction; it is the observation that nothing structurally prevents it. Security for such a system cannot rest on making the model reliably refuse. It rests on bounding what the agent is able to do, so that even a fully subverted model cannot reach beyond a small, pre-authorized envelope.

This flips the usual question. Instead of "will the agent behave?", the engineering question is "what is the agent allowed to do if it behaves as badly as possible?". The controls in this lesson answer that: a short-lived identity that proves who is acting and on whose behalf; delegation that can only narrow authority as it travels; access decisions taken against the requester, not the executor; and information-flow control that labels every value and refuses to let confidential data reach the wrong sink. Each one holds regardless of the model's decisions, which is exactly why it is worth building.

An agent is a peculiar subject. It has an identity of its own (a workload with credentials), it acts on behalf of a human principal or another service, and its behaviour can change within a single session because of what it reads. Token-exchange standards distinguish two semantics for this. Under impersonation, the actor receives all of the subject's rights in a context and third parties treat it as if it were the subject. Under delegation, the actor keeps its own identity, the subject has delegated a subset of its rights, and the token records who is acting on whose behalf. Impersonation is simpler and far more dangerous: whatever an injected agent decides, it decides with the full rights of the person it stands in for.

Ambient authority

Authority a component holds by virtue of who it is — a standing service account or credential — applied automatically to any request it processes. The soil in which confused deputies grow.

Confused deputy

A component with ambient authority that performs a privileged action because it was asked, without checking whether the requester was entitled to it. At the agent layer, injected content supplies the request.

Attenuable capability

An unforgeable, signed token that carries authority with the request and can only be narrowed at each hop, never widened. Delegating a task becomes handing over a capability scoped to that task.

Effective permission

For a session of agent a acting for principal u on task t, the set of actions actually allowed. Bounding it to the intersection of what u, a and t each permit is the only mode that limits exposure to what was asked.

Two single-agent principles carry over directly. Least privilege says a component should hold as little ambient authority as possible, so even a successful confusion yields little. Least common mechanism says every mechanism shared between principals is a potential channel for propagating failures, so share as little as possible. Both were articulated as design principles decades ago and both become sharper for agents, because the most dangerous shared mechanism is often one nobody treats as a channel at all — long-term memory, discussed below.

Three production access-control models give content to the decision point. Role-based access control (RBAC) interposes roles between subjects and permissions; it is easy to audit but represents anything that is not a stable property of the subject only by creating more roles. Attribute-based access control (ABAC) decides with a function of subject, object, action and environment attributes, expressing conditions RBAC cannot, at the cost of depending on attribute integrity. Relationship-based access control (ReBAC) decides from the relationships between subject and resource and scales sharing and inheritance. Attenuable capabilities sit alongside all three: a capability is a pre-computed, portable decision carried with the request, which is the natural way to hand an agent the authority for one task.

The decision an agent needs is not "may this subject act?" but "may this agent, on behalf of this principal, for this task?". The correct answer is the intersection of the three permission sets. Authorizing against the agent's own service account (ambient mode) permits everything any of its users might need — maximal exposure and a classic confused deputy. Authorizing against the user's full rights (impersonation) still exposes everything the user can do to whatever content the agent read. Only intersecting with the task's capability bounds exposure to what was actually requested, and it costs no more under-privilege than the other modes that include the agent. The gate should be deny-by-default with forbid-priority: a request is allowed only if some permit rule matches and no forbid rule does.

synthetic — deny-by-default gate authorizing against principal ∩ agent ∩ task
# Effective permission = intersection; forbid always wins; default deny.
# None of the model's output is trusted as authority here.

def authorize(req, principal_caps, agent_caps, task_cap, policy):
    # req = { action, resource, context }
    # 1) forbid-priority: any matching deny ends it
    for rule in policy.forbids:
        if rule.matches(req):
            return DENY, rule.name            # e.g. forbid share/delete of confidential

    # 2) the action must be inside ALL THREE scopes
    allowed = (req in principal_caps          # P(u): what the human may do
               and req in agent_caps          # P(a): the agent's own role
               and task_cap.authorizes(req))  # Cap(t): derived from THIS task
    if not allowed:
        return DENY, "outside principal ∩ agent ∩ task"

    # 3) a permit must explicitly match
    for rule in policy.permits:
        if rule.matches(req):
            return ALLOW, rule.name
    return DENY, "no permit matched"          # default deny

# A fully injected agent that proposes refund/exfiltrate gains nothing:
# the authority on the request was never refund, whatever the model "decided".

Two design consequences follow. First, because the policy is deny-by-default with forbid-priority, editing it is monotone: adding a permit, removing a forbid, or weakening a permit's condition can only widen what is allowed, and the reverse can only narrow it. That lets changes be classified syntactically, with only widenings needing review. Second, the attributes the decision depends on must come from sources the agent cannot write — if the agent can mark a document as non-confidential, the forbid that protects confidential documents is effectively the agent's to waive.

The confused deputy is one of the oldest security bugs: a privileged component is manipulated into using its legitimate authority for someone who should not have it. Delegation between agents is confused-deputy fuel. Whenever agent A asks agent B to act, a question must be answered — when B acts, whose authority is it using? If B uses its own broad credential, anyone who can get a request to B, including an attacker who injects A, borrows B's full power. The attacker never needs B's credentials; they need only B's willingness to act on a request, and that willingness, ungoverned, is the vulnerability.

Several agent-specific authorization failures escalate this. The all-purpose service account gives an agent the union of its users' permissions. Impersonation without a task scope hands over the user's full rights for what may be a read-only job. The "agent role" is a wildcard role invented so the team need not model each task, which quietly recreates the service-account problem. And there is a timing failure specific to replicated, high-speed systems: stale authorization, where a decision is taken against a view of the access-control state that predates a revocation. An agent chains writes and reads in milliseconds across replicas, so a check evaluated on a stale snapshot can grant access that was just revoked — the same time-of-check/time-of-use gap that bites humans rarely and agents often. The fix is a consistency token that forces the check to be evaluated against a snapshot at least as recent as the last relevant change.

Access control decides what an agent may invoke; information-flow control (IFC) decides what data is allowed to reach each place when something goes wrong. The classical formulation labels every value with a security class, treats combination as the least upper bound of its inputs, and permits information to flow only upward in a lattice. For agents, a useful label pairs confidentiality (which categories may read it) with integrity (whether only trusted sources influenced it). Mixing with something secret makes a value secret; mixing with something untrusted makes it untrusted. Integrity is the dual of confidentiality: it protects not who may read but who may influence.

The control that matters lives at the sinks, not in the content. A sink is any point where a value leaves the agent's boundary or produces an effect: an outbound request, a message, a third-party tool parameter, a write others read. Exfiltration is a value reaching a sink whose admissible categories do not include its confidentiality label; an unauthorized action is a consequential sink whose target or decision depends on low-integrity data. The decentralized label model adds that each principal owns its labels and only its owner may declassify them, which is how a multi-tenant agent keeps one tenant's data from reaching another even inside a shared component.

synthetic — a sink admission check with worst-case mixing
# Worst-case assumption: a model mixes everything it reads, so a value's
# label is the join of every input that reached the context that produced it.

def at_sink(value, sink):
    L = value.label                       # (confidentiality set C, integrity i)

    # confidentiality: deny if the sink is not cleared for every category in C
    if not L.C.issubset(sink.cleared_categories):
        return DENY                       # would be exfiltration -> never negotiated

    # integrity: a consequential sink must not be steered by untrusted data
    if sink.consequential and L.i == 0:   # i == 0 means "influenced by untrusted"
        return ESCALATE                   # route to a human, do not auto-execute
    return ALLOW

# A single edge that drops the label ("whitewashes") voids every later check:
# if any path from a source to a sink crosses a label-dropping edge, the flow
# is a violation no matter how correct the sink checks are.

Isolation reinforces IFC on three planes that are usually treated separately and fail together. Executing model-generated code in a sandbox (a container with a reduced syscall surface, a microVM with its own kernel, or a WebAssembly module with only the host functions it is granted) bounds what that code can reach. Context isolation keeps the component that decides from reading untrusted content directly — a privileged planner that never sees raw content, and a quarantined model with no tools that extracts fields and returns labelled handles the interpreter substitutes without the planner seeing them. And long-term memory, the most dangerous shared mechanism, must be treated as a flow channel: notes stored with the label of the data they derive from, partitioned into compartments by tenant and principal, with expiry measured from the origin so a planner cannot renew a stale note forever by rewriting it.

A widely cited framing names a lethal trifecta: an agent that combines access to private data, exposure to untrusted content, and a channel for external communication is one injected instruction away from exfiltration. The three documented classes of incident all share a structure — the model does not "send" anything, it writes a string, and some later component (a client that renders an image URL, a browser that types text, a tool that receives a parameter) delivers it to a third party. Removing any one leg removes the risk, but you often cannot: the data is the point, the content is the input, and communication is the task.

So you bound rather than forbid. The following steps show how the controls compose into an envelope that holds whatever the model decides.

1. 1Give the agent a short-lived identity and carry authority as an attenuable capability scoped to the task, so the executor authorizes against principal ∩ agent ∩ task, not the agent's standing credential.
2. 2Keep the deciding component (the planner) away from untrusted content; let a quarantined, tool-less model extract fields and return labelled handles instead of free text.
3. 3Label every value with confidentiality and integrity, propagate the labels through tools and context, and check them at every sink — no edge may drop a label.
4. 4Make high-impact or low-integrity sinks escalate to a human rather than auto-execute, and bind that approval to the exact canonical action (covered in the next lesson).
5. 5Run model-generated code in a sandbox with no standing secrets or network beyond what the task grants, so even arbitrary code stays inside the envelope.

The point of the sequence is that none of its guarantees depend on the model's behaviour. A subverted planner cannot widen its capability, cannot read across a compartment boundary, cannot push confidential data past a sink check, and cannot make generated code reach a resource it was not granted. What remains outside these guarantees is the truth of what the quarantined model extracted, the correctness of origin labels, covert channels nobody modelled, and the judgement of the human who receives the escalations.

• ▸PROVEN (by construction): if authority is carried as an attenuate-only capability and the decision is taken against principal ∩ agent ∩ task under deny-by-default, a subverted agent cannot exceed what the chain was granted.
• ▸PROVEN (by construction): if every value is labelled and no edge drops the label, a flow that violates a sink's admission policy is detectable at that sink — isolation into compartments makes some flows structurally unreachable rather than merely denied.
• ▸NOT PROVEN: these controls say nothing about whether a correctly-authorized, in-policy action is a good idea. A valid capability attests scope, not merit.
• ▸Attenuation only helps if authority actually travels with requests; systems that authenticate the channel but authorize against standing service accounts remain confused deputies regardless of their cryptography.
• ▸Over-broad initial grants defeat the chain — the chain can only narrow what it starts with — and a socially-engineered legitimate grant is not fixed by any scoping.
• ▸UNKNOWN / residual: origin labels can be wrong from the first edge, long-term memory can carry flows across sessions that per-session testing never sees, and covert channels (timing, resource use) are outside the modelled sinks.

Expected direction: Confused-deputy sketch: Billing checks "do I hold payments.refund?" and executes, so an injected Triage borrows Billing's authority. Fixed sketch: requests carry the originating principal's attenuated capability (invoice.read for this ticket only); Billing authorizes against that capability, so refund/exfiltrate is never on the request. IFC step: label each invoice value with the customer's confidentiality category; the external sink is cleared only for its own customer, so a value whose label includes another customer's category is denied at the sink, provided no edge between lookup and sink drops the label. The property is that the capability can only be attenuated (never widened) and the label is checked at every sink with no whitewashing edge. Attack not stopped: a socially-engineered legitimate grant (or an over-scoped root capability), and a wrong origin label that mislabels one customer's data as another's — both make a valid, in-policy flow that the controls permit.

Why is bounding an agent's authority preferable to making the model refuse injected instructions?

Because nothing in a language model's architecture separates instructions from data, so in the worst case any content it reads can influence any action; a control that bounds what the agent can do holds regardless of whether the model is subverted, whereas a control that depends on the model behaving does not.

What is the effective permission an agent should be authorized against, and why that one?

The intersection of the principal's, the agent's and the task's permissions. It is the only mode that bounds exposure to what was actually requested (no confused deputy, no over-privilege beyond the task) while costing no more under-privilege than the other modes that include the agent.

In information-flow control, why does a single label-dropping edge void later sink checks?

Because the sink decides admission from the value's label; if any edge on a path from a source to the sink drops (whitewashes) the label, the value arrives unlabelled and the sink cannot tell it is prohibited, so the flow is a violation no matter how correct the sink's checks are.

What is the lethal trifecta, and how do you bound it when you cannot remove a leg?

The combination of access to private data, exposure to untrusted content, and a channel for external communication in one agent. When no leg can be removed you bound it: task-scoped capabilities, a planner isolated from untrusted content, labels checked at every sink, high-impact sinks escalated to a human, and sandboxed code — so a subverted model stays inside a pre-authorized envelope.