Agent collusion & emergent risk

Understand how a set of agents, each acting within its own authorization, can combine to cross a boundary no single agent is permitted to cross — and why per-agent authorization, however correct, cannot by itself bound the aggregate capability of a coalition.

Track

Research

Module

6 · Securing multi-agent ecosystems

Prerequisites

The multi-agent threat landscape · Authorization and action control

All the threats so far have a victim being tricked, injected or impersonated. Collusion is different and, in a way, more unsettling: nobody is necessarily tricked. Each agent does exactly what it is allowed to do, every authorization check passes, every signature verifies — and yet the system as a whole does something no one authorized. The harm is emergent: it exists only in the combination, not in any part. This is why collusion is the hardest class to defend with the tools built for single actions; those tools ask "may this agent do this?" and the answer, truthfully, is yes at every step.

The classic security framing is old. Lampson's "confinement problem" (1973) worried about components leaking information through channels nobody intended, and covert-channel analysis studied how cooperating parties move data past controls. In an agent mesh, the "channels" are the ordinary inter-agent links, and the "cooperation" can be deliberate (two compromised agents coordinating) or merely structural (two honest agents whose individually-fine actions compose badly). Either way, the lesson is the same: a set of permissions that is safe one action at a time can be unsafe in aggregate.

Aggregate capability

The union of what a set of agents can do, including compositions of their individual actions. Often strictly greater than any member's capability and than the designer intended.

Emergent risk

Harm that exists only in the interaction of components, not in any component alone — invisible to controls that evaluate one action or one agent at a time.

Separation of duties

A control that deliberately splits a sensitive operation across parties so no single one can complete it — the intended protection that collusion specifically attacks by re-combining the split pieces.

Collusion

Coordination (deliberate or structural) among agents whose individually-authorized actions combine to achieve an unauthorized aggregate effect.

Confinement / covert channel

Lampson's problem of components moving information or effects past intended controls; in a mesh, ordinary inter-agent links can serve as such channels.

The uncomfortable fact is that correctness of each local authorization does not compose into safety of the whole. Separation of duties is designed precisely so that no single agent can complete a sensitive operation — but collusion is the recombination attack against separation of duties, stitching the split pieces back together across the inter-agent boundary. Defending it therefore cannot be purely local: you must reason about what sets of agents can do together, which is combinatorial and is why collusion risk grows faster than the number of agents.

schematic — two allowed actions, one forbidden sum
POLICY (each individually correct):
  agent READER   may: read customer PII           (for support lookups)
  agent SENDER   may: send external email          (for notifications)
  FORBIDDEN (by design, split across duties):
  nobody may: email customer PII to an external address

COLLUSION:
  READER reads PII  ------ inter-agent link ------>  SENDER emails it out
  step 1 authorized (READER may read).  check passes.
  step 2 authorized (SENDER may send).  check passes.
  aggregate = exfiltration of PII.  NO single check ever saw a violation.

WHY PER-AGENT GATES MISS IT:
  the authorization monitor evaluates (agent, action) pairs.
  the violation is a PROPERTY OF THE PAIR OF ACTIONS, not of either one.

Note two things. First, the two agents need not be malicious in the usual sense — a single injected agent that can route a request to a second honest one is enough; the "collusion" can be one compromised agent weaponising an honest peer's legitimate capability (which is also how confused-deputy and privilege-escalation chains end up here). Second, the defence cannot be a better per-action check, because by construction every per-action check is correct. You need a control that is aware of the combination.

A synthetic operations mesh (invented): an Analyst agent may query the internal data warehouse (read), and a Reporter agent may publish summaries to an external partner portal (write-out). Policy forbids raw customer records leaving the org. Neither agent alone can violate it.

synthetic — separation of duties recombined
# designed separation: read and export are different agents
analyst.query("SELECT * FROM customers WHERE ...")   # allowed: Analyst may read
rows = analyst.result
reporter.publish(partner_portal, rows)               # allowed: Reporter may export
# aggregate effect: raw customer records left the org. policy violated.
# every (agent, action) check returned ALLOW.

# CONTROLS THAT SEE THE COMBINATION:
#  data-flow labels: customer rows carry label=confidential; the export monitor
#    denies ANY export of confidential-labelled data regardless of which agent
#    (a property of the DATA crossing the boundary, not of the agent).
#  aggregate/quorum gate: moving confidential data out needs a human approval,
#    so the sum action is gated even though each step is individually fine.
#  blast-radius design: the Analyst's read scope is minimized; the Reporter
#    cannot accept confidential-labelled inputs at all.
# result: the forbidden SUM is blocked at the boundary the data crosses.

The fix is to move the control from the agent to the thing that actually crosses the forbidden boundary — here, confidential data leaving the org. A data-flow label travels with the records, and the export monitor denies the egress of confidential data by any agent. Now the violation is visible at exactly the point it occurs, regardless of how many well-authorized agents handled the data on the way.

• ▸Identify the boundaries that must not be crossed in aggregate (data egress, total spend, irreversible changes) and place the control at the boundary itself, where the sum is visible — not only on each agent.
• ▸Use data-flow labels / tainting so a forbidden property (confidential, regulated, over-budget) travels with the data or request and is checked at the crossing regardless of which agent presents it.
• ▸Enforce aggregate limits: rate and spend caps across the whole mesh, quorum/human gates on high-impact sums, and separation-of-duties that is monitored for recombination, not just declared.
• ▸Shrink blast radius by least privilege so that even a coalition's union is small: the fewer powerful capabilities exist in the mesh, the fewer dangerous sums are reachable.
• ▸Where cryptography helps: signed, hash-chained audit makes every contributing action attributable, so collusion is detectable and accountable after the fact; attenuated capabilities shrink each agent's share and thus the reachable aggregate; attested policy can bind agents to declared scopes.
• ▸Where cryptography does NOT help: there is no signature for "this combination is unsafe". Every colluding action is validly signed and genuinely authorized; crypto attests who did each step and within what scope, never whether the sum of authorized steps violates a global property. Emergent, aggregate safety is a system-design and policy question, not a cryptographic one.

• ▸Aggregate reasoning is combinatorial: the number of action combinations grows explosively, so you cannot enumerate all dangerous sums. Practical defence targets the specific global invariants you care about (egress, spend, irreversibility) rather than "all collusion".
• ▸Honest agents can collude structurally without intent: two correct components whose outputs compose badly produce the same violation as deliberate coordination. Intent is not required, so "trust the agents" is not a defence.
• ▸Covert and indirect channels are hard to close: agents can coordinate through side effects, timing, or shared state you did not model as a channel (Lampson's confinement problem has no complete solution).
• ▸A single compromised agent can manufacture collusion by weaponising honest peers' legitimate capabilities, so you must assume at least one member of any set may be adversarial.
• ▸Per-agent authorization, however correct, is structurally blind to the sum. The durable control is to make the forbidden aggregate visible at the boundary it crosses and to keep each agent's authority — and thus the reachable union — as small as possible.
• ▸Above all: every colluding action can be perfectly authorized and perfectly signed. Validity of the parts is not safety of the whole — the central honesty of this module.

Expected direction: A scenario where each (agent, action) check returns allow but the composition violates a global invariant (data egress, total spend, separation-of-duties recombination). A control placed at the boundary the aggregate crosses — data-flow labels/taint checked at egress, an aggregate/quorum or human gate on the sum, or mesh-wide rate/spend caps — rather than on individual agents. The learner must explain that because each action is individually authorized and validly signed, per-action authorization and cryptography cannot see the forbidden combination; aggregate safety is a system-design/policy property, not a cryptographic one.

What makes collusion different from every other threat in this module?

There need be no victim who is tricked, injected or impersonated. Each agent does only what it is individually authorized to do — every check passes and every signature verifies — yet the combined effect crosses a boundary no single agent could cross. The harm is emergent, existing only in the aggregate.

Why can't a better per-agent authorization check stop collusion?

Because by construction every individual action is correctly authorized; the violation is a property of the combination of actions, not of any single (agent, action) pair, which is all a per-action monitor can evaluate.

How does collusion relate to separation of duties?

Separation of duties splits a sensitive operation so no one party can complete it; collusion is the recombination attack that stitches the split pieces back together across the inter-agent boundary, defeating the intended protection.

Where should the control for an aggregate violation sit, and why?

At the boundary the forbidden aggregate actually crosses (e.g. a data-egress monitor with data-flow labels, or an aggregate spend/quorum gate), because only there is the sum visible. Controls on individual agents are structurally blind to the combination.

What does cryptography contribute to collusion defence, and what can it never do?

It makes each contributing action attributable (signed, hash-chained audit) so collusion is detectable and accountable, and attenuation shrinks each agent's share of the reachable aggregate. It can never certify that a combination of validly-authorized, correctly-signed actions is safe — there is no signature for "this sum is unsafe".