Where cryptography helps, and where it does not

Draw an honest map of the multi-agent security problem, separating what cryptography can enforce — authenticity, integrity, scope, non-repudiation, attestation — from what it structurally cannot: the semantic correctness, truth, or safety of an authorized action.

Track

Research

Module

6 · Securing multi-agent ecosystems

Prerequisites

Digital signatures · Scope attenuation

It is tempting, after learning signatures, attenuated capabilities and attestation, to believe cryptography can secure a multi-agent system. It cannot — not because the cryptography is weak, but because most of what it proves is orthogonal to most of what can go wrong. Cryptography is extraordinarily good at a specific family of questions: did this exact message come from this key (authenticity), was it altered (integrity), is the holder permitted at most this much (scope), can the holder deny having said it (non-repudiation), and is this program the one that ran (attestation). Every one of those is about provenance and bounds. None of them is about whether the authorized thing was a good idea.

This is the honest map the research program behind this material (published under the name Raven, with the adversarial red-team effort known internally as NEMESIS) keeps returning to: cryptography shrinks the attack surface by removing the forgeable, the tamperable, the over-scoped and the deniable — and then stops, exactly at the line where meaning begins. A system that confuses "cryptographically valid" with "safe" has not reduced its risk; it has hidden it behind a green checkmark.

The clearest way to hold the whole module in mind is a two-column table: what cryptography enforces, and what it leaves to system design. Every mechanism in this track lands in the left column; every residual risk lands in the right.

Authenticity (signatures)

ENFORCES: these bytes came from this key. DOES NOT: tell you the sender was uncompromised, uninjected, or right.

Integrity (hashes, MACs)

ENFORCES: the content was not altered. DOES NOT: tell you the content was true or benign to begin with.

Scope (attenuated capabilities)

ENFORCES: the holder may do at most X. DOES NOT: tell you that doing X, within scope, is wise or intended.

Non-repudiation / audit (hash chains)

ENFORCES: who did what, unforgeably, after the fact. DOES NOT: prevent a harmful-but-authorized action; it only makes it attributable.

Attestation (and ZK proofs)

ENFORCES: this code/statement is the one claimed, or this property holds, revealing nothing more. DOES NOT: make the attested policy or decision correct — a proof of a bad rule is a valid proof of a bad rule.

Read down the DOES NOT column and you have re-derived the entire threat module. Injection (Module-6 lesson 4) lives in "the content was true/benign" gap. Confused deputy (lesson 2) lives in the gap between scope and intent. Collusion (lesson 6) lives beyond scope entirely — every part is in-scope. Impersonation is the one class the left column genuinely closes; Sybil (lesson 5) is back in the right column because valid keys do not imply distinct parties.

Every attack that survives good cryptography shares one shape: it is carried by something that is perfectly valid. The injection is a faithfully-signed summary of a hostile page. The confused-deputy request is a correctly-authorized action. The colluding steps are each in-scope. The Sybil approvals each verify. The attacker is not breaking the crypto; the attacker is operating in the space the crypto never addressed.

schematic — the gap the green checkmark hides
CRYPTO VERIFIES ........................ GAP (semantics) ...............
  signature valid?    YES   ->  but is the content an injection?     ?
  within capability?  YES   ->  but is exercising it intended?       ?
  each step in-scope? YES   ->  but is the SUM a policy violation?    ?
  key authentic?      YES   ->  but is the holder honest / distinct?  ?
  proof verifies?     YES   ->  but is the PROVEN statement correct?  ?

left side: cryptography answers it, cheaply and strongly.
right side: needs independent verification, human gates,
            blast-radius limits, provenance, policy on aggregates.
confusing the two columns is the core mistake.

A zero-knowledge proof makes the trap sharpest, because it feels like the strongest possible assurance. A ZK proof can establish that an agent holds a valid delegation for an action without revealing the delegation — genuinely useful. But it proves the statement in the circuit, nothing more. If the circuit encodes a flawed policy, the proof is a flawless certificate of a flawed policy. The strength of the proof says nothing about the wisdom of what is proven.

A synthetic hardened mesh (invented) does everything right cryptographically: every message is Ed25519-signed and verified against a registry; authority flows as attenuated capabilities; a ZK proof attests that each actuator holds a valid delegation; all actions are hash-chained for audit. An attacker still wins.

synthetic — perfect crypto, real breach
# 1. attacker injects the Researcher via a hostile web page (lesson 4)
#    Researcher's signed report now contains a poisoned recommendation.
#    signature: VALID.  content: hostile.  -> crypto says "trust the bytes".

# 2. Orchestrator, acting on the report, issues an in-scope instruction;
#    its capability is correctly attenuated and the ZK proof verifies.
#    scope: VALID.  intent: attacker's.  -> crypto says "within bounds".

# 3. two actuators each take an individually-authorized step;
#    combined, they export regulated data (collusion, lesson 6).
#    each step: VALID.  aggregate: a policy violation.

# the audit log is perfect: it will attribute every step unforgeably...
#    ...AFTER the data has left. non-repudiation is not prevention.

# WHAT ACTUALLY STOPS IT (right-column controls):
#  provenance + data-not-instructions so the poisoned report is evidence, not a command
#  independent verification of the recommendation before acting on it
#  data-flow labels + egress gate so the forbidden AGGREGATE is blocked
#  human gate on exporting regulated data (high-impact, irreversible)
#  least privilege so the reachable aggregate is small to begin with

Not one cryptographic check failed. The breach happened entirely in the semantic gap: hostile-but-authentic content, in-scope-but-unintended authority, individually-valid-but-collectively-forbidden actions. The audit chain performs flawlessly and tells you, with cryptographic certainty, exactly who helped exfiltrate the data — after the fact. That is the whole point: cryptography bounded and attributed everything, and prevented the one thing it never could, which was a safe decision.

• ▸Use cryptography aggressively for the left column — it is cheap, strong and decisive there. Sign and verify every inter-agent message (kills impersonation and tampering); carry authority as attenuated capabilities (bounds scope and kills cross-agent confused-deputy escalation); hash-chain an audit log (makes everything attributable); use attestation/ZK where you must prove a property without revealing it. This is most of defence-in-depth and you should not skimp on it.
• ▸Do not ask cryptography to do the right column. Whether content is an injection, whether an in-scope action is intended, whether a sum of authorized actions violates a global policy, whether a key-holder is honest or distinct — none of these has a cryptographic answer.
• ▸Fill the right column with different tools: provenance and strict instruction/data separation (semantic trust of content); independent verification and cross-checks (correctness of a claim or plan); data-flow labels and egress/aggregate gates (global-property enforcement across colluding steps); human-in-the-loop on high-impact, irreversible actions (judgement); and least privilege everywhere (so the reachable blast radius — and aggregate — stays small).
• ▸Design reviews should name, for each control, which column it is in. A control in the left column is not evidence for a right-column property. The most common and dangerous design error is a green checkmark standing in for a safety argument it cannot make.

• ▸There is no cryptographic primitive for semantic correctness, truth, intent, or aggregate safety. This is not a gap awaiting a better scheme; it is a category difference between "provable about bytes/scope" and "true about meaning".
• ▸Left-column strength can mask right-column weakness: the better the cryptography, the more authoritative a semantically-bad-but-valid artifact looks. Strong crypto can increase semantic risk if teams read "verified" as "safe".
• ▸Right-column controls are themselves imperfect: human gates fatigue, independent verifiers can be wrong or injected, labels can be stripped, and aggregate policies cannot enumerate every dangerous combination. They reduce residual risk; they do not eliminate it.
• ▸Attestation and ZK prove the statement, not its merit: a valid proof of a flawed policy is a valid proof. Verifying the proof is necessary; auditing what is being proven is a separate, human responsibility.
• ▸The durable posture is defence-in-depth across both columns plus minimized blast radius, under the standing assumption that any individual agent may be compromised and any coalition may collude — exactly the stance the NEMESIS red-team framing enforces when evaluating a mesh.
• ▸The one sentence to carry out of this module: a valid signature or proof proves authenticity and scope, never that the authorized decision was correct or safe. Build so that nothing irreversible depends on confusing the two.

Expected direction: A correct two-column mapping: each cryptographic control paired with its exact guarantee (authenticity/integrity/scope/non-repudiation/attestation) and its exact non-guarantee (not truth, not intent, not aggregate safety, not honesty/distinctness of the key-holder, not correctness of the proven statement). A worked attack (injection, confused deputy, collusion or Sybil) shown passing all cryptographic checks, stopped by right-column controls (provenance + data/instruction separation, independent verification, data-flow labels/egress gates, human gates, least privilege). A crisp closing sentence that a valid signature/proof attests authenticity and scope but never the correctness or safety of the authorized decision.

What family of questions is cryptography genuinely good at in a multi-agent system?

Provenance-and-bounds questions: authenticity (who sent these exact bytes), integrity (were they altered), scope (the holder may do at most X), non-repudiation/audit (who did what, unforgeably), and attestation (this is the code/property claimed). All concern identity, integrity and bounds — not meaning.

Give the single sentence that captures the module's honesty principle.

A valid signature or proof proves authenticity and scope, never that the authorized decision is correct or safe — "verified" answers who, unchanged and within-bounds, never whether.

A ZK proof verifies that an agent holds a valid delegation. Why is that not a safety guarantee?

Because it proves exactly the statement encoded in the circuit and nothing more. If the policy in the circuit is flawed, the proof is a flawless certificate of a flawed policy; proof strength says nothing about the wisdom of what is proven.

Why can strong cryptography sometimes increase semantic risk?

Because a semantically bad artifact that passes every cryptographic check looks authoritative; teams that read "verified" as "safe" will trust hostile-but-valid content more, not less. The green checkmark can hide the gap it never addressed.

Name the non-cryptographic controls that fill the right column, and what each is for.

Provenance + instruction/data separation (semantic trust of content), independent verification/cross-checks (correctness of claims/plans), data-flow labels and egress/aggregate gates (global-property/collusion enforcement), human-in-the-loop on high-impact irreversible actions (judgement), and least privilege everywhere (shrinking blast radius and reachable aggregate).