Red teaming, governance and the verifiable decision log

Understand adversarial testing / red teaming of agents as a defensive engineering process (what a campaign measures and what "zero findings" does and does not mean), governance and accountability that turn written frameworks into machine-checkable artifacts, and the tamper-evident, verifiable decision log that lets every decision be reconstructed with its evidence — and how a secure reference architecture composes the module's controls while exposing their shared assumptions.

Track

Research

Module

11 · Securing & verifying agents

Prerequisites

Verification: formal, runtime and consensus

The controls of the previous lessons protect a single agent. This lesson adds the layer an organization needs once it has many: adversarial testing to find what the design missed, governance to know which agents exist and under what controls, and a record that survives an incident. All three are about honesty under pressure — being able to say precisely what was tested, who is responsible, and what happened, without trusting the party that operated the system.

The recurring trap is reading a reassuring artifact as a guarantee. "Zero failures" from a red-team report, a green compliance checkbox, and a database row showing an approval all look like proof and are not. A campaign that found nothing bounds a rate only over the distribution it sampled; a certified organization can still run an agent that is not in its inventory; a mutable log row proves nothing about what it said yesterday. This lesson replaces each reassuring artifact with a measurable one, and ties them to the architecture that actually carries the guarantees.

Red teaming of your own system is a statistical experiment, not a proof. Its headline number, the attack success rate, is a parameter of three things a report often omits: the system, the distribution of test cases, and the oracle that decides whether a case violated the property. Change the cases and the rate changes; an adaptive adversary is precisely a distribution chosen after seeing the defences, against which a rate measured on a fixed suite no longer holds. Make the oracle a fallible judge and you are measuring a different quantity. So a campaign must separate two aims that demand opposite sampling: finding failures (maximize distinct failures per unit cost; adaptive search is best) and estimating risk (measure a rate with an interval under a fixed distribution; stratified sampling is best).

Attack success rate (ASR)

The probability a system violates the property under a test distribution and oracle. It is a property of the system-and-distribution-and-oracle triple, not of the system alone.

Coverage matrix

The threat-model categories crossed with the campaign's cases. A category with no cases has no estimate, whatever the global number says.

Exact oracle

A function that inspects the environment state after a case and decides whether the prohibited effect occurred. Preferred over a model-based judge, which can be fooled by the same injections being tested.

Regression battery

Each found failure reproduced as a deterministic (or fixed-seed) case, run on every change of model, prompt, tool or policy, so a fix stays fixed.

The key statistical fact for security is the regime of small rates. With zero violations in n independent cases evaluated by an exact oracle, the one-sided 95% upper bound on the rate is below 3/n — the "rule of three". So bounding a rate to 1% with zero failures needs about 300 cases; bounding each of K categories multiplies the cost and the confidence budget must be split across them. And none of it bounds a distribution outside what was sampled: a system that fails only on inputs the campaign never tried produces zero failures and is unbounded there. That is why an architectural guarantee (which does not depend on the distribution) and a campaign bound are different in kind, and the second does not replace the first.

synthetic — the rule of three and why an imperfect oracle weakens it
import math

def zero_failure_bound(n, alpha=0.05):
    # Exact one-sided upper bound when x = 0 failures in n independent cases:
    #   P(X=0 | theta) = (1-theta)^n = alpha  ->  theta = 1 - alpha**(1/n)
    exact = 1 - alpha ** (1 / n)
    rule_of_three = -math.log(alpha) / n        # ~ 3/n, since -ln(0.05) = 2.996
    return exact, rule_of_three

# "0 of 200" is compatible, at 95%, with a rate up to ~1.5% on the TESTED cases.
# With an imperfect judge of sensitivity Se < 1, each real failure is only labelled
# with probability Se, so the bound inflates to about  -ln(alpha) / (n * Se).
# And for ANY distribution outside the tested support, the bound is 1 (no bound).

A disciplined campaign therefore writes its plan first (objective, categories and weights, budget, oracle, stopping rule), runs in an isolated replica with the same controls as production (evaluating the model without its gate measures something else), prefers an exact oracle and treats a judge's error as part of the estimate, and reports what it did not cover so "not tested" is never read as "safe". Every finding is fixed in the architecture — a validator rule, a finer capability — not patched with another instruction to the model, and enters the regression battery.

Governance decides who may act, with what authority, on what information, and how they answer for it. The characteristic failure needs no adversary: an agent operates outside the perimeter where the controls are required, or on evidence that no longer describes it. Separate three layers — the control (a monitor, a policy, a bound approval), risk management (which controls each system needs and whether they work), and governance (who decides and accounts). Public frameworks for AI risk management converge on four demands: know which systems exist, assign each a risk level and a named responsible person, require proportionate controls with evidence, and repeat when the system changes or time passes. The contribution for agents is to make these machine-checkable before a credential is issued or an autonomy level is raised.

Three properties make that work. The risk classification is a deterministic function of attributes you can check against the real configuration (the tools an agent has are the ones its credential permits, not the ones it declares), and it is monotone: adding capability or autonomy never lowers the required set of controls. Evidence is bound to the version it was obtained with and expires, so a control is satisfied only by evidence whose version components still match — changing the model invalidates any evidence bound to the model, which is exactly what should block promoting an agent "because it is only a model change". And change management separates the author of a change from its approvers, while lowering autonomy during an incident requires no approvers at all, just as the kill switch must not depend on the agent.

synthetic — a deterministic, monotone risk class and the no-inherited-evidence rule
def risk_level(card, mode):
    # Deterministic function of declared, checkable attributes. Higher = stricter.
    eff = max(tool_effect(t) for t in card.tools)   # read<reversible<irreversible<financial
    data = max(data_class(d) for d in card.data)    # public/internal < personal < special
    level = 1 + min(eff, 2)
    if mode == "autonomous" and eff >= 2:
        level += 1                                   # irreversible & unattended: raise
    trifecta = (data >= 1 and card.untrusted_input and card.external_sink)
    if trifecta or data == 2 or card.legal_high_risk:
        level = max(level, 3)                        # floor
    return 1 if mode == "shadow" else min(4, level)

def control_satisfied(control, card, evidence, now):
    # Evidence must be fresh AND obtained with the current version of every
    # component the control is bound to. A model/prompt/tool change invalidates it.
    for e in evidence.get(control.id, []):
        fresh = (now - e.time) <= control.validity
        same_version = all(e.version[k] == card.version[k] for k in control.bound_to)
        if fresh and same_version:
            return True
    return False     # no inheritance: new version => re-earn the evidence

Monotonicity does not stop someone splitting an agent into pieces that each fall under a threshold: two low-level agents that pass data to each other can compose the lethal trifecta, so exposure must be evaluated on the flow (from the IFC lesson), not per card. And the classification's force depends entirely on the version being observable — with a provider that changes the model behind a stable alias, the version must be an immutable identifier or the fingerprint of a reference suite run periodically.

After an incident the question is "who approved this, and with what information?", and the answer is worth only as much as the guarantee that nobody altered the record. Observability (sampled, mutable traces) serves operation; an audit log serves proof and needs other properties: completeness over a class of events, tamper evidence (any modification, deletion or reordering is detected), non-equivocation (all observers see the same history), attribution, and governed retention. Tamper evidence is not tamper impossibility — whoever controls storage can always delete — it is that they cannot do so without it being noticed. The techniques are the ones certificate transparency took to production: an append-only log whose state is summarized by a Merkle tree root, periodic signed tree heads, and inclusion and consistency proofs of logarithmic size.

Hash chain

Each record binds the hash of the previous one, so altering any record requires altering every later record — the time-stamping idea behind tamper evidence.

Merkle tree / signed tree head

A root hash commits to all entries; a signed tree head is (size, root, time, signature). Inclusion and consistency proofs cost a logarithmic number of hashes.

Consistency proof

Lets an auditor check that the history behind an earlier root is a prefix of the later one — detecting rewrite, deletion, reordering and rollback.

Split view (equivocation)

An operator showing different auditors different, each-self-consistent histories. A single isolated auditor cannot detect it; auditors must compare signed heads.

synthetic — a hash-chained decision-log entry with a salted content commitment
import hashlib, json

def canonical(obj):                       # deterministic bytes: sorted keys, UTF-8
    return json.dumps(obj, sort_keys=True, separators=(",", ":")).encode()

def append_entry(prev_hash, event, content, salt):
    # Content (which may hold personal data) is committed, not stored inline,
    # so it can be deleted on retention while the log stays verifiable.
    commitment = hashlib.sha256(salt + canonical(content)).hexdigest()
    entry = {
        "prev":       prev_hash,          # chain link: breaks if any earlier entry changes
        "actor":      event["actor"],     # who acted
        "principal":  event["principal"], # on whose behalf (delegation chain)
        "version":    event["version"],   # model/prompt/policy/tool versions used
        "action_fp":  event["action_fp"], # fingerprint of the canonical, approved action
        "commitment": commitment,         # salted hash of the content, deletable later
    }
    leaf = hashlib.sha256(b"\x00" + canonical(entry)).hexdigest()  # 0x00 = leaf prefix
    return entry, leaf

# The model does NOT write the log; deterministic components emit events and the
# executor obtains an index BEFORE producing an effect, so completeness is checkable.
# Deleting (salt, content) on retention leaves the commitment -> still verifiable,
# no longer complete. The salt stops brute-forcing low-entropy content like an IBAN.

From the proofs follows what an auditor detects: keeping the last accepted (size, root) and only accepting a larger size with a valid consistency proof catches any rewrite, deletion, reordering or rollback of earlier events. What a lone auditor cannot catch is the split view, which needs auditors in different trust domains to compare signed heads — two valid heads of equal size with different roots are transferable proof of equivocation. On top of the log, recording provenance (which entity each step used and generated) lets a decision be reconstructed by taking the provenance closure of an effect and verifying each step's inclusion proof; and because the decision lived in a versioned policy, it can be re-computed from the logged inputs rather than merely explained. The forensic procedure is: fix a signed head an external auditor already accepted, obtain inclusion proofs, verify consistency to today, reconstruct the closure, record gaps and retention-deleted content, and reproduce the decisions.

The controls compose into a reference architecture organized by zone. Zone 1, perception, holds the model with no credentials: a quarantined extractor and a privileged planner that never reads untrusted content. Zone 2, deterministic decision, holds the validator (canonical intent with per-argument provenance), the policy point computing the intersection of principal, agent and task, the information-flow checks at the sinks, and the risk ladder with approval bound to the canonical action. Zone 3, execution, holds a single executor with credentials, a sandbox for generated code, monitors and a circuit breaker at the enforcement point, and the verifiable log. The model proposes in zone 1; zone 2 decides with data the model does not control; zone 3 executes what was decided and nothing else.

Each control carries an explicit guarantee, a set of assumptions it depends on, and a pointer to where it is justified. The authority chain guarantees every effect is authorized by policy — under deployment assumptions, not that the policy is correct. Attenuated delegation guarantees nobody acts with authority the chain did not grant — assuming the identity provider is intact. Information-flow and isolation guarantee untrusted content neither chooses nor parameterizes an effect, and that memory does not whitewash labels or cross tenants — assuming origin labels are correct. Bound approval guarantees what executed is what was approved — assuming the approver reviews what they sign, the most fragile assumption of all. Monitors and the log guarantee enforcement and reconstruction — assuming every effect passes the enforcement point. Formal verification and red teaming add no coverage of their own; they reduce the chance that the controls carrying guarantees are themselves defective.

• ▸PROVEN (statistically, over the tested distribution): zero failures in n independent cases with an exact oracle bounds the rate by about 3/n at 95%; with an imperfect oracle of sensitivity Se the bound inflates to roughly -ln(alpha)/(n·Se).
• ▸NOT PROVEN (outside the tested support): a campaign implies no bound on any distribution it did not sample — an adaptive adversary is exactly such a distribution, which is why the architectural guarantee is different in kind.
• ▸PROVEN (cryptographically, barring a hash collision): inclusion and consistency proofs let an auditor detect rewrite, deletion, reordering and rollback; tamper evidence means detectable, not impossible.
• ▸NOT PROVEN by the log: a split view needs auditors comparing heads across trust domains; completeness is an architectural property (what was emitted), and a faithfully recorded falsehood is still false.
• ▸Governance bounds only what it covers: an agent outside the inventory, or evidence not bound to the version, silently voids the property; the classification's force depends on the version being observable.
• ▸UNKNOWN / common cause: the reference architecture's effective depth is governed by shared assumptions (correct labels, a reviewing approver, complete mediation), and these have not been evaluated as a whole outside synthetic, illustrative settings.

Expected direction: Rule of three: 0 of 200 bounds the rate at about 3/200 = 1.5% at 95%, and only over the distribution of cases actually tested. An imperfect judge of sensitivity Se inflates the bound to roughly -ln(0.05)/(200·Se), and any threat category with no cases has no estimate at all, so "not tested" must not be read as "safe". Governance: raising to autonomous is a change whose risk class is recomputed (monotone, so it does not drop), and because the autonomy change (and any model change) alters a version component, evidence bound to that component is no longer valid, so the promotion is blocked until evidence is re-earned on the new configuration; the change also needs approvers distinct from the author. Decision log: each decision and effect is appended with versions, the bound action fingerprint and a salted commitment, chained and summarized by a signed tree head an external auditor accepts, so any later rewrite fails a consistency proof and the pre-change decisions can be reconstructed and re-computed from the logged policy version. Shared assumption: a correct origin label — a flow monitor, a planner-isolation control and a labelled memory can all depend on it, so one wrong label (e.g. a third-party connector registered as internal) defeats all three together; nominal depth three, effective depth one.

What does "zero failures in n cases" actually tell you, and what does it not?

With an exact oracle it bounds the violation rate by about 3/n at 95% over the distribution that was sampled. It tells you nothing about any distribution outside that support, so an adaptive adversary who searches outside the tested cases is not bounded by the result.

Why must audit evidence be bound to a version and expire, and what does that block?

Because changing the model, prompt, tool or policy can change what the agent does, so evidence obtained with an old version no longer describes the system. Binding and expiry block promoting or continuing an agent on stale evidence — e.g. raising autonomy "because it is only a model change" is blocked until evidence is re-earned on the new version.

What does a Merkle-tree audit log with consistency proofs detect, and what still needs multiple auditors?

An auditor keeping the last accepted (size, root) detects any rewrite, deletion, reordering or rollback of earlier events via the consistency proof. It cannot detect a split view (equivocation) alone; that needs auditors in different trust domains comparing signed tree heads, where two valid heads of equal size with different roots prove equivocation.

In the reference architecture, why can three controls give a defence-in-depth of one?

Because defence in depth counts independent failures, not controls. If three controls share an assumption — such as a correct origin label — a single common-cause failure (a wrong label) defeats all three at once, so the effective depth against that failure is one even though the nominal depth is three.