Case study — Threat model, limitations & practitioner synthesis

Close the case study by reasoning about what the design still cannot promise: that a verifier can be wrong even when it is deterministic, what UNKNOWN actually protects against, why circular findings are the quiet failure mode, and what an engineer building an evidence-gated agent pipeline should concretely copy, avoid, and measure.

Track

Research

Module

5 · Research case studies

Prerequisites

Case study — Architecture & algorithms deep-dive (Module 5, lesson 4) · Case study — Evaluation deep-dive (Module 5, lesson 5)

The membrane guarantees that every finding was produced by a FAIL-plus-evidence verifier and that no agent wrote it. It does not guarantee the verifier is right. The paper states this plainly as its first limitation: verifier correctness is assumed, not proven, and a wrong oracle produces a wrong finding with full custody. The design has relocated trust — from a stochastic model judge to a deterministic re-derivation — which is a real improvement in reproducibility and auditability, but trust still has to live somewhere.

This is why the custody layer, however strong, does not help here. An Ed25519 signature and a Merkle anchor prove the record is authentic and untampered; they attest the pipeline’s output faithfully, including a faithfully recorded wrong finding. The two trust questions — "is this the record the pipeline produced?" (custody, answered) and "is the re-derivation logic sound?" (verifier correctness, assumed) — are independent, and only the first is closed.

The most instructive threat in the paper is not an attacker but a self-confirming finding. A circular finding is one that a verifier can only "confirm" by re-running the very process that produced the belief — so the confirmation adds no independent evidence. The paper’s audit surfaced a large pool of these (a synthetic adjacency class with no real dataflow path) and filtered them out, keeping only a small pool whose findings were re-derived by an independent path.

the distinction, made operational
# CIRCULAR: confirmation re-uses the belief's own source
belief   = detector_X(query_Q)            # produced the hypothesis
"confirm"= detector_X(query_Q) again      # same source -> no new evidence
=> verdict UNKNOWN; NOT a finding

# RE-DERIVED: confirmation comes from an independent derivation
belief   = agent_guess("param reaches sink")
confirm  = reopen_source(); recompute_def_use_path()   # independent
=> verdict FAIL with a real path trace -> a finding

The lesson for any evidence-gated system is that "has evidence" is too weak a bar; the bar must be "has independent evidence." A pipeline that counts a belief and its own echo as two supports will confidently promote nonsense. The paper’s provenance filter — removing candidates confirmable only by re-running their own query — is the concrete defence, and it is what makes the surviving count meaningful.

Making UNKNOWN a first-class terminal state is the design’s main asymmetry: the system would rather under-claim than over-claim. Two concrete protections follow. First, it cannot report a target clean because an agent failed to find a path — absence of a found path is UNKNOWN, never PASS, so the system never launders "we did not find it" into "it is not there." Second, budget or time exhaustion forces UNKNOWN and never CONFIRMED, so resource limits are recorded as not-knowing rather than as safety.

the asymmetry as transition rules (as the paper reports them)
no path observed    -> UNKNOWN        # never PASS ("clean")
budget exhausted    -> UNKNOWN        # never CONFIRMED
unsettled transcript-> UNKNOWN        # never a laundered finding
UNKNOWN is terminal and non-promotable

The cost is equally explicit: recall. Anything the oracles cannot settle stays UNKNOWN, so a real vulnerability with no available oracle is simply not reported as a finding. The paper frames the whole design as a position — that in security, under-claiming is preferable to over-claiming — and insists it is a position, not a free lunch. An engineer adopting this stance is choosing a false-negative-tolerant, false-positive-averse posture, which is defensible for high-trust reporting and wrong for exhaustive discovery; the choice should be deliberate.

Reasoning adversarially about the architecture (not the targets) surfaces where trust concentrates. These are the attack surfaces a reviewer should probe, paired with what the design does and does not do about each.

A wrong-but-deterministic verifier

Defence: none structural — this is the assumed-trust core. Mitigation is review and replay, not proof. The risk is a confidently sealed false finding.

Target over-fitting (precision collapse)

Defence: partial. The measured off-target collapse shows a handful of detectors over-fire on unfamiliar code; the strict gate contains the flood but does not fix the detectors. Mitigation: negative traps and held-out evaluation.

Bypassing the mint path

Defence: structural. The finding type re-checks promotability in its constructor and copy hooks, so hand-building a finding fails. This is the design’s strongest guarantee (conditional on all creation going through typed paths).

Custody tampering

Defence: strong but partly unmeasured. Content digests, an append-only Merkle chain and an Ed25519 seal give tamper-evidence; but the chain lives in memory with only anchors persisted, and an offline verification against a published key is not yet a recorded field run.

Colluding or self-promoting agents (interactive)

Defence: experimental. The frontier red-team Tribunal separates agents from the authority and counts evidence not votes, with forged/replay defences — but it is lab-only, synthetic, and the collusion detector is measured only on its own corpus.

• ▸OPEN: an independent, third-party target besides the single internal one — the strongest mitigation for self-selection and favourable tuning.
• ▸OPEN: a controlled true/false-positive comparison against existing red-team tools on a shared labelled target; none exists in the system today, so no head-to-head benchmark is reported.
• ▸OPEN: a versioned, machine-readable per-repository artefact for the external-generalization result (today it lives in prose and a non-git directory pinned by a freeze manifest).
• ▸OPEN: a durable external Merkle store (the chain currently lives in memory) and a documented offline verification of a signed report against a published public key.
• ▸OPEN: held-out collusion metrics on a corpus distinct from the six synthetic scenarios, and at least one registered mission so the importable collusion engine runs a non-zero mission count.
• ▸LIMITATION (not closable by more engineering): verifier correctness remains assumed; the design reduces how much you must trust, not whether you must trust.

Stripped of its security framing, this paper is a template for any multi-agent system where models propose and something must decide. The transferable discipline is a short list.

1. 1Make the authority deterministic and singular. Let agents and models propose freely; mint the authoritative object through exactly one rule-based path, and put the check in the type so the illegal state cannot be constructed.
2. 2Require independent evidence, not merely evidence. Filter out self-confirming (circular) results whose only support is a re-run of the belief’s own source; count independent derivations.
3. 3Make "don’t know" a first-class output. Record UNKNOWN with a cause; never let a failed search imply safety, and never let budget exhaustion imply a pass.
4. 4Keep verdict vocabularies separate and give only one authority. Track lifecycle state apart from the promoting verdict so you always know which component decided.
5. 5Seal for custody, and say so precisely. Canonicalize, digest (never over secrets), chain and sign — then state that the seal proves authenticity and integrity, not correctness.
6. 6Label every claim by evidence status, including your own. Distinguish proven-structural, tested, measured, reported, experimental and open — and when your own log is missing, say not-established, not refuted.
7. 7Budget for the recall you are trading away. A false-positive-averse gate drops real findings it cannot settle; decide consciously whether your use case tolerates that.

Expected direction: Flagging twice re-uses the belief’s own source, so the second flag adds no independent evidence — it is a circular finding that will confidently promote nonsense (the paper’s synthetic-adjacency failure mode). Redesign: confirm only when an independent derivation reproduces the defect — e.g. re-compute the metric from raw data via a different code path, or check the row against an external constraint the agent did not use — so the evidence does not trace back to the original guess. Missed class: defects with no independent oracle (e.g. a subtle semantic error only a human or the original model can judge) now stay UNKNOWN rather than being promoted. Whether that is acceptable depends on posture: acceptable for high-trust reporting where false positives are costly; not acceptable if the goal is exhaustive discovery, in which case UNKNOWNs must be routed to human review rather than dropped.

The membrane guarantees no agent writes a finding and no finding lacks evidence. What does it still not guarantee?

That the verifier is correct. Verifier correctness is assumed, not proven — a wrong deterministic oracle produces a wrong finding with full custody.

What makes a finding "circular," and why is "has evidence" too weak a bar?

A circular finding can only be confirmed by re-running the process that produced the belief, so its evidence traces back to itself and adds nothing independent. "Has evidence" permits self-echoes; the real bar is independent evidence from a separate derivation.

Give the two concrete things making UNKNOWN terminal protects against, and the price paid.

It prevents reporting a target clean just because an agent found no path (absence is UNKNOWN, not PASS), and it records budget/time exhaustion as not-knowing rather than safety. The price is recall: real vulnerabilities with no available oracle go unreported as findings.

Why does strong custody not reduce the residual verifier-correctness risk?

Because custody answers "is this the record the pipeline produced, unchanged?" while correctness answers "is the re-derivation sound?" — independent questions. A seal faithfully attests a faithfully recorded wrong finding.

Name two of the paper’s open problems that an independent reviewer would most want closed.

An independent third-party target (to counter self-selection and favourable tuning), and a controlled true/false-positive comparison against existing red-team tools on a shared labelled target; also acceptable: a versioned per-repo generalization artefact, a durable Merkle store, or a recorded offline signature verification.