Case study — Evaluation deep-dive

Read the paper’s empirical section critically: the 70-threat coverage matrix, the single-target audit with its reported (not reproducible) figures, the measured external-generalization result where precision collapses off-target, and the experimental frontier red-team module. The goal is to sort every number into proven-structural, tested, measured, reported, or experimental — and to see why a valid seal proves authenticity, not correctness.

Track

Research

Module

5 · Research case studies

Prerequisites

Case study — Deterministic verification over LLM-as-judge (Module 5, lesson 2) · Case study — Architecture & algorithms deep-dive (Module 5, lesson 4) · Reading precision / recall and false-positive rates

Before any number, the paper fixes a vocabulary for how strongly each claim is supported, and applies it to itself as strictly as it would to a target. Internalising these labels is the whole skill of this lesson — a figure means something different under each.

MEASURED

A number or artefact in the codebase supports it and is cited by path. Reproducible from the source tree.

REPORTED

A written project report states it, but the raw run is not reproducible from the source tree (for example an external target with no committed logs).

IMPLEMENTED

The code exists but its performance is not measured.

DESIGN

Specified, but not implemented or not measured.

PRELIMINARY

Measured once, on the system’s own or a synthetic corpus, not independently.

The paper reports a coverage matrix that is generated from the code rather than claimed in prose: for each threat it counts whether an invariant, a verifier, a mission and an end-to-end test exist, and derives status from live registries. The generator’s rule is that a framework tag is not coverage — being mapped to a published threat taxonomy does not count unless the chain from attack surface through verifier to an end-to-end evidence test is actually present.

coverage of a 70-threat catalogue (as the paper reports it)
catalogue: 70 GenAI threats, mapped to published taxonomies
  57  verified     (invariant + verifier + mission + end-to-end test)
  11  implemented   (not end-to-end)
   1  partial
   1  mapped only
   0  gaps
note: older totals (e.g. 52 or 64) are superseded book-chapter counts
      and should not be cited.

The main empirical evidence is one documental-AI backend used as the assessment target (a retrieval-and-classification system, anonymized by the paper). A caveat governs this entire section: the figures are reported — stated by project reports and not reproducible from the source tree — because the target is external and no raw run logs are committed. The paper cites each figure against the report that contains it and treats it as what the report stated, not as a re-executable artefact.

The decomposition matters more than the headline. The paper reports that of a large set of candidate hypotheses, the majority were promoted after a provenance filter removed a block that had stopped promoting — but that this promotion count comes from two pools that must not be conflated. One pool was a large set of candidate "poisonable-store" findings, all of a synthetic adjacency class that a verifier could only confirm by re-running the query that produced them (circular, carrying no real dataflow path). A separate, much smaller pool of genuinely re-derived findings carried real def-use dataflow paths and survived. The paper’s empirical core is that the small surviving count is more informative than any raw total.

the decomposition (shape, not exact figures — all REPORTED)
candidate hypotheses  --(provenance filter)-->  promoted
   large circular pool   -> synthetic adjacency -> confirmable only by
                            re-running its own query -> NOT a real finding
   small re-derived pool -> real def-use dataflow -> the informative result

root-cause buckets among survivors (reported): hardcoded secrets,
   weak cryptography, time-of-check/time-of-use file access,
   non-atomic shared state, and a single critical server-side template
   injection. A live red-team run against the target is reported as
   zero successful landings across fourteen attacks, with one apparent
   "landing" hand-verified to be a detector artefact, not obedience.

The paper’s most scientifically interesting result is negative, and it reports it precisely because it is. When the deterministic verifiers were taken off the familiar target and run over a held-out corpus of external repositories, recall degraded gracefully but precision collapsed.

held-out external evaluation (MEASURED, per the paper)
corpus: ~26 in-domain repos tuned on; ~40 held-out external repos
micro-recall     ~0.339  ->  ~0.196   (degrades gracefully;
                                        every external repo still yields
                                        at least one true positive)
micro-precision  ~0.497  ->  ~0.102   (collapses)
false positives per repo   ~9    ->  ~51

dominant over-firing detectors off-target:
  - a retrieval-scoping detector (fires on code absent from the corpus)
  - a hardcoded-secrets detector
  - an authorization detector (trips labelled false-positive traps)
caveat the paper raises on itself: in-domain precision was partly
  INFLATED because the familiar corpus lacked negative traps.

The paper argues this cuts both ways, and this double reading is the lesson. It supports the thesis: without a strict promotion gate that leaves un-re-derived accusations as UNKNOWN, an off-target run would flood a user with false findings — the collapse is exactly what the membrane and the circular-finding filter are built to contain. And it bounds the thesis honestly: the deterministic oracles are not target-independent, and an off-target micro-precision near 0.10 with roughly fifty false positives per repository is a real limitation, not a footnote.

The paper describes a laboratory module for frontier red teaming (referred to in the live system as NEMESIS): an autonomous adversarial investigator operating inside an authorized cyber-range, with global status reported as experimental and lab-only. Its structure restates the thesis in an interactive setting: the attacking agent starts from an agent view with no topology or ground truth and can never self-declare success; a separate Tribunal, running outside the agent’s reach, holds the scenario ground truth and is the sole authority on what happened.

The Tribunal’s oracle is deterministic — equality or membership against ground truth, append-only, no clock and no model — with explicit anti-forgery and anti-replay defences: a claim about a non-existent domain is FORGED, already-seen evidence is REPLAY. A separate swarm-collusion detector adjudicates whether a set of agents colludes, scored only when the Tribunal verifies the pattern against ground truth so a forged claim counts as a false positive, not a hit.

This is the same discipline as a digital signature in general: it attests authorship and integrity of the bytes, not the truth of the statement inside them. In the evaluation it means a client can confirm the report is the one the pipeline produced, unchanged — and still must read every figure at its stated status. Verifier correctness is assumed, not proven, so even a perfectly sealed, deterministically re-derived finding could be wrong if the oracle’s logic is wrong. Custody closes the gap between "what the pipeline produced" and "what I received"; it does nothing about the gap between "re-derived" and "true".

• ▸PROVEN-structural: no finding without a FAIL verdict and matching evidence; no agent writes a finding — properties of the object graph (conditional on all creation going through the typed paths).
• ▸TESTED: zero model calls and zero cost on the deterministic promotion/verification path; an import barrier against LLM clients in the verification layer; the end-to-end signed-report path.
• ▸MEASURED: the 70-threat coverage matrix (57 fully verified); the held-out precision/recall collapse (reproducibility of the per-repo artefact still open).
• ▸REPORTED: the single-target promotion ratio, root-cause histogram, and zero-landings-in-fourteen-attacks — from written reports, not committed logs.
• ▸EXPERIMENTAL / PRELIMINARY: the frontier red-team module and Tribunal; the collusion detector’s self-corpus metrics with zero registered missions.
• ▸OPEN: an independent second target; a controlled true/false-positive comparison against existing red-team tools; a versioned per-repo generalization artefact; a durable Merkle store; a recorded offline verification against a published key.

Expected direction: Honest version (shape): "On one internal documental-AI target, project reports state that a majority of candidate hypotheses were promoted after filtering, of which only a small pool were genuinely re-derived real-dataflow findings; a reported red-team run saw zero successful landings across fourteen attacks. These are reported figures from written reports, not reproducible from source, and the report’s signature proves the record is authentic, not that the findings are correct." Overstatements: (1) "independently verified" — it is a single internal target, not independent; (2) "1,000+ vulnerabilities" conflates the large circular/synthetic pool with the small re-derived pool; (3) "signed and tamper-proof" implies correctness — the seal proves only authenticity/integrity of the record; (4) "zero successful attacks" is a REPORTED figure, and absence of a found path is UNKNOWN, not proof the target is secure.

What is the difference between a MEASURED and a REPORTED figure in this paper, and which one are the single-target audit numbers?

MEASURED is backed by a reproducible artefact in the codebase; REPORTED is stated by a written report but not reproducible from source. The single-target audit numbers are REPORTED (external target, no committed raw logs).

What happened to recall and precision when the verifiers were run on held-out external repositories?

Recall degraded gracefully (micro-recall ~0.339 to ~0.196, with every external repo still yielding at least one true positive) but precision collapsed (micro-precision ~0.497 to ~0.102; false positives per repo rising from ~9 to ~51).

Why does the paper report this negative result as both supporting and bounding its thesis?

Supporting: the collapse is exactly what the strict gate and circular-finding filter exist to contain, so a user is not flooded with false findings off-target. Bounding: it proves the oracles are not target-independent, making off-target precision near 0.10 a real limitation.

How mature is the frontier red-team module, by the paper’s own labels?

Experimental and lab-only: a synthetic arena, a binary containment pass without measured frontier-model pass/fail counts, and a collusion detector with perfect metrics only on its own six-scenario synthetic corpus (PRELIMINARY) and zero registered missions.

Does sealing a reported figure with an Ed25519 signature make it reproducible or correct?

No. The seal proves the record is authentic and untampered; it does not change the figure’s status. A sealed REPORTED figure is still reported, and a re-derived finding can still be wrong if the verifier is wrong.