Threat model, limitations & practitioner synthesis

Understand the Detection Separation Property as the load-bearing assumption of the whole architecture, what concretely breaks if it fails, the threat model the paper scopes, the honest limitations and open problems, and the transferable lessons for engineers building AI-augmented analysis pipelines.

Track

Research

Module

5 · Research case studies

Prerequisites

Architecture & algorithms deep-dive (lesson 4) · Evaluation deep-dive (lesson 5)

Every formal guarantee in this architecture — recall preservation and no-hallucination-introduction — rests on a single property the paper calls Detection Separation: the refinement layer may only remove or re-label findings that originate from a tool; it may never add a finding that did not come from one. This is load-bearing. If it holds, the structural guarantees follow. If it silently fails, both guarantees evaporate while every architecture diagram still looks correct.

The property is violated the moment any post-detection component writes a finding not traceable to a tool's concrete evidence. Concrete ways this can creep in:

• ▸An enrichment or correlation agent that, while "combining" observations, emits a new Finding record describing a vulnerability it inferred rather than one any tool grounded — a hallucination with no provenance.
• ▸A summary agent that promotes a speculative remark in prose into a tracked finding downstream consumers treat as detected.
• ▸A remediation step that "notices" an unrelated issue and injects it into the finding set instead of leaving detection to tools.
• ▸A merge/dedupe bug that fabricates a synthetic finding from fragments of two unrelated ones, losing the provenance link.

In each case the no-hallucination theorem no longer applies, because a surviving finding cannot be traced to a tool. The defense is architectural and testable: enforce that every Finding leaving layer 4 carries a non-empty provenance link to at least one layer-2 tool output, and treat any finding without one as a bug, not a feature. The guarantee you advertise must match the invariant you actually enforce.

The paper scopes threats to two systems: the analysis pipeline itself, and the contracts it audits. For the pipeline it names three trust boundaries worth generalizing to any AI-augmented analyzer.

Adversarial input to the LLM

Maliciously crafted source could try to manipulate the agents via prompt injection. Mitigation: agents receive structured, normalized Finding objects rather than raw source, and untrusted source text is confined to clearly-marked evidence fields.

Tool supply chain

A compromised detector could emit false output. Mitigation: pin tools to verified versions and require multi-tool agreement before escalating the most severe findings.

LLM non-determinism

Stochastic outputs could yield different results on identical input. Mitigation: low temperature and measuring run-to-run variance — though the paper acknowledges it did not fully characterize this (see limitations).

Out of scope, by the paper's own statement: network-level attacks on the infrastructure, side-channels on the model API, and zero-day flaws in tool dependencies. Naming what is out of scope is itself good practice — it prevents readers from assuming coverage that was never claimed.

The paper is unusually candid about its own threats to validity. These are the ones a practitioner must carry forward, each a LIMITATION:

• ▸Ground-truth construction: labels came from an AI-assisted classifier plus a single expert reviewer. The paper discloses two biases — shared-LLM bias (the same family of model used both in the system and to help build ground truth can inflate agreement) and experimenter bias (the system's own author performed manual review). Inter-rater reliability with an independent evaluator was not measured.
• ▸LLM non-determinism: results use one backend at low temperature; run-to-run variance across repeated executions was not fully reported, and confidence intervals are absent.
• ▸Benchmark scale: the corpus is modest (a few dozen contracts) relative to some related work, which weakens generalizability of the accuracy numbers even though the vulnerability taxonomy is broad.
• ▸Model and tool dependency: results may differ with other models; specific tool versions were tested and updates could change outcomes; the pipeline depends on an external model service being available and incurs a per-contract API cost.
• ▸Partial automation: exploit skeletons require manual refinement, and a meaningful share of exploits cannot be auto-reproduced (the multi-transaction / external-state failure modes from lesson 5).

• ▸OPEN: cross-domain generalization. The paper presents extensions to other domains (general code review, document analysis) as design guidelines, explicitly not empirically validated.
• ▸OPEN: whether the measured accuracy and verification rates transfer to larger, more diverse corpora, other protocols, or other models.
• ▸OPEN: reducing the irreducible LLM-modeling-error share in verification — this tracks model capability, which architecture alone cannot fix.
• ▸OPEN: independent, multi-auditor validation of the accuracy claims, which the paper names as necessary future work.

Distinguishing these from the settled structural guarantees is the whole discipline of reading the paper honestly: the architecture's correctness properties are settled (under their assumption); almost everything quantitative is corpus-specific and some things are frankly open.

If you are building an AI-augmented analysis pipeline, the transferable lessons distill into a short, opinionated checklist:

1. 1Keep detection and understanding separate: let deterministic tools originate findings; let the LLM only filter, correlate, enrich. Advertise only the guarantee your enforced invariant actually supports.
2. 2Enforce provenance mechanically: every finding that leaves the pipeline must trace to a tool's concrete evidence; reject any that does not. This is how you keep the no-hallucination property real rather than aspirational.
3. 3Make orchestration content-agnostic so tools are swappable and coordination is independently testable.
4. 4Normalize to one schema before any model runs, and give agents one shared context instead of re-deriving state.
5. 5Order agents cheap-to-expensive: prune false positives first so expensive reasoning works on a clean set.
6. 6Prefer execution-grounded verification over model self-judgement: let a harness, not an LLM, decide whether a claim holds; generate candidates with more than one model to cover blind spots.
7. 7Report a failure taxonomy, not just a success rate, and never let a non-reproduction be read as a false positive.
8. 8Label every claim proven / measured / open, disclose your ground-truth process and its biases, and treat independent validation as unfinished until it actually happens.

The deepest takeaway is a trust discipline: an AI component is safe to rely on exactly to the extent that its ability to cause harm is structurally bounded and that bound is enforced and tested. The paper's value is less its numbers than its demonstration that you can give an LLM a real, useful job (refinement) while provably denying it a dangerous one (origination) — provided you actually enforce the line.

Expected direction: Invariant: every Finding emitted by the refinement layer must carry a non-empty provenance link to at least one detector output (by id); assert it at the pipeline boundary. CI test: feed inputs through the pipeline and fail the build if any output finding has empty/unresolvable provenance, or if the set of output finding-ids is not a subset (modulo relabeling) of the detector-grounded candidate ids. A failure proves the Detection Separation Property does not hold in the implementation — the refinement layer is originating findings — which voids the no-hallucination guarantee regardless of how correct the architecture diagram looks.

What is the single assumption the architecture's formal guarantees depend on?

The Detection Separation Property: refinement may only remove or re-label tool-originated findings, never add new ones.

Give one concrete way Detection Separation can silently fail.

A correlation/enrichment agent emits a new finding it inferred rather than one grounded by a tool, so a surviving finding has no provenance — a hallucination enters despite every layer being present.

Name the two biases the paper discloses in its ground-truth construction.

Shared-LLM bias (same model family used in the system and to help build ground truth) and experimenter bias (the system's author performed the manual review); inter-rater reliability with an independent evaluator was not measured.

Which pipeline-threat mitigation reduces prompt-injection risk, and how?

Agents consume structured, normalized Finding objects rather than raw source, confining untrusted source text to marked evidence fields so it is data, not instructions.

What is the deepest transferable discipline from the case study?

Make an AI component's ability to cause harm structurally bounded, then enforce and test that bound — give the LLM a useful job (refinement) while provably denying it a dangerous one (origination).