Evaluation deep-dive — setup, ablation, verification rates, failure taxonomy

Read the paper's empirical evaluation like a reviewer: understand the experimental setup and its research questions, interpret each measured result with its caveats, trace the ablation that attributes the F1 jump to refinement, and internalize why "unverified" is not "false positive".

Track

Research

Module

5 · Research case studies

Prerequisites

The refinement pipeline & unified schema (lesson 2) · Generative exploit verification (lesson 3)

A results section is a set of claims with evidence attached, and a careful reader asks three things of every number: what exactly was measured, on what data, and what would change it. This lesson applies that lens to the paper. Keep one rule in mind throughout: numbers here are reported outcomes on one specific corpus with one specific model backend, not universal constants. Where the paper proves something, it is proven only under its stated assumption; where it measures something, it is measured and transferable only with care.

The paper organizes its evaluation around four research questions: (RQ1) what speedup does parallel execution achieve; (RQ2) how much does LLM refinement improve precision/recall; (RQ3) what does the combined system detect that individual tools miss; and (RQ4) what is the success rate of automated exploit generation.

Corpus

A few dozen smart contracts (on the order of ~50) drawn from a known deliberately-vulnerable DeFi benchmark plus real-world protocol samples — thousands of lines of code, a spread of vulnerability types.

Toolset

Around two dozen components: traditional detectors (pattern matching, symbolic execution, fuzzing, decompilation), a set of domain-focused modules, and the LLM agents — orchestrated over the distributed runtime.

Model backend

A single commercial LLM at low temperature for the agents; a second model is added only in the verification stage for diversity.

Ground truth

Each finding was classified true/false positive by an AI-assisted step plus manual expert review, with publicly documented solutions used as extra validation where available.

MEASURED: the paper reports parallel execution cutting wall-clock time by roughly 7x versus running all tools sequentially (on the order of ~160 minutes down to ~23 minutes for tool execution), with the full pipeline including agents and verification completing in under about half an hour. It reports throughput of a couple of contracts per minute.

How to read it: this confirms the speedup model from lesson 4 held in practice for this toolset, where a few slow symbolic/fuzzing tools dominate. It does NOT promise 7x elsewhere — a different mix (e.g., one extremely slow tool) would push the ratio toward the slowest tool, exactly as the Amdahl bound predicts.

MEASURED: the raw union of all tools scored high recall but low precision (many false positives), for an F1 around 0.55. After LLM refinement the paper reports precision rising sharply, recall staying high, and F1 rising to about 0.86 — with the headline being roughly a 54% reduction in false positives and a large precision improvement as the dominant driver.

the shape of the reported accuracy change (illustrative, re-expressed)
raw tool union :  precision low   recall high   F1 ~0.55   FP-rate high
after refine   :  precision high  recall high   F1 ~0.86   FP-rate ~ -54%

dominant factor: precision (false-positive reduction), not extra recall

The interpretation the paper draws, and the one worth keeping: once recall is adequate, the marginal value is in discarding false positives, not in detecting more. An LLM used as a discriminator over grounded candidates buys more here than another detector would.

MEASURED: the paper breaks down where validated findings came from. A single fast pattern-matcher dominates raw detection volume (a majority of findings), the remaining traditional tools add a modest single-digit percentage, domain-focused modules contribute a slice of DeFi-specific findings, and a double-digit percentage of validated findings came only from the LLM correlation/attack-chain agents — i.e., from combining partial observations rather than from any single tool's primary detection.

Read carefully, this is the empirical face of the correlation walkthrough in lesson 4: the agents are credited not with originating findings (they cannot) but with surfacing compound vulnerabilities that emerge from cross-referencing multiple partial, tool-grounded observations.

The most load-bearing experiment is the ablation: starting from the fastest tool alone and adding one capability at a time, measuring F1 at each step.

1. 1Pattern-matcher alone: high finding count, high recall, but F1 around 0.50 due to false positives.
2. 2Add symbolic execution: a modest F1 gain (depth, but more candidates).
3. 3Add the remaining traditional tools: diminishing returns per tool — small F1 movement.
4. 4Add LLM refinement: the single largest F1 jump (the big precision gain), reported as the dominant step.
5. 5Add domain-focused modules: a further gain from DeFi-specific coverage.
6. 6Add cross-tool LLM correlation: reaching the full-system F1 of about 0.86.

MEASURED: targeting Critical and High findings, the paper reports roughly 68% automatically verified with a runnable exploit (about 39 of 57), and roughly 64% overall when a sample of Medium findings is included (about 84 of 132). Average generation time is on the order of seconds, with a small average number of refinement iterations.

Why these are stronger evidence than detection numbers: a verified finding is backed by an executed test, so it does not depend on the ground-truth classifier at all — the harness is the arbiter. That makes the verification rate the most robust number in the paper, though still corpus- and model-specific.

The paper categorizes why verification failed on the unverified Critical+High findings. This is the most important lesson in the whole evaluation for a practitioner.

Multi-transaction complexity (~44%)

The exploit needs a sequence beyond a single test's scope (timelocks, multi-block ordering). The finding is likely real; the harness just cannot express it yet.

External state dependency (~33%)

The exploit needs external protocol state absent from a local fork (oracle prices, other protocols' liquidity). Again, not evidence of a false positive — a fork/snapshot limitation.

LLM modeling errors (~22%)

The generator mis-modeled contract logic (wrong storage-layout or inheritance assumptions). The paper calls this the current irreducible ceiling of LLM-based code generation — but even this is a generation failure, not proof the vulnerability is absent.

So the correct inference from "unverified" is "not reproduced in this harness", which is categorically different from "false positive". The paper reports that the first two failure modes are addressable with better tooling (multi-step orchestration, mainnet fork snapshots), while the third tracks model capability. Reading the verification rate as a false-positive rate would be a serious misinterpretation.

• ▸PROVEN (only while the Detection Separation Property holds): recall is preserved up to the triage agent's false-negative rate; no hallucinated finding can enter the output. These are structural theorems about the architecture, not about any measurement.
• ▸PROVEN (as arithmetic): the Amdahl/Gustafson speedup relations.
• ▸MEASURED (one corpus, one backend): ~7x speedup; F1 ~0.55 -> ~0.86; ~54% false-positive reduction; the detection-source breakdown; the ablation's attribution to refinement; ~68% Critical+High and ~64% overall GAEV verification; the failure-mode percentages.
• ▸OPEN: whether any of the measured numbers transfer to other domains, corpora, protocols or models — the paper does not claim they do.

When you evaluate your own AI-augmented pipeline, copy the method, not the numbers: ablate so you can attribute gains to a specific component rather than to "the system"; prefer execution-grounded metrics (did the generated artifact run?) over classifier-graded ones wherever possible, because they do not inherit your ground-truth bias; and always report a failure taxonomy, since "why it failed" tells you where to invest and prevents readers from misreading a non-result as a negative result. Above all, label each claim proven, measured, or open — the way the paper does — so nobody downstream over-generalizes a corpus-specific figure.

Expected direction: Supports: "On this corpus, about 68% of Critical+High findings could be automatically backed by a runnable exploit test." Does not support: "About 32% of Critical+High findings are false positives." Justification: the failure taxonomy shows most unverified cases failed due to multi-transaction complexity (~44%) and external-state dependency (~33%) — harness limitations, not evidence the findings were false — and even LLM modeling errors (~22%) are generation failures, not proof the vulnerability is absent. So the 32% is an "unverified in this harness" rate, not a false-positive rate.

What are the paper's four research questions, in brief?

RQ1 speedup from parallelism; RQ2 accuracy improvement from LLM refinement; RQ3 coverage the combined system adds over single tools; RQ4 automated exploit-generation success rate.

What does the ablation let the paper attribute, and how?

It attributes the single largest F1 jump to the LLM refinement step by adding capabilities one at a time and measuring F1, showing extra tools gave diminishing returns while refinement gave the big precision gain.

Why is the GAEV verification rate more robust than the detection accuracy numbers?

Because a verified finding is backed by an executed test, so it does not depend on the ground-truth classifier — the harness is the arbiter.

State the single most important interpretive rule from the failure taxonomy.

Unverified is not false-positive: most unverified findings failed due to harness limitations (multi-transaction / external state), so they may well be real but simply not reproducible here.

Which reported results are proven versus measured?

Proven (under the Detection Separation assumption): recall preservation and no-hallucination; proven as arithmetic: the speedup relations. Measured on one corpus/backend: the ~7x speedup, F1 0.55->0.86, ~54% FP reduction, the ablation attribution, and the GAEV rates and failure percentages.