Generative exploit verification (GAEV) — from finding to proof-of-concept

Understand how a dual-LLM component tries to turn high-severity findings into executable proof-of-concept exploits, read its reported success and failure rates honestly, and extract the general lesson about verifying agent-generated claims.

Track

Research

Module

5 · Research case studies

Prerequisites

The refinement pipeline (lesson 2)

A flagged finding is a hypothesis. A verified finding is a demonstration. The paper adds a verification stage whose job is to answer "can this actually be exploited, in a runnable test?" — converting high-severity findings into executable proofs-of-concept where possible. This is the "verify" phase that sits on top of the detect-and-refine pipeline from lessons 1 and 2.

GAEV is described as a dual-LLM system: for a given finding it generates candidate exploit code with more than one model in parallel, optionally refines each candidate against the finding's context, and then runs the candidate in a test harness. A finding is marked VERIFIED only if some candidate actually executes the attack; otherwise it is returned as UNVERIFIED with a report.

schematic — verification loop (re-expressed, not verbatim)
for finding in high_severity_findings:
    skeleton   = generate_skeleton(finding)
    candidates = [model.refine(skeleton, finding.context)
                  for model in (model_A, model_B)]   # dual-LLM
    for exploit in candidates:
        result = run_in_test_harness(exploit)
        if result.success:
            mark(finding, VERIFIED, exploit)
            break
    else:
        mark(finding, UNVERIFIED)

Two design choices stand out. Using more than one model reduces the chance that a single model's blind spot blocks a true exploit. And the accept/reject decision is made by execution, not by an LLM's say-so — the harness, not the model, is the judge. That keeps the component honest: the LLM proposes, the test disposes.

On the paper's corpus, GAEV targeted Critical and High severity findings. The reported numbers, MEASURED on that specific set with specific models:

• ▸MEASURED: roughly 68% of attempted Critical+High findings were automatically verified with a runnable exploit (about 39 of 57).
• ▸MEASURED: roughly 64% overall when a sample of Medium findings is included (about 84 of 132).
• ▸MEASURED: average generation time on the order of seconds per exploit, with a small average number of refinement iterations.

The paper categorizes why verification failed on the unverified Critical+High findings, and this is arguably its most useful contribution for practitioners — it names where automated exploit generation hits a wall.

Multi-transaction complexity (~44%)

The exploit needs a sequence of transactions spanning more than a single test scope (e.g., timelocks, multi-block ordering). A single-shot test cannot express it.

External state dependency (~33%)

The exploit depends on external protocol state not available in a local fork (oracle prices, liquidity states from other protocols).

LLM modeling errors (~22%)

The model simply mis-modeled contract logic — wrong assumptions about storage layout or inheritance. The paper calls this the current irreducible limitation of LLM-based code generation.

The paper reads this honestly: the first two failure modes are addressable with better harnesses (multi-step orchestration, mainnet fork snapshots), while the third is a genuine ceiling on today's models. So "unverified" is not the same as "false positive" — it often means "real but not reproducible in this harness".

The general principle outlives smart contracts: when an agent produces a claim that can be executed, verify it by execution rather than trusting the generation. Generate candidates with more than one model to cover individual blind spots; make an external, deterministic harness the arbiter; and record not just passes but structured failure reasons, because the failure taxonomy tells you where to invest next.

This is the strongest answer to LLM hallucination in an agent loop: a fabricated exploit simply fails its test and is discarded. The verifier converts "the model is confident" into "the artifact demonstrably works", which is the only confidence a security reviewer should act on.

Across the three lessons the architecture is a detect-then-refine-then-verify pipeline: tools detect (lesson 1), a unified schema and agent chain refine and correlate (lesson 2), and GAEV verifies high-severity findings by execution (this lesson). The paper is careful to present cross-domain generalization of the overall pattern as a design guideline rather than a validated result — an honest boundary worth keeping when reusing these ideas elsewhere.

• ▸LIMITATION: verification is partial by construction — a large share of exploits still need manual adjustment, and multi-transaction / external-state cases often cannot be auto-reproduced.
• ▸LIMITATION: the reported rates depend on the specific models, harness and corpus; the paper does not claim they transfer.
• ▸LIMITATION: exploit generation has a real per-finding compute and API cost, and depends on an external model service being available.
• ▸OPEN: the irreducible LLM-modeling-error share marks a current ceiling; improving it depends on model capability, not architecture alone.

Expected direction: Unverified is not false-positive. The paper's failure analysis shows most unverified Critical+High cases failed for reasons unrelated to being false: multi-transaction complexity beyond a single test (~44%) and external state dependency not available in a local fork (~33%) are harness limitations, not evidence the finding was wrong. Only the LLM-modeling-error share (~22%) reflects the generator getting the logic wrong, and even that is a generation failure, not proof the vulnerability is absent. So many unverified findings are likely real but not reproducible in the current harness.

What makes the accept/reject decision in GAEV, and why does that matter?

An external test harness that executes the candidate exploit — not an LLM's judgement — so a hallucinated exploit simply fails and is discarded.

Why does GAEV generate exploit candidates with more than one model?

To reduce the chance that a single model's blind spot blocks a genuinely exploitable finding.

Which GAEV failure mode does the paper call the current irreducible limitation?

LLM modeling errors — the model mis-modeling contract logic (e.g., wrong storage-layout or inheritance assumptions).

Does an "unverified" finding mean it was a false positive?

No. It often means the exploit needs multiple transactions or external protocol state the local harness cannot reproduce — the finding may still be real.