The refinement pipeline — unified schema and sequential LLM agents

Understand how heterogeneous tool outputs are normalized into one schema and refined by a chain of specialized LLM agents, and read the paper's measured accuracy results honestly — what improved, by how much, and under what caveats.

Track

Research

Module

5 · Research case studies

Prerequisites

The Semantic Refinement Pattern (lesson 1)

Each detection tool speaks its own dialect: different severity labels, locations, evidence formats. Before any LLM can reason across tools, the paper normalizes everything into a single record type. This is Layer 3, and it is deliberately boring on purpose — it is what makes cross-tool reasoning possible at all.

schematic — the unified finding record (re-expressed)
Finding:
  id          # unique identifier
  tool        # which detector produced it (provenance)
  type        # standardized vulnerability category
  severity    # CRITICAL / HIGH / MEDIUM / LOW / INFO
  confidence  # 0.0 .. 1.0
  location    # file / contract / function / line range
  description # human-readable explanation
  evidence[]  # code snippet, trace, constraint, ...
  cwe_id?     # optional standardized classification

Beyond individual findings, the paper gives the agents a shared global context (it calls this the World): the parsed targets, all normalized findings, a dependency graph, entry points, state variables and cross-call information. Agents query it (e.g., "all findings affecting this contract", "all findings of this type") instead of each re-deriving context. This shared context is what lets a correlation agent connect observations that no single tool related.

Layer 4 is a pipeline of specialized LLM agents, each with one narrow responsibility, each consuming the enriched output of its predecessor. The paper's instantiation uses six.

1. Triage

Filters false positives and adjusts confidence, judging reachability, input validation and business context. This is the step that drives most of the precision gain.

2. Correlation

Groups findings by shared root cause, reducing redundant noise and surfacing systemic issues.

3. Attack-chain

Combines individual findings into multi-step attack scenarios — compound vulnerabilities that no single tool reports.

4. Exploit-skeleton generation

Produces proof-of-concept test scaffolds for high-severity findings. The paper is explicit that these skeletons need manual refinement and are not fully automated.

5. Remediation

Suggests concrete fixes / code patches following best practices.

6. Summary

Synthesizes everything into an executive summary and a detailed technical report.

Note how each agent only transforms grounded findings: triage removes, correlation groups, attack-chain links, remediation annotates. None originates a vulnerability — the Detection Separation Property from lesson 1 is enforced by giving each agent a filtering/enriching role, not a detecting one.

The paper's most illustrative case: several tools each reported fragments of a flaw. One flagged an external-call pattern at medium confidence; another flagged a missing access check; a property-based tool found nothing because the needed invariant was never specified. Individually, none described an end-to-end exploit.

1. 1Triage promoted the external-call finding from medium to high after reasoning about how accounting values are read.
2. 2Correlation linked that finding to the access-control flag, identifying that unprotected deposits enabled the manipulation.
3. 3Attack-chain assembled the full sequence: a transfer path that diverges from internal accounting, leading to a logic failure under specific conditions.
4. 4The verification step (covered in lesson 3) then produced a test reproducing the chain.

The paper reports that no individual tool detected this vulnerability end-to-end; it emerged from the semantic combination of partial observations. This is the concrete argument for why correlation across a unified schema adds value beyond any single detector.

On its evaluation corpus (a few dozen contracts drawn from a known vulnerable-DeFi benchmark plus real protocols), the paper reports the following. These are MEASURED on that specific corpus with a specific model — read them as reported outcomes, not laws.

• ▸MEASURED: parallel execution achieved roughly a 7x wall-clock speedup over running all tools sequentially.
• ▸MEASURED: adding the LLM refinement layer raised F1 from about 0.55 (union of raw tools) to about 0.86, driven mainly by a ~54% reduction in false positives while recall stayed high.
• ▸MEASURED: an ablation attributes the single largest F1 jump to the LLM refinement step itself (a large precision gain), with additional tools showing diminishing returns beyond the first two.
• ▸MEASURED: a share of validated findings came only from cross-tool correlation — vulnerabilities no single tool reported in isolation.

The design transfers to any multi-agent system that post-processes noisy signals. Normalize heterogeneous inputs into one schema with provenance before reasoning; give agents a shared context object rather than re-deriving state; decompose the hard job into a chain of single-responsibility agents; and keep each agent in a filter/correlate/enrich role so the pipeline never fabricates primary facts. The measured win here — precision from triage — is the general pattern: an LLM is often most valuable as a discriminator over grounded candidates, not as a generator of them.

• ▸LIMITATION: ground truth was built with AI-assisted classification plus a single expert reviewer; the paper discloses shared-LLM bias and experimenter bias, and that inter-rater reliability with an independent evaluator was not measured.
• ▸LIMITATION: results use one LLM backend at low temperature; the paper notes run-to-run variance from LLM non-determinism was not fully characterized, and other models may differ.
• ▸LIMITATION: the benchmark is modest in size relative to some related work, which weakens generalizability claims — the authors say a larger, more diverse corpus would strengthen them.
• ▸OPEN: exploit skeletons are explicitly not fully automated and require manual refinement.

Expected direction: The raw tool union already had high recall but low precision (many false positives). Adding more tools mostly adds more candidate findings — it can raise recall slightly but tends to add noise, so precision does not improve and may fall; F1 stays flat. The refinement layer instead discards false positives, so precision jumps while recall is largely preserved, which is what moves F1. The ablation in the paper shows exactly this: extra tools gave diminishing returns, the LLM triage step gave the big jump.

Why must tool outputs be normalized into one schema before the LLM agents run?

So agents can reason across tool boundaries (correlate, group, chain) and so every finding keeps provenance back to the detector that grounded it.

Which agent is responsible for most of the reported precision gain, and how?

The triage agent, by filtering false positives and adjusting confidence based on reachability, input validation and business context.

What does the paper say produced findings that no single tool reported?

Cross-tool correlation and attack-chain reasoning over the shared context — combining partial observations into one vulnerability.

How should the reported F1 of ~0.86 be interpreted?

As a measured result on one specific corpus with one specific model, subject to disclosed ground-truth and non-determinism caveats — not as a universal benchmark.