The Semantic Refinement Pattern — tools detect, AI understands

Understand an architectural pattern that combines many traditional analysis tools with LLM post-processing, where the LLM never performs primary detection — and understand precisely what that separation can and cannot guarantee.

Track

Research

Module

5 · Research case studies

Prerequisites

Basic static analysis concepts · What an LLM agent is

Automated security analysis faces a recurring dilemma. Traditional static-analysis tools (pattern matchers, symbolic executors, fuzzers) have high recall and systematic coverage but produce overwhelming volumes of findings with high false-positive rates and little semantic understanding. Large language models, by contrast, reason about intent and business logic but hallucinate, vary run to run, and miss patterns they were never trained on. The paper under study argues these are complementary paradigms rather than competitors.

Its proposal is an architecture it calls the Semantic Refinement Pattern, summarized by the slogan "tools detect, AI understands". Specialized tools do all the primary detecting; LLM agents run afterwards purely as a post-processing layer that triages, correlates and explains. The paper instantiates this for smart-contract security, but presents the pattern itself as domain-agnostic.

The paper decomposes the pattern into four layers with a strict rule: the LLM layer is last and never originates a finding.

Layer 1 — Pure Dispatcher

An orchestration layer that contains zero domain logic. It launches tools, allocates resources and collects results through one uniform interface, so it could orchestrate an entirely different analysis domain unchanged.

Layer 2 — Parallel Tool Execution

All detection tools run concurrently and in isolation, each with its own dependencies. Failures are contained; tools are unaware of one another.

Layer 3 — Schema Normalization

Per-tool adapters translate each tool's idiosyncratic output into one unified "Finding" record so later stages can reason across tool boundaries.

Layer 4 — Sequential LLM Refinement

A chain of LLM agents that filter false positives, correlate related findings, and enrich them — explicitly forbidden from performing primary detection.

The "Pure Dispatcher" idea is the architectural heart: by forbidding domain logic in the orchestrator, the paper argues you can substitute a tool, or reuse the whole dispatcher for another domain, without touching coordination code — a separation-of-concerns claim, TESTED here only in one domain.

Layer 2 is built on a distributed actor/task runtime (the paper uses Apache Ray). Each tool is dispatched as an independent task; the dispatcher waits for all of them and then flattens and deduplicates the results. Because run times across tools are highly heterogeneous — from roughly a second to several minutes — a task-based runtime fits better than batch frameworks that assume uniform work.

schematic — parallel dispatch (re-expressed, not verbatim)
# one task per tool, launched concurrently
futures = [run_tool_async(tool, target) for tool in tools]

# wait for all, with per-tool timeouts and isolation
results = await_all(futures)        # a failed tool returns [] , not a crash

# flatten + deduplicate by (type, location)
findings = dedupe(flatten(results))

The paper frames the achievable speedup with Amdahl's law (orchestration overhead bounds it) and Gustafson's reframing (speedup scales as you add more tools / larger workloads). The practical takeaway: when detectors are independent, wall-clock time approaches the slowest single tool rather than the sum.

The paper's most interesting claim is a conditional correctness argument. It defines a Detection Separation Property: the refinement step may only remove or re-label findings, never add ones that did not originate from some tool. Under that assumption it argues two things.

• ▸PROVEN (under the stated assumption): recall is preserved up to the triage agent's own false-negative rate. If the LLM layer can only discard, the only way to lose a true positive is for triage to wrongly drop it — a quantity measurable on ground truth, not an unbounded risk.
• ▸PROVEN (under the stated assumption): no hallucinated vulnerability can enter the output, because every surviving finding traces back to at least one tool's concrete code evidence. The LLM can reclassify, correlate or enrich, but cannot originate.

The paper positions the pattern against "LLM-primary" approaches where a model is the primary detector. In those systems the LLM originates findings, so a hallucination becomes a false vulnerability report with no grounding. Here the LLM only refines findings that tools already grounded in concrete code evidence, which is why the no-hallucination argument is even available. This is the architectural line the paper draws between the two families.

The pattern generalizes beyond security to any agent pipeline that mixes deterministic tools with LLM reasoning. The design lessons transfer directly: keep the orchestrator free of domain logic so tools are swappable; run independent tools in parallel and isolate their failures; normalize heterogeneous outputs into one schema before any LLM sees them; and constrain the LLM to post-processing grounded data rather than generating primary facts.

That last constraint is the reusable safety idea: an agent that may only filter, correlate and annotate grounded evidence has a much smaller blast radius than one trusted to originate claims. The guarantee you get is exactly as strong as your ability to enforce "refine only, never originate".

• ▸LIMITATION: the correctness arguments depend entirely on the Detection Separation Property truly holding. A refinement step that quietly synthesizes a new finding would void both guarantees — enforcement is an engineering responsibility, not an automatic property.
• ▸OPEN: the pattern is demonstrated in a single domain (smart contracts). The paper itself labels cross-domain generalization as a design guideline, not a validated result.
• ▸LIMITATION: "no hallucinated detection" is not "no mistakes". The LLM can still mis-triage (drop true positives or keep false ones); the guarantee only bounds where errors can occur, not their rate.

Expected direction: The assumption is the Detection Separation Property: refinement may only remove or re-label findings that originate from a tool, never add new ones. A bug that breaks it: a correlation/enrichment agent that, when combining partial observations, emits a new Finding record not traceable to any tool's evidence (e.g., an LLM "inferred" vulnerability written into the output set). Then a hallucination can enter even though each layer is present as drawn.

In the Semantic Refinement Pattern, what is the LLM layer forbidden from doing?

Performing primary detection — it may only filter, correlate, re-label and enrich findings that tools already produced; it may never originate a finding.

What does the paper mean by a "Pure Dispatcher"?

An orchestration layer with zero domain-specific logic that treats all tools through one uniform interface, so tools can be swapped and the dispatcher reused for other domains.

Is the recall-preservation result an unconditional guarantee?

No. It holds only under the Detection Separation Property, and it bounds recall loss by the triage agent's false-negative rate — it does not claim the system detects everything.

Why does the no-hallucination argument require tools to detect first?

Because every surviving finding must trace to concrete code evidence from a tool; if the LLM originated findings instead, hallucinations would have no grounding and the argument would not hold.