Evaluation deep dive

Read the paper's evaluation the way a careful reviewer would: understand the simulation setup and measurement protocol, walk the per-archetype detection-rate results epoch by epoch, see exactly how those numbers illustrate (not prove) the no-equilibrium theorem, and account for the real-world costs the simulation deliberately omits — keeping the PROVEN-but-model-relative, MEASURED, and ARGUED labels crisp.

Track

Research

Module

5 · Research case studies

Prerequisites

The evasion arms race · Formal model deep dive

The evaluation does not try to measure how well real AML systems catch real launderers. Its stated job is narrower: to illustrate the qualitative dynamics predicted by the no-stationary-equilibrium theorem — adversary learning drives detection down, defender response pushes it partway back up, and the system keeps oscillating instead of settling. Keeping that purpose in mind is the difference between reading the table correctly and over-quoting it.

The paper reports a preliminary simulation in a controlled testbed (not a live network). It pits the adversary — running the financial domain with the five archetypes — against a compliance system equipped with the velocity-check and amount-range circuits, using parameters described as representative of current testnet deployments of a behavioral-verification compliance system.

Initialization

The adversary's detector model starts uniform over the feature space (it has learned nothing). The defense starts from static baseline threshold parameters typical of a non-adaptive configuration.

Epoch structure

Each epoch, the adversary executes a batch of transactions and observes the per-transaction detection flags to update its model. The defense updates its threshold parameters on a slower cadence, from aggregated statistics.

Measurement protocol

Reported detection rates are averaged over 10 independent runs with different random seeds. The paper states standard deviations are small (typically under 0.03) but does not treat the point values as precise performance claims.

Policy

Both sides operate under the adaptive learning dynamics from the theory section — the adversary via its black-box update, the defense via periodic threshold adjustment — so the run exercises the mutual-memory regime the theorem is about.

Two asymmetric cadences matter for reading the results: the adversary updates every epoch (fast), the defender updates on a slower multi-batch schedule. That asymmetry is itself part of what the oscillation illustrates — the theorem says the oscillation's rhythm is set by the two sides' relative learning rates.

The paper reports detection rates per archetype across five sampled epochs (call them E1, E5, E10, E20, E50). Lower means the archetype evaded more successfully. The reported figures:

the paper reports (detection rate; averaged over 10 runs; illustrative)
archetype    E1     E5     E10    E20    E50
Ghost        0.12   0.08   0.06   0.05   0.09
Runner       0.41   0.28   0.19   0.14   0.21
Patient      0.09   0.07   0.06   0.06   0.08
Opportunist  0.38   0.22   0.16   0.12   0.19
Harmonious   0.21   0.14   0.10   0.08   0.11

E1->E20 : adversary learning phase (rates fall)
E20->E50: defender responds (rates partially recover)

Three patterns the paper draws out, each tied to the theory:

1. 1Adversary learning (E1 -> E20): detection falls for every archetype as the adversary refines its detector model. The Runner falls the most and fastest — the paper reports roughly 0.41 down to 0.14 — because its high-velocity evasion maps most directly onto the velocity-check circuit's feature space, so its feedback signal is the most informative.
2. 2Defender response (E20 -> E50): once the defense updates its thresholds, detection partially recovers rather than staying at its floor — the paper reports the Runner rebounding from about 0.14 toward 0.21. The recovery is partial, not a return to E1 levels.
3. 3No convergence: across every sampled epoch the system oscillates rather than stabilizing — exactly the qualitative signature the no-stationary-equilibrium theorem predicts under mutual memory.

A quieter observation the paper makes: the Ghost and Patient archetypes keep the lowest detection rates throughout, consistent with their high stealth and long patience settings (recall the reported parameters: stealth 1.0/0.8, patience 1.0/1.0). Behavioral mimicry over a long horizon is, by this account, structurally hard to catch without intent inference — which the behavioral-verification defense addresses by checking mathematical envelopes rather than inferring behavior-vs-purpose.

Here is the careful reading, with labels kept crisp:

• ▸PROVEN (within the model): the EXISTENCE of oscillation and the absence of a stationary equilibrium. This comes from the theorem, not the table — the table only visualizes it.
• ▸MEASURED (simulation, illustrative): the specific trajectories and point values above. They are corpus-of-runs observations in a controlled testbed, averaged over 10 seeds, and the paper explicitly declines to treat them as predictions of production detection rates.
• ▸ARGUED (not demonstrated here): that this dynamic transfers to a real deployed compliance system. The paper names full empirical validation against a deployed, adaptive ZK-compliance system as ongoing future work — the simulation is a preliminary illustration, not that validation.

The gap between this simulation and a real deployment is not noise — it is a set of named, structural factors the simplified model omits. Each one would change the arms-race timing, and in a direction that matters.

Cryptographic overhead

Generating and verifying zero-knowledge proofs is not free; in a real behavioral-verification system this latency sits inside every transaction, changing how fast the defender can act and re-tune.

Gas / execution cost

On-chain verification has a monetary cost per operation that the simulation does not charge; it bounds how often circuits can be run and how rich the checks can be.

Oracle staleness

The detector's model is only as fresh as its last update; between updates it is operating on a stale view of the adversary, widening the window the adversary exploits.

Adversarial latency

Real adversaries also face delays (settlement, confirmation, propagation) that the idealized epoch structure abstracts away; these change the effective relative learning rates that set the oscillation.

There is also a modeling assumption worth flagging: the convergence result assumes the target detection function lies within the adversary's model class. If a real detector is richer than the adversary's hypothesis space, learning it is harder than the clean theory suggests — a limit that cuts in the defender's favor and that the paper lists honestly. In short: the simulation is a faithful illustration of the THEOREM'S shape under idealized conditions, and an explicitly incomplete picture of a real deployment's economics.

Expected direction: Problems: (1) it is not an independent real-world study but a preliminary, controlled-testbed simulation by the same authors; (2) 0.14 is one archetype (Runner) at one epoch (E20), the low point before the defender responds, not a stable or representative rate — it recovers toward 0.21; (3) the paper explicitly says the figures are illustrative of dynamics, not predictions of real detection rates; (4) the simulation omits crypto overhead, gas, oracle staleness, and adversarial latency, so it is not a performance benchmark. Honest rewrite: "In the paper's illustrative simulation, the velocity-targeting (Runner) archetype's detection rate fell as the adversary learned and then partially recovered once the defender adapted — demonstrating the predicted oscillation, not a real-world evasion rate."

What is the stated purpose of the simulation?

To illustrate the qualitative dynamics predicted by the no-stationary-equilibrium theorem (learning-driven decline, partial recovery, no convergence) — not to measure real-world detection performance.

Why does the Runner archetype show the largest, fastest detection drop?

Its high-velocity evasion maps most directly onto the velocity-check circuit's feature space, so its binary feedback is the most informative for learning that circuit.

What does the partial recovery between E20 and E50 represent?

The defender updating its threshold parameters in response to the adversary's evasive policy; detection rebounds partway but does not return to initial levels, and oscillation continues.

Name three costs the simulation omits that would affect a real deployment.

Cryptographic proof overhead, on-chain gas/execution cost, oracle staleness, and adversarial latency (any three) — all change the relative learning rates that set the oscillation.

How should the point values in the results table be cited?

As illustrative simulation figures attributed to the paper, quoting the qualitative shape rather than treating the numbers as real-world detection benchmarks.