Threat model, limitations, and a practitioner synthesis

Turn the formalism into engineering judgement: map its stated assumptions to concrete attack surfaces (adversarial sub-holons, non-stationarity, tribunal abuse), lay out the open problems honestly, and distil a practical checklist for anyone composing multi-agent systems with arbitration.

Track

Research

Module

5 · Research case studies

Prerequisites

The formal model in full: six theorems, written out · Evaluation deep-dive: what one feasibility run actually proves

Every theorem is a conditional: IF the hypotheses hold, THEN the conclusion follows. The security of a system built on such theorems lives entirely in whether the hypotheses hold in the wild — and an adversary's whole job is to make them not hold. So the honest way to read this paper's assumptions is as an enumeration of attack surfaces. The paper states four: finite intention sets, a global/stationary event distribution, an abstract interface, and no adversarial sub-holons. We take them one at a time.

The composition and arbitration theorems assume sub-holons that may conflict but are not actively hostile. The paper explicitly leaves adversarial sub-holons to future work. That gap is worth making concrete, because composition hands a sub-holon real leverage over the whole.

Meshing spoofing (against C1 / T1)

A malicious sub-holon advertises intentions and a plan that appear to mesh at composition time, then—once admitted—acts on intentions it withheld. T1 guarantees well-definedness given honest meshing; it says nothing about a sub-holon that lies about its own intention set.

Responsiveness starvation (against C2)

An adversary with influence over the substrate delays or drops the write–read updates that C2 requires, pushing the holon into the exact divergence regime the theorem warns about — without ever violating any rule a naive monitor checks.

Unification poisoning (against C3)

If an attacker can influence identifier unification, two distinct real-world objects can be forced to unify (or one object to split), breaking confluence and making the collective intention order-dependent — a stealthy integrity failure rather than a crash.

Tribunal capture (against T6)

The tribunal decides which intentions survive conflict. An adversary that games the arbitration policy (priorities, votes, learned weights) can get its own intentions retained and rivals' dropped while every formal property (conservativity, consistency, maximality) still appears satisfied on the surface.

synthetic — a meshing-spoofing sequence
t0  attacker sub-holon joins; declares I_adv = { benign_goal }
t1  composition checks C1: benign_goal meshes -> admitted, lifted up
t2  attacker now pursues withheld intention via the shared substrate
    (the lifted intention set never modelled it)
=>  holon-level intention set is well-defined per T1, yet the holon's
    ACTUAL behaviour diverges from its modelled intentions.
mitigation direction: bind declared intentions to verifiable,
    attenuated capabilities; treat a sub-holon's self-report as a
    CLAIM to be checked, not a fact (see delegation / authorization).

The pattern across all four: the theorems protect you against honest-but-conflicting partners, not against partners who misrepresent their intentions or manipulate the substrate. This is the same lesson the security knowledge base teaches about agent collusion and insecure delegation — composition multiplies trust, and trust that is not verified is an opening.

The bounded-reconsideration result (T5) assumes a global, stationary event distribution: fixed class priors and fixed relevance densities, with the concentration bound proved over independent, identically distributed draws. Real environments drift, and an adversary can drift them on purpose.

• ▸Relevance drift: the relevance density of a class changes (routine inputs start carrying real consequences). A policy whose per-class true-positive rate was tuned to the old density now misses relevant events — the coherence guarantee of T5(iii) was conditioned on densities that no longer hold.
• ▸Prior shifting / flooding: an attacker inflates the frequency of a low-relevance class (a denial-of-service against attention). A relevance-informed policy degrades gracefully because it down-weights that class, but the i.i.d. assumption behind the concentration bound is violated, so the measured miss rate no longer tracks the theoretical one.
• ▸Sensor drift: the sensor-soundness proposition underwrites estimating class priors and relevance densities from traces; if the classifier's accuracy degrades under distribution shift, those estimates drift too, and every downstream T5 quantity inherits the error.

synthetic — attention denial-of-service
attacker goal: keep the holon committed to a stale plan
method:        flood a class the policy learned to trust LITTLE
              (low rho_c) with high volume, consuming review budget
              and masking a genuinely relevant event in another class
why it works:  T5's guarantees assume stationary p_c, rho_c and i.i.d.
              draws; a deliberately shifted, bursty stream breaks both.
mitigation direction: monitor per-class rho_c and selectivity in
              production; re-fit the policy on drift; rate-limit and
              quarantine anomalous classes.

The constructive reading: T5 is still useful because it tells you exactly which quantities to monitor — per-class priors, relevance densities, true/false-positive rates. Non-stationarity does not make the theorem wrong; it makes those quantities something you must measure continuously rather than assume once.

Two more assumptions carry engineering consequences. First, the interface is treated abstractly — as a learnable function, without a concrete learning algorithm or its sample-complexity and regret analysis. Second, intention sets are finite.

• ▸Abstract interface: because the emergence result (T3) depends on the interface being learnable, the realism of "a topology jump will occur" depends on a learning procedure the paper does not specify. The capability to jump is proven; whether a given training setup reaches it is an empirical and learning-theoretic question left open. An adversary who can shape the trace memory the interface learns from can, in principle, steer or suppress the jump — a training-data / memory-integrity surface.
• ▸Finite intention sets: the theorems assume a bounded set of intentions. Systems whose agents generate intentions open-endedly (a common LLM-agent failure mode) step outside the proven regime; nothing guarantees the aggregation stays well-defined when the intention set can grow without bound.
• ▸Trace-memory integrity: the interface learns from M_h and the reconsideration policy reads it. Poisoning that memory is the holonic analogue of memory-poisoning and retrieval-manipulation attacks — it corrupts the very signal the "learnable" parts depend on.

Keeping the paper's own honesty, here are the questions it leaves open — useful both as research directions and as caveats when quoting the work.

Empirical validation

The entire six-hypothesis programme (case lesson 5) is unrun; no theorem is yet tested in a deployed system.

Type-5 closure

Whether the middle coupling class is closed under composition is unresolved; the emergence proof relies on non-separability at Type 6, a property Type 5 does not share.

Spectrum of topology jumps

The paper proves jumps EXIST; characterising WHICH jumps are attainable as a function of how diverse the sub-holons are is left open.

Learning theory of the interface

Sample complexity and regret bounds for the learnable interface are future work — the gap between "can learn the jump" and "will, efficiently."

Adversarial and non-stationary extensions

Robustness against hostile sub-holons and drifting distributions is explicitly named as a direction, not a result; integrating robust multi-agent-systems techniques is flagged.

Stronger joint agency

The paper notes that meshing is a relatively thin notion of "togetherness" and points at richer we-mode accounts of collective intention as a way to strengthen the joint-agency claim.

What can a careful engineer take from a paper whose theorems are proven but untested? Quite a lot, provided you use the proven-about-the-model parts as design discipline and treat the rest as hypotheses. A concrete checklist for composing multi-agent systems with arbitration:

1. 1Before composing, check C1–C3 as a design gate: is there a genuine joint plan (meshing), a bounded-latency responsiveness channel, and a confluent identifier-unification rule? If any is absent, expect sequence-dependent, hard-to-reproduce inconsistencies.
2. 2Treat every sub-holon's self-declared intentions as a claim to verify, not a fact. Bind declared intentions to attenuated, verifiable capabilities so a sub-holon cannot act beyond what it advertised — this is where delegation, scope attenuation, and authorization controls do the work the theorems do not.
3. 3If sub-holons can conflict, use a tribunal with the three properties (conservative, consistent, maximal) — but harden the arbitration policy itself against capture, and log which intentions it drops so maximality can be audited rather than assumed.
4. 4Type your event stream and run a class-typed reconsideration policy; then MEASURE per-class priors, relevance densities, and true/false-positive rates in production, because the coherence guarantee is conditional on those staying put. Re-fit on drift.
5. 5Protect the trace memory and the sensor: they are the integrity roots of both the learnable interface and the reconsideration policy. Poisoned memory or a drifting classifier silently invalidates the quantities the guarantees depend on.
6. 6When you report results, keep two columns: proven-about-the-model vs measured-in-your-system. Do not let a throughput number stand in for a correctness or emergence claim — the mistake case lesson 5 is built to prevent.

The deepest practical point: this formalism is most valuable not as a guarantee you inherit but as a vocabulary of failure. It names the exact conditions whose violation causes collective incoherence, the exact quantities whose drift erodes coherence, and the exact places (self-reported intentions, the substrate join, the tribunal, the trace memory) where trust must be verified rather than assumed. Used that way — honestly, with the empirical caveats intact — it is a strong tool for building and for red-teaming multi-agent systems.

• ▸PROVEN (elsewhere, deductive): the six theorems under their stated assumptions (case lesson 4). This lesson adds no theorems.
• ▸ENGINEERING REASONING (not proven): every attack scenario here — meshing spoofing, responsiveness starvation, unification poisoning, tribunal capture, attention denial-of-service, memory poisoning — is disciplined threat modelling grounded in the model's assumptions, not a vulnerability the paper proves or demonstrates.
• ▸STATED BY THE PAPER: the assumptions (finite intentions, stationary distribution, abstract interface, no adversarial sub-holons) and the open problems listed above are the paper's own; the paper names adversarial robustness and non-stationarity as future work.
• ▸OPEN: all empirical questions remain open pending the deferred validation programme.

Expected direction: Example 1 — "no adversarial sub-holons": a malicious sub-holon declares a benign, meshing intention set at composition, is admitted and lifted, then acts on withheld intentions via the substrate (meshing spoofing). Mitigation: bind declared intentions to attenuated, verifiable capabilities and treat self-reports as claims to check (authorization/scope attenuation), so the sub-holon cannot act beyond what it advertised. Example 2 — "stationary event distribution": an attacker floods a low-relevance class to exhaust review budget and mask a genuinely relevant event (attention denial-of-service), breaking the i.i.d./stationarity assumption behind T5. Mitigation: continuously monitor per-class relevance densities and selectivity, rate-limit/quarantine anomalous classes, and re-fit the policy on detected drift. Both mitigations are engineering controls, not theorems.

Why are the paper's assumptions effectively a threat model?

Each theorem is conditional on its hypotheses; an adversary's goal is to make those hypotheses fail. So the assumptions (finite intentions, stationary distribution, abstract interface, non-adversarial sub-holons) enumerate exactly the surfaces an attacker or a drifting environment can exploit.

What does "tribunal capture" exploit, and why is it stealthy?

It games the arbitration policy so the attacker's intentions are retained and rivals' dropped, while the formal properties (conservativity, consistency, maximality) still appear satisfied — a trust/integrity failure that passes surface checks.

Does non-stationarity make T5 wrong?

No — it makes T5's quantities (per-class priors, relevance densities, TPR/FPR) something you must measure continuously rather than assume once; the theorem tells you precisely what to monitor and re-fit under drift.

What is the single most valuable practical use of this formalism?

As a vocabulary of failure: it names the exact conditions whose violation causes collective incoherence, the quantities whose drift erodes it, and the places where trust must be verified (self-reported intentions, substrate join, tribunal, trace memory) — strong for both building and red-teaming.