Evaluation deep-dive: what one feasibility run actually proves

Read the paper's empirical section with discipline: understand exactly what the preliminary feasibility deployment measured, why it supports a claim about the runtime and not about the theorems, and how the planned (but not-yet-run) validation programme is designed to test the six theorems.

Track

Research

Module

5 · Research case studies

Prerequisites

The formal model in full: six theorems, written out · Deterministic vs probabilistic agents

Here is the mistake to avoid. A reader sees "six theorems" and "a working deployment with real throughput" in the same paper and merges them into "the composition and emergence results are empirically backed." They are not. The theorems are proven deductively (case lesson 4); the deployment measures something much narrower. The paper itself is careful about this, and a good engineer reading it should be at least as careful.

The empirical content is labelled by the paper as preliminary feasibility evidence — the wording matters, and we keep it. The paper reports a reference deployment that instantiates a known agent-correctness pattern and runs two independent holons of distinct architectural classes at the same time: one reactive holon, and one of the learnable Topology-2-prime class from the earlier lessons. Both process long-form, multi-page documents end to end, and crucially they do so without sharing any intention state.

what the paper reports (feasibility run)
setup (as reported):
  - 2 holons, DISTINCT architectural classes, running CONCURRENTLY
      holon A: reactive
      holon B: learnable Topology-2-prime
  - both process long-form multi-page documents END TO END
  - NO shared intention state between them
  - consumer-grade hardware (generic; multi-core CPU + commodity GPUs)

reported observation:
  - sustained throughput on the ORDER OF tens of multi-page
    documents per minute, across both holons, running asynchronously
  - NO observed substrate-level contention

Note the hedged language, kept faithfully: "on the order of tens … per minute," not a precise benchmarked figure; "no observed contention," not a proof of contention-freedom. The paper reports an operational observation on one configuration. We keep the hardware description generic on purpose — the point is "commodity hardware, not a supercomputer," and no more detail is needed to understand the claim.

Draw the line precisely. The feasibility run supports one claim and refutes a different, tempting one.

• ▸SUPPORTED (the paper reports): the runtime/substrate can host heterogeneous holons of different architectural classes concurrently, at a production-relevant throughput, without substrate-level contention. In short: the plumbing carries the load.
• ▸NOT SUPPORTED, and the paper says so: the composition theorem (T1), the emergence/topology-jump theorem (T3), and the other structural results are NOT tested by this run. Running two holons side by side with no shared intention state is, if anything, the opposite of testing composition — composition is about what happens when holons DO share and aggregate intentions.
• ▸NOT A BENCHMARK OF QUALITY: throughput says nothing about whether the holons' decisions were correct, whether a topology jump occurred, or whether arbitration removed the minimal set of intentions. Those are precisely the quantities the deferred programme is meant to measure.

the two columns, kept apart
PROVEN-DEDUCTIVELY (case lesson 4)   |  MEASURED-AS-FEASIBILITY (this run)
-------------------------------------|-----------------------------------
T1 composition                       |  runtime hosts 2 holons at once
T2 hierarchical persistence          |  throughput ~ tens of docs/min
T3 emergence / topology jump         |  no observed substrate contention
T4 Topology-6 closure                |
T5 bounded reconsideration           |  (says NOTHING about any theorem)
T6 arbitrated composition            |

Why does running two holons concurrently NOT test composition? Because the theorems are about the intention-level structure that appears when sub-holons are aggregated under the composition operator — meshing, lifted interfaces, tribunals, cross-modal conditioning. A deployment that deliberately keeps intention state unshared exercises the substrate's capacity to carry independent workloads. That is a real and necessary engineering result (you cannot test composition on a substrate that collapses under two tenants), but it is a precondition, not the test itself.

The paper does not pretend the theorems are self-validating. It lays out an empirical programme that operationalises each theorem as a falsifiable hypothesis, and states plainly that it has not been run — it requires substantial compute the authors did not have, in particular for the multi-holon scaling and the cost-coherence sweeps. We paraphrase the hypotheses; each is a test you could in principle design.

H1 (tests T1)

Holons whose substrate violates the mutual-responsiveness condition C2 should show detectable plan divergence under controlled event reorderings, at a rate that scales with how badly C2 is violated.

H2 (tests T2)

Lifted intentions should persist across a hierarchy when the parent's reconsideration policy is calibrated, with abandonment latency predictable from that policy.

H3 (tests T3)

Composites of individually Topology-2-prime sub-holons should show measurable cross-modal conditioning consistent with Type 5, detectable by intervening on the substrate — this is the direct test that a topology jump actually occurs in a running system.

H4 (tests T4)

A Type-6 holon composed with varied sub-holons should yield composites that resist symbolic-core extraction across all trials, which would falsify any "topology decreases under composition" hypothesis.

H5 (tests T5)

The measured cost–coherence trade-off of a relevance-informed policy should track the concentration bound within a constant factor, and the gap to a relevance-blind baseline (matched on marginal selectivity) should scale as the informed–blind separation predicts.

H6 (tests T6)

Arbitrated compositions of non-meshing sub-holons should show holon-level consistency, with the number of intentions removed by arbitration matching the minimum needed — the maximality property — measured against an oracle.

Status of the whole programme: PROJECTED / DEFERRED. These are experiment designs, not results. The paper is explicit that access to large-scale computing is anticipated but not yet in hand, and that full empirical results would appear in a separate companion paper. We keep the specific facility and funding details out — the only load-bearing fact is "the validation needs a lot of compute the paper did not yet have, so it has not happened."

Generalise the discipline beyond this paper; you will meet the pattern often in agent research.

1. 1Separate the claims by TYPE before judging them: deductive results (checked by reading the proof) vs empirical results (checked by scrutinising the measurement).
2. 2For each empirical number, ask what it is a measurement OF. Throughput measures the runtime; it does not measure correctness, emergence, or coherence.
3. 3Ask what the measurement would look like if the interesting claim were FALSE. If a topology jump failing would not change the throughput number, then throughput is not evidence about topology jumps.
4. 4Honour the authors' own hedges. "On the order of," "no observed," "preliminary," "deferred" are not weasel words here — they are the correct calibration, and dropping them would overstate the paper.
5. 5Treat a well-designed but unrun validation programme as a promissory note, not a payment. It raises your confidence that the authors know what would settle the question; it does not settle it.

The practitioner takeaway: you can adopt the formalism as a design discipline today (the C1–C3 checklist, the tribunal properties, the typed-event policy) because those are proven about the model and useful regardless of empirics. What you cannot yet do is cite this paper as evidence that holonic composition or emergence has been demonstrated in a deployed system — because, by the paper's own account, it has not been.

• ▸MEASURED (feasibility only): concurrent operation of two heterogeneous holons at a throughput on the order of tens of multi-page documents per minute on commodity hardware, with no observed substrate contention.
• ▸SCOPE of that measurement: one deployment, one configuration, hedged language; it establishes runtime capacity for heterogeneous-holon coexistence, nothing about decision quality or any theorem.
• ▸DEFERRED: the six-hypothesis validation programme (H1–H6) that would actually test the theorems; not run, pending large-scale compute.
• ▸PROVEN elsewhere, unchanged: the six theorems remain deductive results (case lesson 4); this lesson does not add or subtract from their status.
• ▸OPEN: every empirical question the theorems raise — does a topology jump occur in practice, does arbitration hit the maximality bound, does the cost–coherence curve match the concentration bound — remains open until the programme runs.

Expected direction: T1 and T3 are about the intention-level structure that arises when sub-holons are aggregated under the composition operator — meshing, lifted interfaces, cross-modal joint conditioning. With no shared intention state the holons are not composed in that sense; the run only exercises the substrate's capacity to carry two independent workloads (a precondition, not the test). To begin testing T3 you would compose sub-holons so their interfaces can condition jointly, feed cross-modal inputs that mix both sub-holons' modalities, and intervene on the substrate to check whether the composite conditions on the joint neural state (Type-5 behaviour) when neither sub-holon does alone — i.e. run hypothesis H3.

What single claim does the feasibility run actually support?

That the runtime/substrate can host heterogeneous holons of different architectural classes concurrently at a production-relevant throughput with no observed substrate contention — the plumbing carries the load.

Why is throughput not evidence for the emergence theorem?

A topology jump failing to occur would not change the documents-per-minute figure, so throughput is not a measurement of emergence at all; it measures runtime capacity, not decision structure.

What is the status of the H1–H6 validation programme?

Projected/deferred: these are experiment designs mapping each theorem to a falsifiable hypothesis, not results. The paper states they are not yet run, pending large-scale compute.

How should you treat a well-designed but unrun validation programme?

As a promissory note, not a payment — it shows the authors know what evidence would settle the question, but it does not settle it.