Guarantees, results and honest limits

Separate what this proposal proves from what it merely measures or projects: understand the "learn before the boundary" argument that preserves the deterministic agent's invariants, and read the empirical and projected results with the right level of confidence.

Track

Research

Module

5 · Research case studies

Prerequisites

Learning to calibrate: experts + human feedback

The hardest question about putting a learning component inside a safety-relevant agent is: does the learning void the guarantees? A regulator does not want "the model got better at calibration" to also mean "the auditable decision logic changed underneath us." This lesson is about the argument the authors use to answer "no," and about reading their evidence honestly.

The host architecture is characterized by a set of structural invariants that make it auditable. Paraphrased, they are:

Determinism

Given fixed model state, the agent's output is deterministic.

Single epistemic entry

The LLM sensor is the only point where non-determinism enters; there is one perception boundary.

Downstream determinism

Every component after the verifier is a deterministic function of already-verified beliefs.

Pre-execution verification

Plans are checked before anything is executed.

Complete audit trail

The full chain — embedding, calibrated confidence, verifier outcome, reward — is recorded.

The calibrator must not break any of these for the architecture to keep its regulatory story.

The proof is by construction rather than by a deep theorem. The calibrator modifies exactly one field — confidence — of a candidate belief. The verifier gate consumes confidence to decide admit/contest/reject, but the set of beliefs that pass the gate is what every later component sees, and those components do not read the raw confidence again. Therefore two runs that differ only in how much the calibrator has learned will present the verifier with the same candidate beliefs (same extracted values) and, for any belief the verifier admits, produce identical downstream inputs. The planner and audit machinery cannot tell the two runs apart.

why training cannot change decisions (sketch)
calibrator changes:  confidence field only
verifier:            uses confidence -> admit / contest / reject
downstream (planner, audit): functions of ADMITTED beliefs, not of raw confidence

=> for any admitted belief, downstream input is independent of how much
   the calibrator has trained  => decision guarantees preserved

Status of this claim: PROVEN, but in the specific sense of "proven by the construction / placement," conditional on the architecture actually enforcing that the learned component touches only the confidence field and that the field is consumed solely by the verifier. It is a structural guarantee about information flow, not an empirical result and not a statement that the calibrated confidences are themselves correct.

A natural hope is that per-type experts also catch out-of-distribution inputs: if a document of an unseen type arrives, it will have low similarity to every expert's neighbors, dragging the calibrated confidence below the escalation threshold even when the raw LLM sensor is confident. The paper describes this cross-type anomaly behavior — but is careful about its status.

the projected property (NOT empirically validated here)
unseen-type document:
  low cosine similarity to every expert's neighbors
  -> weighted expert confidence << raw sensor confidence
  -> calibrated conf < escalation threshold
  -> escalate to human, even if raw sensor said "confident"

Honest status: this is PROJECTED from the k-NN geometry on a homogeneous corpus, not demonstrated on a heterogeneous one. The evaluated corpus contained a single document type, so the cross-type escalation story is a reasoned expectation about how the mechanism should behave, explicitly flagged by the authors as reserved for future work. Treat it as a hypothesis, not a result.

The transferable engineering principle is strong even if this particular system is early: if you must put a learner inside an auditable agent, confine it to a pre-verifier signal and make the verifier deterministic, so you can argue — not just hope — that training never moves your decision boundary. This is a cleaner safety story than retraining the decision model and re-certifying it each time.

For security specifically: calibration-driven escalation can be a useful second line against out-of-distribution or manipulated inputs, but only in-distribution integrity was measured here, and the OOD-escalation property is unproven. Do not retire input-integrity defenses (injection, poisoning, retrieval manipulation) on the strength of a projected property.

• ▸PROVEN (by construction): placing the learner strictly before the verifier and letting it touch only the confidence field preserves the architecture's invariants regardless of training progress — a structural information-flow guarantee, conditional on that placement being enforced.
• ▸MEASURED (one small homogeneous single-session corpus): embeddings stored and verified at the expected dimensionality; the expected mode transition and threshold relaxation occurred; no in-distribution false negatives were introduced by calibration (reported containment on valid beliefs).
• ▸PROJECTED (not tested): cross-type / out-of-distribution anomaly escalation, inferred from k-NN geometry on a single-type corpus.
• ▸LIMITATIONS: single session; one document type; small corpus; k-NN experts and a bandit update (stronger classifiers and policy-gradient learning are named as future work); operational-feedback reward with no labeled ground truth.
• ▸OPEN: validation on heterogeneous corpora; adversarial / out-of-distribution evaluation of the containment rate; temporal strategy learning.

Expected direction: First overstatement: "proved … detects out-of-distribution documents" — cross-type/OOD detection is PROJECTED from k-NN geometry on a single-type corpus, not demonstrated. Second overstatement: "so we can reduce human review" — only in-distribution containment was measured on one small corpus; adversarial/OOD evaluation is open, so reducing oversight is unjustified. What IS established is the structural guarantee that learning does not alter the deterministic decision logic — which says nothing about whether reducing review is safe.

What exactly is proven, and in what sense?

That the learned calibrator cannot change the agent's decision guarantees — proven by construction, because it only rewrites confidence and all downstream components are deterministic functions of already-verified beliefs. It is a structural information-flow guarantee, conditional on that placement being enforced.

Is cross-type anomaly detection a measured result?

No. It is projected from the nearest-neighbor geometry on a single-type corpus and explicitly left to future work; treat it as a hypothesis.

What does the proven guarantee NOT tell you?

It does not say the calibration is accurate, that escalation decisions are good, or that the system is robust to malicious inputs — only that training cannot secretly move the deterministic decision boundary.