Threat model, limits & practitioner synthesis

Draw the precise line between what the calibration layer guarantees and what it does not, work through an attacker-centric threat model of the learned component, and extract concrete, transferable lessons for engineers who want to place a learner before a deterministic verifier.

Track

Research

Module

5 · Research case studies

Prerequisites

Evaluation, in depth

It is worth saying the guarantee exactly, because most mistakes in deploying a system like this come from stretching it. PROVEN (by construction, part 3): if the learned component rewrites only the confidence field, the verifier gate consumes that field, and every downstream component is a deterministic function of already-verified beliefs, then two instances that differ only in how much the component has learned produce identical decisions. Training cannot move the decision boundary.

Enumerating the gaps is the whole point, because each is an attack surface or an operational risk:

• ▸NOT guaranteed — calibration accuracy: the proof does not claim c_cal is a good estimate of trustworthiness, only that whatever it is, it cannot corrupt downstream logic.
• ▸NOT guaranteed — OOD detection: the cross-type escalation property is projected from geometry and unmeasured (part 5). You may not rely on it as a control.
• ▸NOT guaranteed — adversarial containment: the containment rate was measured only on clean, in-distribution data; the ECR = 1 (catch-all) target on adversarial inputs was not demonstrated.
• ▸NOT guaranteed — robustness of the learned weights: the gate and expert memories are shaped by operational feedback and stored state, both of which an attacker may be able to influence.
• ▸NOT guaranteed — anything during cold start: the layer is pass-through until n_min, so it provides no protection for a context's first documents.

Assume the symbolic verifier and the decision core behave as specified (the guarantee covers them). The interesting attack surface is the learned calibration layer and the state it depends on. For each vector below, the key question is: does the structural guarantee help? The answer is almost always no — the guarantee protects decision integrity, not calibration integrity.

Expert-memory poisoning

If an attacker can get benign-looking malicious documents accepted and remembered, they seed the expert memory with neighbors that later vouch for similar malicious inputs — raising c_cal for the attack class. The guarantee does not help: poisoned memory still only changes confidence, but confidence is exactly what drives escalation.

Feedback (reward) poisoning

Because the gate learns from operational and human reward, an attacker who influences reviewer verdicts, or who games the automatic signals (e.g. by shaping which beliefs get escalated), can steer the gate toward trusting the wrong expert.

Cold-start abuse

Attack during the first n_min documents, when the layer is pass-through and adds no protection. Equivalently, force a context reset to re-enter cold start.

Embedding-space evasion

Craft a malicious input whose embedding lands among trusted in-distribution neighbors, so the experts assign high confidence and the input clears the (relaxed) threshold. This directly defeats the projected anomaly property.

Threshold grooming

Feed a long run of benign in-type documents to drive tau down toward its floor, then submit the real payload against the now-loosened oversight threshold.

Two of these — memory poisoning and embedding-space evasion — are the mirror image of the projected benefit. The same k-NN geometry that is supposed to escalate far-away inputs will wave through anything an attacker manages to place near trusted neighbors. A projected defense and a concrete evasion share one mechanism.

None of these threats are exotic; they are instances of failure classes documented elsewhere in the knowledge base, now specialized to a calibration layer:

• ▸Expert-memory and feedback poisoning are forms of memory/retrieval poisoning — the learned state is corrupted by controlled inputs over time.
• ▸Embedding-space evasion is retrieval manipulation at the vector level: controlling nearest-neighbor outcomes by crafting representations.
• ▸Threshold grooming and cold-start abuse are forms of excessive agency — the system is allowed to reduce human oversight based on state an attacker can shape.
• ▸Reward gaming via escalation patterns resembles insecure delegation: the signal that is supposed to encode "a human approved" can be manufactured if the escalation path is attacker-influenced.

Strip away the specifics and the reusable idea is a design pattern: you can embed an evolving, learned component inside a safety-relevant system WITHOUT voiding its guarantees, provided you confine the learner to the perception side of a deterministic verifier. The value is that you get to keep learning in production while the certified decision logic stays frozen and auditable.

the pattern and its preconditions
PATTERN:  learner --(writes ONE signal)--> deterministic verifier --> frozen decision core

HOLDS ONLY IF:
  1. the learner writes a single, well-identified signal (here: the confidence field);
  2. the verifier is the sole consumer of that signal for the decision;
  3. every downstream component is a function of VERIFIED outputs, not of the raw signal;
  4. the audit trail is append-only (learning extends it, never rewrites it);
  5. the verifier's own rules are fixed and not themselves learned.

VIOLATE ANY ONE  ->  the non-interference guarantee is lost.

Most real-world breakage of this pattern comes from quietly violating precondition 2 or 3 — letting some downstream component peek at the raw learned signal "for convenience," at which point learning can once again move the decision boundary. The guarantee is only as strong as the discipline that keeps the learner's output funneled through the verifier.

If you are considering this architecture, concrete steps that follow from the analysis:

1. 1Confine the learner to exactly one signal and prove (by code review, not comment) that no downstream component reads that signal independently of the verifier.
2. 2Keep the verifier deterministic and its parameters out of the learning loop; never let the thing being learned also set its own admission rules.
3. 3Treat calibration as advisory, not protective: do not let projected anomaly behavior substitute for real input-integrity controls (injection, poisoning, retrieval manipulation).
4. 4Protect the learned state: gate what can enter expert memory, authenticate and sample reviewer feedback, and watch for threshold grooming (sudden benign runs that relax oversight before an anomaly).
5. 5Harden the cold-start window explicitly — it is unprotected by design; apply stricter external controls until the context has warmed up.
6. 6Keep the audit trail append-only and log the calibration inputs (which experts, which neighbors, which weights) so an escalation decision can be explained after the fact.
7. 7Measure what the paper left open before trusting it: a pass-through baseline, adversarial/OOD containment, and sensitivity to k, alpha and the threshold schedule.

The authors name several; the threat model adds a few more. Honestly stated as open:

• ▸Heterogeneous-corpus validation: does cross-type anomaly escalation actually occur, and at what false-escalation cost, when multiple real types are present?
• ▸Adversarial containment: measure the containment rate on genuinely adversarial/OOD beliefs, not just clean data.
• ▸Poisoning resistance: quantify how many crafted documents or feedback events it takes to shift a context's calibration, and design defenses.
• ▸Stronger learners: the authors plan per-type trained classifiers in place of k-NN and a policy-gradient update in place of the bandit step for temporal strategy — both unevaluated.
• ▸Calibration quality: establish that c_cal correlates with real trustworthiness, which the non-interference proof deliberately does not address.

Expected direction: The guarantee depends on precondition 3: every downstream component must be a function of verified outputs, not of the raw learned signal. If the planner reads the raw calibrated confidence directly, a downstream component now depends on the learner's output outside the verifier. Even if the intent is only telemetry, any branch, filter, or timing that depends on that value means two differently-trained instances can diverge downstream — so training can again move behavior, and the non-interference proof no longer applies. The fix is to log the signal out-of-band without letting any decision-relevant component consume it.

What does the structural guarantee NOT protect against?

Calibration accuracy, out-of-distribution detection (projected only), adversarial containment, and poisoning of the learned state. It guarantees only that learning cannot change the deterministic decisions.

Why are expert-memory poisoning and the projected anomaly defense "two sides of one mechanism"?

Both rely on k-NN geometry: the layer escalates inputs far from trusted neighbors, so an attacker who places a malicious input near trusted neighbors (by poisoning memory or crafting embeddings) is waved through by the same mechanism meant to catch anomalies.

Which precondition is most often violated in practice, and with what effect?

That no downstream component reads the raw learned signal independently of the verifier. Letting any component peek at the raw confidence re-couples learning to decisions, voiding the non-interference guarantee.

Why is the cold-start window a security concern?

During the first n_min documents the layer is pass-through and provides no calibration or anomaly protection, so it is an unprotected window that external controls must cover.