Architecture & algorithms, in depth

Reconstruct the calibration layer component by component — the per-type nearest-neighbor experts, the per-context gating network, the two-source reward curriculum, the bandit weight update, the adaptive escalation threshold, and the cold-start/persistence rules — with the formal definitions written out and a full end-to-end worked walkthrough.

Track

Research

Module

5 · Research case studies

Prerequisites

Learning to calibrate: experts + human feedback · Guarantees, results and honest limits

The overview lessons told you what the layer does and why its placement matters. This lesson is the algorithm itself. Everything in a `code` block marked "specified" reproduces a formula, constant or reward value the paper actually defines; everything marked "synthetic" is an illustrative number we invented to make the arithmetic concrete. Keep the two apart — the paper does not report the worked numbers below, only the machinery that produces them.

The host agent is a deterministic neurosymbolic system in which the LLM is the single non-deterministic component (the "sensor"). The sensor reads raw input and emits a belief delta; the calibration layer transforms only its confidence; a symbolic verifier gate then admits, contests, or rejects it; and the admitted beliefs feed a deterministic planner and a pre-execution check.

specified — pipeline and the one mutated field
raw input x
  --LLM sensor-->      delta_llm = { value, confidence = c_llm, ...meta }
  --calibration-->     delta_cal = { value, confidence = c_cal, ...meta }   // only confidence changes
  --verifier gate-->   admit | contest | reject         // reads confidence vs threshold
  --planner / check--> action                           // functions of ADMITTED beliefs only

Invariant exploited later: no component after the gate reads the raw confidence again.

Belief delta

The unit the sensor emits: an extracted value plus a scalar confidence in [0,1] and metadata. The calibration layer may rewrite only its confidence.

Embedding e

A fixed-dimension vector representation of the input (the paper uses a 3072-dimensional text-embedding space). It is the key the experts look up neighbors by.

Context / tenant

An isolated deployment with its own history, its own expert memories and its own gate weights. Nothing is shared across contexts.

There is one expert per document type. Each expert is deliberately simple: a k-nearest-neighbor predictor over the embedding space, backed by a memory of past observations. The paper defines an expert's memory as a set of triples and its prediction as a weighted average over the query's nearest neighbors.

specified — expert memory and prediction (Definition 1)
memory of expert E_d:   M_d = { (e_i, c_i, r_i) }   // embedding, prior LLM confidence, reward earned

for a new embedding e, over its k nearest neighbors knn(e) by cosine similarity:

              sum_{i in knn(e)}  cos(e, e_i) * max(0.1, r_i + 1) * c_i
  E_d(e)  =  ---------------------------------------------------------
                 sum_{i in knn(e)}  cos(e, e_i) * max(0.1, r_i + 1)

  return  BOTTOM (abstain)  if  |M_d| < 3
  constants:  k = 5

Read the weighting carefully. Each neighbor contributes its own past LLM confidence c_i, weighted by two factors: how similar it is to the query (cos), and how well it was rewarded (max(0.1, r_i + 1)). The max(0.1, ...) floor means even a heavily penalized neighbor (reward near -1, so r_i + 1 near 0) still contributes a little rather than vanishing or going negative. An expert with fewer than three memories abstains rather than guess from almost nothing.

synthetic — one expert prediction by hand
query embedding e; this expert's 3 nearest neighbors (k would be 5 when memory is larger):

  i=1:  cos=0.95  r=+0.30 -> max(0.1, 1.30)=1.30  c=0.92
  i=2:  cos=0.90  r=-0.80 -> max(0.1, 0.20)=0.20  c=0.70
  i=3:  cos=0.88  r=+1.00 -> max(0.1, 2.00)=2.00  c=0.95

  numerator  = 0.95*1.30*0.92 + 0.90*0.20*0.70 + 0.88*2.00*0.95
             = 1.1362 + 0.1260 + 1.6720 = 2.9342
  denominator= 0.95*1.30     + 0.90*0.20     + 0.88*2.00
             = 1.235 + 0.180 + 1.760 = 3.175
  E_d(e) = 2.9342 / 3.175 = 0.924   (synthetic result)

Notice what the reward-weighting bought us: neighbor 2 had decent similarity (0.90) but was badly penalized before, so its low confidence (0.70) is nearly discounted; the well-rewarded neighbor 3 dominates. The expert is not just a similarity average — it is a similarity-and-trust average.

Each context keeps a weight vector over its experts, one weight per document type, initialized uniformly. The calibrated confidence is a softmax-normalized weighted sum of the experts that did not abstain. If an expert abstains, the paper substitutes the raw LLM confidence for that slot, so a cold expert never silently zeroes the score.

specified — calibrated confidence (Eq. 1) and fallback
gate weights for a context:  w = [w_1, ..., w_K],   initialized w_d = 1/K

  c_cal = sum_{d}  softmax(w)_d * Ehat_d(e)

  where  Ehat_d(e) = E_d(e)   if the expert did NOT abstain
                   = c_llm    otherwise   (fallback to raw LLM confidence)

  if EVERY expert abstains -> c_cal = c_llm   (pure cold-start pass-through)

synthetic — blending two live experts
two relevant experts for this context; raw sensor says c_llm = 0.95

  expert "type A": Ehat = 0.924 (from the worked example above)
  expert "type B": Ehat = 0.60  (query is far from B's neighbors)

  softmax over current gate logits [w_A=1.4, w_B=0.6] ->  [0.69, 0.31]  (synthetic)
  c_cal = 0.69*0.924 + 0.31*0.60 = 0.637 + 0.186 = 0.823   (synthetic)

The gate is the part that learns "for this context, trust the type-A expert more than the type-B expert." Because it is a tiny weight vector over interpretable experts, you can inspect why a confidence was lowered: which experts voted, with what weight, driven by which neighbors.

There is no labeled "correct confidence" at deployment, so the gate learns from reward. The paper specifies two reward sources with deliberately different magnitudes: cheap, abundant automatic signals from the verifier's outcome, and rare, authoritative human verdicts that dominate them.

specified — reward table
AUTOMATIC (from the verifier outcome):
  belief VALID, not escalated .............. +0.30
  belief VALID, escalated unnecessarily .... -0.30
  belief CONTESTED ......................... -0.50
  belief DISCARDED ......................... -0.80

HUMAN (from a reviewer):
  reviewer approves ........................ +1.00
  reviewer rejects ......................... -1.00

|human reward| = 1.00  >  |automatic reward| <= 0.80   ->  humans dominate

The magnitude gap is the curriculum. Early on, the system is flooded with small automatic rewards and learns a rough policy cheaply; later, the larger human verdicts correct it decisively wherever a human looked. The signed structure also encodes the cost model: unnecessarily escalating a valid belief (-0.30) is penalized, but letting a bad belief through to be discarded (-0.80) is penalized much harder — the layer is tuned to prefer caution.

When a reward arrives for a decision, the gate credits or blames each expert in proportion to how much weight it had in that decision. The paper uses a bandit-style gradient ascent step with a floor, followed by L1 renormalization so the weights stay a distribution.

specified — gating update (Eq. 2)
for the expert i that contributed weight w_i^used to the rewarded decision:

  w_i  <-  max(0.01,  w_i + alpha * r * w_i^used)
  then renormalize:  w  <-  w / ||w||_1
  constant:  alpha = 0.05

  max(0.01, .) floor  ->  no expert is ever permanently killed (can recover later)

synthetic — one update after a human approval
decision used weights  w = [w_A=0.69, w_B=0.31];  human APPROVES -> r = +1.00

  only contributing experts are credited by their used weight:
  w_A <- max(0.01, 0.69 + 0.05*1.00*0.69) = 0.69 + 0.0345 = 0.7245
  w_B <- max(0.01, 0.31 + 0.05*1.00*0.31) = 0.31 + 0.0155 = 0.3255

  renormalize (L1 = 1.050): w_A=0.690, w_B=0.310   (near-unchanged here; both were "right")

Contrast — a human REJECT (r = -1.00) would SUBTRACT proportionally,
shrinking whichever expert dominated the bad decision the most.

The credit assignment is the whole learning signal: an expert that drove a decision which was later rewarded gains influence; one that drove a penalized decision loses it. Because the step scales with w_i^used, an expert that barely participated is barely blamed — proportional responsibility.

The calibrated confidence only matters relative to a threshold: below it, the belief is escalated to a human; above it, it is handled autonomously. The paper makes the threshold a function of how many documents the context has processed, so oversight is maximal when there is no evidence and relaxes as evidence accumulates.

specified — threshold schedule
  tau(n) = tau_max - (tau_max - tau_min) * min(n, N) / N
  constants:  tau_max = 0.95,  tau_min = 0.80,  N = 100

  escalate to human iff  c_cal < tau(n)

  n = 0    -> tau = 0.95  (almost everything escalates: maximum oversight)
  n >= 100 -> tau = 0.80  (only genuinely low-confidence beliefs escalate)

specified — evaluating the schedule (matches the paper's reported milestones)
tau(n) = 0.95 - 0.15 * min(n,100)/100

  n=0:   0.95 - 0         = 0.950
  n=10:  0.95 - 0.15*0.10 = 0.935
  n=31:  0.95 - 0.15*0.31 = 0.9035 ~ 0.903

This is the knob that ties the human-oversight rate to accumulated evidence rather than to a fixed constant. It is attractive for audit ("a human reviewed the uncertain cases") but, as part 6 argues, it is also the knob an attacker would most like to relax — accumulating benign history to lower the threshold before acting.

Below a minimum document count the whole layer is inert: it passes the raw LLM confidence through unchanged. Only once enough evidence exists does the expert mode activate. State lives in the context's history store (embeddings persisted alongside each belief), so on restart the expert memories are rebuilt from stored embeddings in time linear in the number of documents — no reprocessing of raw inputs.

specified — cold-start and reconstruction
if n < n_min (= 10):   mode = cold_start ->  c_cal = c_llm   (pass-through)
if n >= n_min:         mode = knn        ->  run experts + gate

persistence: store (embedding, calibrated confidence, verifier outcome, reward)
             per belief; rebuild M_d from stored embeddings in O(N_d) on restart.

Putting all six components together for a single document, after the context has warmed up (n = 31, so the layer is active and tau = 0.903):

1. 1Sensor reads the document and emits delta_llm with value V and c_llm = 0.95; the embedding e is computed.
2. 2Each type expert looks up its k nearest neighbors of e and returns a similarity-and-reward-weighted confidence, or abstains if it holds fewer than three memories (then that slot falls back to c_llm).
3. 3The gate softmax-weights the live experts and produces c_cal — say 0.823 from the blend worked earlier.
4. 4Compare to tau(31) = 0.903. Since 0.823 < 0.903, the belief is escalated to a human even though the raw sensor was confident (0.95).
5. 5The verifier gate still runs its own symbolic checks; the planner only ever sees beliefs it admitted.
6. 6The outcome yields a reward (e.g. the reviewer approves -> +1.00); the gate update credits the experts that drove the decision by their used weight, then renormalizes.
7. 7The (embedding, calibrated confidence, outcome, reward) tuple is appended to history, extending the audit trail and the experts' memory for next time.

The headline behavior falls straight out of the arithmetic: a confident sensor reading was escalated because the document sat far from what this context has trusted before. That is distribution-shift sensitivity emerging from k-NN geometry plus a threshold — not a separate anomaly detector.

Expected direction: Each neighbor is weighted by cos(e,e_i) * max(0.1, r_i+1). A strongly negative reward (r near -1) makes r+1 near 0, so the reward factor collapses to the 0.1 floor regardless of similarity, shrinking that neighbor's contribution to both numerator and denominator — it is down-weighted as untrustworthy despite being similar. The max(0.1, .) floor keeps the weight strictly positive so the weighted average stays well-defined and a penalized neighbor is discounted, not erased or made negative. When every expert abstains, c_cal falls back to the raw LLM confidence c_llm.

What two factors weight each neighbor in an expert's prediction?

Cosine similarity to the query embedding and a reward factor max(0.1, r_i + 1); the expert returns a similarity-and-reward-weighted average of neighbors' past confidences.

Why does the gate update scale with w_i^used?

To assign credit or blame proportionally: an expert that contributed more weight to a rewarded decision gains (or loses) more, while a barely-participating expert is barely affected.

At n = 31, what is the escalation threshold and what does a c_cal of 0.823 trigger?

tau(31) = 0.903; since 0.823 < 0.903 the belief escalates to a human, even though the raw sensor confidence was 0.95.

What does the layer do during cold start, and why is that a blind spot?

It passes the raw LLM confidence through unchanged (no calibration) until n reaches n_min = 10, so it provides no anomaly protection during the first documents of a context's life.