Learning to calibrate: experts + human feedback

Understand the mechanism the paper proposes for context-aware calibration: a per-context mixture of experts whose gating network is trained with reinforcement learning from human feedback, plus an adaptive escalation threshold and a cold-start rule.

Track

Research

Module

5 · Research case studies

Prerequisites

The LLM-sensor confidence problem

Part 1 established where calibration sits and what it is allowed to touch. This lesson is about how it actually computes a calibrated confidence and how it learns. Two ideas are combined in a non-obvious way: a Mixture of Experts for the "which specialist should I trust here?" question, and Reinforcement Learning from Human Feedback (RLHF) applied not to a language model but to the MoE gating network.

Mixture of Experts routes an input to one or more specialist sub-models via a gating function that outputs weights over experts; the final output is the weighted combination. RLHF trains a component from human-preference-derived reward signals rather than labeled targets. The paper uses both because, at deployment time, there is no labeled "correct confidence" dataset — only operational feedback.

Expert

One specialist per document type. In this proposal each expert is a k-nearest-neighbor predictor over a fixed text-embedding space, holding a memory of (embedding, prior confidence, reward) triples.

Gating network

A per-context weight vector over the available experts; it decides how much each expert's opinion counts for the current input.

Reward

A scalar feedback signal. Here it is sourced automatically from verifier outcomes and, with larger magnitude, from human reviewer approve/reject verdicts.

Each expert predicts a confidence for a new input by looking at its nearest neighbors in embedding space and returning a similarity-weighted blend of their recorded confidences; if an expert has too few neighbors it abstains. The gating network produces softmax-normalized weights over the experts that did not abstain, and the calibrated confidence is the weighted sum of their predictions. If every expert abstains, the layer falls back to the raw LLM confidence — a deliberate cold-start safety default.

calibrated confidence (schematic)
for each document type d:
    E_d(e) = similarity-weighted blend of k nearest neighbors' recorded confidence
             ( abstain if fewer than k neighbors in memory )

gate weights w = softmax over experts that did not abstain
calibrated_conf = sum_d  w_d * E_d(e)
# if all experts abstain -> calibrated_conf = raw LLM confidence

The gate is updated with a bandit-style gradient step: experts that contributed more to a decision that was later rewarded gain weight; those that contributed to a penalized decision lose weight. Weights are floored at a small positive value and renormalized, so no expert is ever permanently killed off.

The reward design creates a natural curriculum. Automatic signals from the verifier (for example, a belief validated without needing escalation is a small positive; a belief that had to be discarded is a larger negative) have small magnitude. Human reviewer verdicts (approve / reject) carry the largest magnitude. So the system first learns cheaply from its own automatic outcomes and is later corrected, more forcefully, by humans when they are in the loop.

Finally, an adaptive escalation threshold decides what fraction of documents go to a human. With no history the threshold is set for maximum oversight (almost everything escalates); as the context accumulates evidence, the threshold relaxes toward a floor so that only genuinely low-confidence items escalate and the system handles most routine documents autonomously. A cold-start rule holds the whole layer inert until a minimum number of documents has been seen, after which the expert mode activates.

The paper reports a single-session run over a small homogeneous corpus and gives concrete milestones. These are measured values on that one corpus, not benchmarks — read them as an illustration of the dynamics, not as performance guarantees.

reported state at milestones (one small corpus — measured, not a benchmark)
n = 0    mode = cold_start   threshold ~ high (max oversight)
n = 10   mode -> expert (k-NN) activates as designed
n = 31   threshold relaxed to a lower value; most routine docs handled autonomously

The qualitative story the authors show is a mode transition from cold-start to active calibration once enough evidence exists, and a monotone relaxation of the human-escalation threshold as the context proves itself consistent. The exact numbers are specific to that corpus and session.

For agent builders, the transferable pattern is: learn calibration from operational feedback you already collect, not from a labeled dataset you do not have. Verifier outcomes and human overrides are free supervision. Keeping the learner as a small gate over interpretable experts (here, nearest-neighbor memories) also keeps the learned part auditable — you can inspect which neighbors drove a calibrated score.

The adaptive threshold is the safety knob: it ties the human-oversight rate to accumulated evidence rather than to a fixed constant, which is appealing in regulated settings where "a human reviewed the risky cases" must be demonstrable. But note the dependency: if the accumulated history is itself poisoned or out-of-distribution, relaxing oversight is exactly the wrong move — calibration must be paired with input-integrity controls.

• ▸MEASURED: mode transition at the designed document count, and a relaxing escalation threshold, observed on one small homogeneous single-session corpus. Reported as a demonstration of the mechanism.
• ▸EXPERIMENTAL / design choice: experts are k-nearest-neighbor predictors; the authors describe stronger experts (per-type trained classifiers) and a policy-gradient update as planned successors, i.e. not yet evaluated.
• ▸LIMITATION: no labeled correct-confidence ground truth exists at deployment, so the reward signal is operational feedback, which can be biased by whoever reviews and by the distribution seen so far.
• ▸OPEN: behavior across heterogeneous document types and under adversarial inputs is not evaluated here.

Expected direction: It creates a curriculum: the gate first learns cheaply and quickly from abundant automatic verifier outcomes, then is corrected more forcefully by the rarer, higher-magnitude human verdicts. Risk: the learned calibration inherits any bias in the human reviewers or in the distribution of documents seen so far, since there is no labeled ground truth to anchor it.

What is trained with RLHF in this proposal, and what is NOT?

The Mixture-of-Experts gating network is trained with RLHF; the language model (the sensor) is never modified.

What happens when all experts abstain?

The layer falls back to the raw LLM confidence — a deliberate cold-start safety default.

What does the adaptive escalation threshold tie the human-oversight rate to?

To accumulated evidence for the context: maximum oversight with no history, relaxing toward a floor as the context proves consistent.