Uncertainty, calibration and the limits of LLMs

Understand what calibration means, the difference between confidence and correctness, decision under uncertainty, interpretability as partial insight, and the honest limits of LLMs — grounding the lab's stance that what cannot be proven is UNKNOWN, never SAFE.

Track

Research

Module

9 · Agent cognition & reasoning

Prerequisites

Verifiers, reward models and metacognition

Accuracy is an average over cases; a decision is made on one case. If a system is right 90% of the time but cannot say which answers belong to the other 10%, its average is useless at the moment of acting. The remedy is to treat uncertainty as a first-class quantity — something estimated, checked against reality, and fed into a decision — rather than as a vague sense of doubt. This lesson makes that quantity precise and then asks the harder question: how much of it can we actually trust.

It helps to separate two sources from the start. Aleatoric uncertainty is the irreducible variability that would remain even if the model were perfect: label noise, ambiguous questions, facts the context does not determine. Epistemic uncertainty comes from not knowing the model's parameters — too little data in a region, extrapolation. The distinction matters because the two call for different responses, and because, as we will see, a single frontier LLM is poor at reporting the epistemic kind precisely where it matters most.

Calibration asks a specific question: of all the times the system says 0.8, is it right 80% of the time? Formally, a predictor emitting a class with confidence p is calibrated (on the predicted label) if the probability of being correct, given that it reported p, equals p for every p. That is not the same as being good — a predictor that always says 0.6 on a task it gets right 60% of the time is perfectly calibrated and discriminates nothing.

Calibration

Reported confidence matches the empirical frequency of being right. It is necessary for the abstention rule to work, but says nothing about separating hard cases from easy ones.

Discrimination (resolution)

Assigning more confidence to correct answers than to wrong ones. Measured by the risk-coverage curve or AUROC of errors; it depends only on the ordering of confidences, not their absolute values.

Proper scoring rule

A loss minimised in expectation by announcing the true distribution. The two in common use are negative log-likelihood and the Brier score; a proper rule cannot be improved by calibrating alone or discriminating alone — it measures both at once.

Aleatoric vs epistemic

Irreducible task noise versus reducible ignorance of parameters. Total predictive uncertainty decomposes into the two, and the epistemic part is the disagreement among plausible models — which is zero if every model predicts the same, even where none of them knows anything.

The Brier score decomposes (Murphy's decomposition) into reliability (the calibration term), resolution (the discrimination term) and an irreducible uncertainty term fixed by the task. That decomposition is why calibration and discrimination cannot be collapsed into one number: recalibrating moves the reliability term and leaves resolution untouched. A model that does not discriminate cannot be fixed by any amount of recalibration — recalibration changes where you put the threshold, not how much any threshold buys you.

Two cautions keep confidence from being mistaken for correctness. First, the expected calibration error (ECE) that everyone reports is a biased estimator. Because confidence is continuous, it is estimated by binning, and binning has two biases of opposite sign: noise pushes it up (even a perfectly calibrated predictor shows a positive gap per bin, growing roughly as the square root of the number of bins over n), while aggregation within a bin pushes it down. The practical consequence is blunt — an ECE reported without n, without the number of bins, and without the binning scheme is not comparable to another, and a one-point difference at n ~ 1000 can be estimator noise.

synthetic — equal-width ECE sketch, with the bias caveat made explicit
import numpy as np

def ece(conf, correct, n_bins=15):
    """Equal-width ECE. conf in [0,1]; correct in {0,1}.
    NOTE: biased. Report n, n_bins and scheme; a perfectly
    calibrated predictor still shows a gap ~ sqrt(n_bins / n)."""
    edges = np.linspace(0.0, 1.0, n_bins + 1)
    e, n = 0.0, len(conf)
    for lo, hi in zip(edges[:-1], edges[1:]):
        m = (conf > lo) & (conf <= hi)
        if m.sum() == 0:
            continue
        acc  = correct[m].mean()        # empirical accuracy in the bin
        avg  = conf[m].mean()           # mean reported confidence
        e   += (m.sum() / n) * abs(acc - avg)
    return e
# A low ECE is necessary, not sufficient: it can hide subpopulations
# that over- and under-confide and cancel in the average.

Second, the signal the model reports about itself is not automatically trustworthy. Reported studies disagree in revealing ways: large models can be well calibrated on multiple-choice and true/false in the right format; verbalized confidence ("I am 80% sure") is sometimes better calibrated than the model's conditional probabilities, and sometimes systematically over-confident, improving with capability and with agreement across samples. These do not contradict — they compare different signals and models, and a signal that is better calibrated than another can still discriminate poorly. Recalibration (temperature scaling, isotonic regression, binning) can repair the scale of a confidence on held-out data, but it cannot manufacture discrimination, and a verbalized confidence is itself a model output a system can be optimised to fake.

Turning a calibrated probability into a decision is a cost calculation. If a correct answer is worth 1, a wrong one costs c, and abstaining is worth 0, the expected utility of answering with probability-of-correct p is p - c(1 - p), so it is worth answering only when p > c/(1+c): the threshold is 0.5 at c = 1 and 0.8 at c = 4. This is the classical optimal-rejection rule, and its fragility is worth stating plainly — it is optimal only if p is calibrated. An over-confident model answers too often and an under-confident one abstains too often, so the same model needs recalibration before the rule means anything.

There is also a method whose guarantee does not depend on the model being good. Conformal prediction returns a set of answers that contains the true one with probability at least 1 - alpha, using a calibration set and a non-conformity score. The guarantee is remarkable for what it does not require: not that the model is accurate, not that it is calibrated, not any shape of the distribution — a bad model just yields larger sets, not lower coverage.

If the model will not reliably report its own uncertainty, can we read it from the inside? Interpretability asks what a trained model computes and how, from its activations and weights, and it is honest about what each tool can prove. Probes (linear classifiers on intermediate representations) measure what information is available to be read, not whether the model uses it. Interventions (ablation, activation patching, linear erasure) contrast necessity and sufficiency. Circuits describe a mechanism with variables. Sparse autoencoders try to undo superposition — the phenomenon by which a network represents more features than it has dimensions.

The crucial distinction, provable in a minimal model, is a ladder: available, used, explained. A variable can be perfectly decodable from a representation and never used by the model — a probe reading a channel nothing downstream consumes decodes it at 100% and that proves nothing about use. Establishing use requires intervention; explaining requires a high-level causal model whose swap-interventions reproduce the behaviour; and validating requires predicting interventions not used to build the explanation. Every claim must declare which rung it reached.

The honest state of the field is that interpretability is a scientific tool, not an engineering guarantee. The cleanest results are on small models and bounded tasks; techniques for large models explain a surrogate (a sparse-autoencoder dictionary, a transcoder) whose distance from the model is part of the uncertainty of the explanation; reported at-scale analyses give a satisfying account on roughly a quarter of the prompts tried, on a biased sample. Interpretability can refute a hypothesis about how a task is solved and surface unexpected mechanisms, but it does not certify that a model will not do something on inputs no one analysed. Localising a mechanism is not the same as controlling it.

"Does the model understand?" is not answerable as stated; several smaller, measurable questions hide behind it, and the evidence should be read on a ladder that each rung narrows. A benchmark score is compatible with memorisation or shortcuts; contamination controls (new sets built after the model's cut-off) separate performance from memory; compositional, length and counterfactual splits test the rule rather than a lookup table; perturbation robustness measures sensitivity to irrelevant changes; and mechanistic evidence links behaviour to a representation. The conclusion out of the ladder is never "the model understands" but "the model has capability K under conditions C, measured with instrument M".

• ▸Compositional generalisation: models trained from scratch on controlled splits generalise within distribution and fail systematically out of it; in LLMs a measured compositionality gap (failing a two-hop question whose sub-questions both succeed) does not shrink with scale.
• ▸The reversal asymmetry: a fact learned as "A is B" is not automatically available as "B is A" — reported directly-versus-inverse accuracy differs sharply — unless the fact is in context or both directions are in the data.
• ▸Attraction to the probable: on counterfactual variants, ciphers, or local conventions, accuracy stays above trivial but consistently below the default version, because the autoregressive objective pulls answers toward what is frequent in the corpus.
• ▸Format and order sensitivity: changes to the prompt template or example order that do not alter the task can move accuracy by large margins, persisting with scale.
• ▸Depth decay and contamination: long chains fail even when each step is reliable (the p^m effect), and some benchmark performance measures memory of the test set rather than generalisation.

These are not noise; they are the coherent response of a model to an objective that does not penalise them, and the first three are especially dangerous because the model answers with the same fluency when it is wrong as when it is right. Whether any of this amounts to "understanding" mixes an empirical part, which advances, with a conceptual part, which no benchmark settles. The engineering response is the one that recurs: use the model where its in-distribution performance suffices, and check outside it with components whose generalisation is known — an interpreter, a validator, an explicit policy.

Pulling the threads together gives the lab's operating rule. Uncertainty is a separate channel from the model: a confidence signal is recalibrated or turned into a conformal set using domain data, and a decision policy outside the model sets costs and thresholds and is what gets audited. Changing the cost of an error changes the threshold, not the model. For an agent, the relevant uncertainty is not only over the answer but over the consequences of acting, and there the epistemic kind matters most — an action on a system the model has never seen should trigger escalation even when the verbalized confidence is high, precisely because the model is poor at epistemic uncertainty where it has no data.

• ▸Confidence is not correctness: a calibrated number tells you how often "0.8" is right on average, not whether this answer is right, and a model answers wrong cases as fluently as right ones.
• ▸A guarantee is only as good as its conditions: conformal coverage is marginal, average over calibration sets, and void under shift; cite it with n, alpha and the exchangeability assumption.
• ▸Interpretability is partial insight: it can refute and discover, but it does not certify behaviour on unexamined inputs; available is not used, and localising is not controlling.
• ▸The model informs; it does not decide: the threshold, the cost and the authority to act are external, and must not be read off the model's own confidence.
• ▸What cannot be proven is UNKNOWN, and UNKNOWN is treated as unsafe. High confidence, a passing check, or a plausible chain-of-thought is not proof — when verification is impossible, the agent abstains or escalates.

Expected direction: With c = 4 the abstention threshold is p > c/(1+c) = 0.8, so 0.88 clears it only if 0.88 is a CALIBRATED probability of correctness — and the rule is optimal only under calibration, so the confidence must first be recalibrated on labelled invoices, ideally per field and per vendor. The verbalized "high confidence" is a model output that can be over-confident or faked and carries no guarantee. Crucially this is a NEW vendor: the relevant uncertainty is epistemic, and a single model reports epistemic uncertainty poorly exactly where it has no data, so a high confidence here is least trustworthy. A sound design: a per-field signal, a per-vendor recalibrator, a cost-based threshold that rises with the amount (or a conformal set that routes any ambiguous field to review), and a labelled sample of what was auto-posted to detect drift. The confidence is an input to an external policy, not the policy; for an unverified new vendor the extraction is UNKNOWN, not SAFE, so it should go to human review rather than auto-post.

What is the difference between calibration and discrimination, and can recalibration fix a lack of discrimination?

Calibration means reported confidence matches empirical accuracy; discrimination means assigning more confidence to correct than to wrong answers. Recalibration moves the reliability (calibration) term of the Brier decomposition but leaves resolution (discrimination) untouched, so it cannot fix a model that does not discriminate — it only sets where the threshold goes.

State the abstention rule and its fragility.

Answer only when the probability of being correct p > c/(1+c), where c is the cost of a wrong answer relative to a correct one. It is optimal only if p is calibrated: an over-confident model answers too much and an under-confident one abstains too much, so recalibration is required first.

Why does a perfect probe not prove the model uses a variable?

Availability is not use: a variable can be perfectly decodable from a representation the model never reads downstream (a probe on such a channel decodes at 100%). Establishing use requires intervention showing that changing the representation changes the output coherently; explaining requires a causal model validated on interventions not used to build it.

Give the lab's rule for what to do when verification is impossible.

What cannot be proven is UNKNOWN, and UNKNOWN is treated as unsafe — never safe-by-default. High confidence, a passing check, or a fluent chain-of-thought is not proof; the model informs a decision but the threshold, cost and authority are external, and when verification is impossible the agent abstains or escalates.