Verifiers, reward models and metacognition

Understand the mechanisms that score correctness: outcome vs process reward models (ORM/PRM), executable and formal verifiers, generative verifiers, self-critique and self-correction, and computational metacognition (estimating one's own uncertainty and choosing to abstain). Derive the verifier as a noisy signal — why the best-of-N ceiling is a verifier's precision, why a small fraction of "convincing" errors makes more candidates WORSE, and why an independent verifier beats self-report — and how this grounds the lab's principle that a finding exists only when a verifier confirms it.

Track

Research

Module

9 · Agent cognition & reasoning

Prerequisites

Reasoning and test-time compute

Separating generation from verification is an old idea in computer science: the NP problems are precisely those where checking a solution is easy even when finding it is hard. Brought to language models, it becomes the engine of test-time compute — generate many candidates, then score them. But the scoring is itself fallible, and this lesson is about what a verifier adds and how much you can trust it.

The central question has two halves. The first is epistemic: a verifier improves the result only if its error is smaller than, or at least DIFFERENT from, the generator's — a model judging itself with the same information adds no evidence. The second is about optimization: an imperfect verifier used as a selection criterion is a surrogate objective, and optimizing it hard exploits its failures. The lesson gives both halves a quantitative form.

Outcome reward model (ORM)

Estimates the probability that a COMPLETE solution is correct. Trained as a classifier on the generator's own solutions, labeled by final-answer correctness — cheap when a reference answer exists. It receives one label per solution and must infer where the error is.

Process reward model (PRM)

Scores each STEP given the prefix. It receives the localization of the error, so it can flag a solution with a correct answer but faulty reasoning — which an outcome model trained on the final answer rewards. The cost is expensive or noisy step labels and a notion of "step" that is fuzzy outside mathematics.

Executable verifier

Checks a property by running something: unit tests on a program, evaluating an expression, a constraint checker on a plan. Its error is of a different nature — exact about the property checked, blind to everything else. Tests can be incomplete, and if generated by the model itself, wrong.

Formal verifier

Accepts an output only if it is a valid proof in a logical system, checked by a small trusted kernel. Its false-positive rate is in principle zero with respect to the FORMAL statement; all the risk moves to autoformalization — translating the informal statement into the formal one, which can be wrong or trivialize the problem. It guarantees that what was WRITTEN was proved, not what was MEANT.

A generative verifier is a model that produces a verification in text and ends with a judgment whose probability is the score; it can reason before judging and can be sampled several times. The source reports sizable best-of-N gains from such verifiers in the studied conditions. But it inherits the lesson's central hazard: if it shares the generator's biases, its errors are CORRELATED with the generator's, and correlated errors defeat the whole point of a second opinion.

Let a generator produce candidates that are each correct with probability p (the base rate). Generate N, select one with a binary verifier that accepts a correct candidate with probability TPR and an incorrect one with probability FPR. The probability of accepting any candidate is α = p·TPR + (1 − p)·FPR, and the verifier's precision (positive predictive value) is PPV = p·TPR / α. With the rule "pick a random accepted candidate, or a random one if none is accepted", the selection accuracy rises toward PPV as N grows.

This is the number to memorize: the CEILING of best-of-N with a binary verifier is its precision, not its accuracy, and precision depends on the base rate p. The source gives a worked synthetic case — a verifier with 90% on both classes, applied where only 30% of candidates are correct, ceilings out around 0.79 and stops improving no matter how many candidates you add. A verifier that looks excellent leaves a large irreducible error because the incorrect candidates are the majority and a tenth of them slip through. If the base rate falls further, the ceiling collapses. It is exactly the base-rate phenomenon of diagnostic testing.

synthetic — best-of-N ceiling with a binary verifier (schematic)
# base rate p, true/false positive rates of the verifier
def best_of_n_accuracy(p, TPR, FPR, N):
    alpha = p*TPR + (1-p)*FPR            # P(a candidate is accepted)
    PPV   = p*TPR / alpha                # precision = the CEILING
    none_accepted = (1 - alpha)**N
    acc_if_reject = p*(1 - TPR) / (1 - alpha)
    return (1 - none_accepted)*PPV + none_accepted*acc_if_reject

# e.g. p=0.30, TPR=0.90, FPR=0.10  ->  PPV ~= 0.79
#   N=1: 0.300   N=4: 0.652   N=8: 0.767   N=16: 0.793  (then flat)
# MORE SAMPLES CANNOT EXCEED THE VERIFIER'S PRECISION.

With a CONTINUOUS score things can get worse. Model correct candidates as drawing higher scores and incorrect ones lower, and pick the maximum: accuracy tends toward one as N grows — UNLESS a fraction of the incorrect candidates are "convincing", scoring even higher than the correct ones. That fraction is the minimal model of a subtly-wrong-but-well-presented solution, a long confident answer a judge prefers, or a program exploiting a gap in the tests. The source shows a worked case where only about 3.5% of candidates are convincing errors, yet accuracy rises to a maximum and then DECLINES as N keeps growing, because the argmax of more samples is more likely to find one. Formally, if the upper tail of a class of incorrect scores dominates the correct one, selection accuracy tends to zero as N grows. This is the minimal, provable form of reward hacking: optimize a surrogate hard enough and you select its failures.

Verifiers order naturally by cost and by guarantee. Formal and executable verifiers are exact about what they check and cheap to run, but exist only for certain properties. Learned ORMs and PRMs are general but have systematic errors. Generative verifiers are flexible and expensive. A sound architecture applies them in that order.

1. 1Apply EXACT filters first: discard anything that does not compile, fails the available tests, or violates hard constraints. This removes many "convincing" errors cheaply, before any learned scorer sees them.
2. 2Apply the learned verifier (ORM/PRM/generative) to the survivors only, where it has the best chance of adding real signal.
3. 3Apply a selection rule WITH a threshold that can abstain — produce an answer and a confidence, and below the threshold abstain or escalate rather than commit.
4. 4Combine verifiers only when their errors are independent, and MEASURE that independence on labeled cases — two models of the same origin, or a generator judging itself, give a false sense of redundancy.
5. 5Sweep N and watch for non-monotonicity: if accuracy falls as N grows, you have convincing errors, and the fix is another exact filter, not more samples.

By computational metacognition we mean something operational and limited: a system estimating the probability that its own output is correct and acting on it (answer, abstain, ask for more compute, escalate). No introspection or self-awareness is implied. There are three families of signal. Explicit self-evaluation: ask the model for the probability its proposed answer is correct — the source reports this behaves reasonably and scales on many tasks but calibrates worse on NEW tasks. Agreement among samples: the fraction of the mode under self-consistency is a free confidence signal, refined by grouping samples by meaning before measuring disagreement (so paraphrases of the same answer do not count as disagreement). And the signal of a verifier: an ORM/PRM/checker score as a correctness estimate.

All three feed the same decision — selective prediction: set a confidence threshold and answer only above it, trading coverage for risk. The source illustrates with a synthetic abstention simulation where answering everything gives some baseline risk, abstaining on the least-confident fraction cuts it substantially, and abstaining on more cuts it further. These are MEASURED shapes of a curve in a simulation, not results about any specific model.

Now the epistemic half. Self-correction WITHOUT an external signal is not reliable in reasoning: the source documents that apparent gains from self-critique often came from using the correct labels to decide when to STOP — an oracle — and that without it, the model swaps more correct answers for incorrect ones than the reverse. With a reliable external signal (execution, tools, a correct verifier), self-correction DOES help, and simply re-asking with the verifier's signal, without elaborate critiques, keeps most of the benefit. Large-scale reinforcement fine-tuning can CREATE a revision ability — but then what is trained is a revision policy, not a free property of critique. This is the empirical backbone of "an independent verifier beats self-report".

In a retrieval-augmented system, the natural verifier checks that each claim in the answer is SUPPORTED by the retrieved documents — a learned verifier with all the failures above, but also a valuable abstention signal when the evidence does not support the answer. In an agent, the verifier decides whether a step had the intended effect and whether the goal was reached; when there are external effects, executable verification against the environment state (does the file exist? does the ledger balance?) is far more reliable than asking the model whether it did well.

This is where the lab's thesis gets its teeth: the verifier is the natural place to take the decision OUT of the model. A finding exists only when an independent verifier confirms it — because best-of-N math shows a selected answer is correct only up to the verifier's precision, and self-report is exactly the correlated, non-independent signal that adds no evidence. But a LEARNED verifier is not an authority either; it is just another estimate. Using it as an authorization barrier — deciding that an action is permitted — is a separate, stricter question than using it to rank candidates, and belongs to an external policy with its own guarantees. Reasoning is not authorization, and neither is a high verifier score.

• ▸Reward hacking of the verifier: hard selection picks errors the verifier scores high (the non-monotone curve); its symptom is the verifier score rising while accuracy measured by another signal falls — so sweep N and watch the whole curve.
• ▸Surface biases: preference for long, confident, or specifically-formatted answers, which an optimized generator learns to produce on purpose.
• ▸Distribution drift: a PRM trained on one generator's solutions evaluates a stronger generator's solutions poorly — exactly when it is needed most.
• ▸Degrading self-correction: revising without an external signal swaps correct answers for incorrect ones on net; never evaluate self-correction with an oracle stop condition.
• ▸Unfounded confidence: verbal self-evaluations that do not track accuracy, especially out of the training distribution — a reason to prefer an independent, external check.
• ▸The filtered-oracle trap: evaluations that secretly use the correct answer to decide when to stop or what to fix, reporting gains that vanish in deployment.

Expected direction: Coverage rising while selected accuracy falls is the signature of convincing errors against an imperfect verifier: some programs pass the VISIBLE unit tests but are actually wrong (the tests have limited coverage), so they score at the verifier's ceiling, and as N grows the argmax is increasingly likely to pick one — the non-monotone best-of-N curve, i.e. reward hacking of the verifier. The ceiling is the verifier's precision at this base rate, not N. The fix is NOT more samples; it is to strengthen the exact filter — add hidden or generated tests, property-based checks, or static analysis to catch the convincing failures before selection — and to cap N near the measured maximum. If independence can be established, add an independent verifier whose errors do not correlate with the visible tests.

Why is the ceiling of best-of-N with a binary verifier its precision rather than its accuracy, and why does the base rate matter?

As N grows, almost surely at least one candidate is accepted, and the selected one is a uniform random accepted candidate — correct with probability PPV = p·TPR / (p·TPR + (1−p)·FPR), the verifier's precision. No number of samples can exceed it. The base rate p enters PPV directly: when correct candidates are rare, even a small false-positive rate on the abundant incorrect ones drags precision down, so a verifier with great per-class accuracy can still ceiling low.

Explain how a small fraction of "convincing" errors makes more candidates worse, not better.

If a fraction of incorrect candidates receive scores drawn from a distribution whose upper tail beats the correct candidates', then as N grows the maximum score is increasingly likely to come from a convincing error, and argmax selects it. Accuracy rises then declines, with a finite optimal N. It is the minimal form of reward hacking: optimizing an imperfect surrogate hard enough selects its failures.

Why does an independent verifier beat self-report, and what makes a second model NOT independent?

A verifier adds evidence only if its error is smaller than or different from the generator's; a model judging itself with the same information contributes no new evidence, and self-correction without an external signal swaps correct answers for incorrect ones on net. A second model is not independent when its errors correlate with the first — e.g. same origin, shared biases, or the generator judging itself — which gives a false sense of redundancy; independence must be measured on labeled cases, not assumed.

How does this lesson ground the principle "a finding exists only when a verifier confirms it", and where is the limit of that principle?

Because best-of-N shows a selected answer is correct only up to the verifier's precision, and self-report is the correlated signal that adds no evidence — so confirmation must come from an independent (ideally exact) verifier outside the model. The limit: a LEARNED verifier is itself just another estimate with systematic errors and can be hacked, so confirming a finding is not the same as AUTHORIZING an action. Authorization is a stricter, separate decision for an external policy; a high verifier score is not an authority.