How Models Verify Their Reasoning: Energy, Likelihood, and Reward

Large language models tackle reasoning problems by generating chains of intermediate steps, but a fluent derivation is no guarantee of a correct one. Sampling many candidates and selecting among them helps, but the best selectors - learned verifiers or reward models - are typically as large as the generators they judge. The Energy Outcome Reward Model (EORM) offers a lightweight alternative: a small energy-based model, trained on binary outcome labels alone, that gives correct candidates lower energy and reranks by it.
This thesis rebuilds that recipe in a controlled setting: a synthetic shortest-path benchmark where correctness is checked exactly, difficulty is tunable, and wrong candidates are locally valid yet globally wrong, so plausibility and correctness come apart. We train million-parameter energy verifiers from outcome labels, use them to rerank the samples of a small supervised generator, and finally promote them to reinforcement-learning rewards, measuring where the learned signal departs from ground truth.

Three findings follow. First, verification is learned late and through a shortcut: under a conventional budget the verifier matches a zero-parameter "prefer the short path" heuristic; only hard negatives, a listwise objective, and a long schedule carry it through a delayed transition to near-oracle selection (0.816 vs an oracle of 0.835). Second, test-time selection beats training-time reinforcement at this scale: Best-of-8 reranking improves on greedy decoding (0.765 vs 0.728), while policy optimization, even against the exact checker, improves only its training distribution. Third, and centrally, selection competence does not imply optimization robustness. The verifier that wins every selection comparison is destroyed as a reward - proxy rising while true accuracy collapses from 0.63 to 0.26 - because selection queries it only on the policy's distribution, while reinforcement learning drives the policy where it was never constrained.