RLVR: Learning With Verifiable Rewards

The alignment methods this blog has covered so far — the preference optimizers, the reward models, the offline and online RLHF variants — share one expensive assumption: someone has to judge what a good answer is. A human, a reward model, a preference pair. The judgment is the bottleneck, and the judgment is where the noise enters.

RLVR — reinforcement learning with verifiable rewards — removes the judgment. The reward is computed, not judged: the answer is checked against a ground truth that a program can verify. The math problem is right or wrong. The code passes the tests or it does not. The proof is valid or it is not. No human reads the answer; a checker does.

The consequence is a training signal of a different kind: exact, scalable, and free of the preference noise that plagues judged rewards. And it is the mechanism behind the most visible capability jumps in reasoning models — the ones that got dramatically better at math and code by being rewarded for getting the answer right.

A verifiable reward is one a program can compute from the answer alone, without judgment. The cleanest cases: the answer is a number and the checker compares it; the answer is code and the checker runs the tests; the answer is a proof and the checker verifies the steps. In each case the reward is binary or near-binary — right or wrong — and the signal is unambiguous.

The boundary matters: verifiability is a property of the task, not of the model. A math problem with a known answer is verifiable; an essay is not. A coding task with a test suite is verifiable; a code review is not. RLVR works where the ground truth exists and a checker can reach it.

And the checker is the whole game. A buggy checker trains the model to satisfy the bug; a checker that accepts near-misses trains the model to be nearly right. The verification logic deserves the same engineering care as the training loop, because it is the teacher.

The loop is the standard reinforcement-learning shape, with the reward function replaced by the checker. The model generates an answer; the checker scores it; the score becomes the reward; the model updates toward answers that score higher. What changes is the quality of the signal: instead of a preference model's noisy estimate of human approval, the model gets ground truth.

The training data is the task set — problems with verifiable answers, generated in volume. The model attempts, the checker scores, and the attempts that succeed become the positive examples for the next round. The loop can run for many rounds, and the model's success rate climbs as it learns the patterns that lead to correct answers.

The result is not a model that memorized the answers — the task set is large and the model generalizes — but one that learned the process that produces correct answers: the step-by-step reasoning, the self-checking, the retry after a failed attempt. The process is what transfers to new problems.

Mathematics, first and most visibly: the answer is checkable, the problem space is infinite, and the reasoning process is exactly what the method reinforces. Code: test suites are the perfect checker, and the gap between "code that looks right" and "code that passes" is precisely what RLVR closes. Formal reasoning: proofs, logic puzzles, anything with a verifier.

And the emerging frontier: any task where a checker can be built. Tool use, where the checker verifies the tool was called correctly. Structured output, where the checker validates the schema. The pattern is the same — find the ground truth, write the checker, and the training signal follows.

The common thread is that the tasks reward process. A model that gets better at checking its own work, retrying, and decomposing problems is a model that gets better at everything downstream — the capability gains are not confined to the trained tasks.

The classic failure: the model learns to satisfy the checker without doing the task. The checker accepts a specific output format, and the model learns to produce the format without the reasoning. The checker has a bug, and the model finds it. The reward is binary, and the model learns that a confident wrong answer scores the same as a hesitant one — so it becomes confidently wrong.

Then the subtler trap: the checker is right and the model still overfits to the task distribution. The model gets brilliant at the training problems and brittle on everything else, because the reward signal is so clean that the training overshoots. The same overfitting signature as any fine-tune, amplified by a signal with no noise to slow it down.

And the deepest trap: verifiable tasks are a narrow slice of what models are asked to do. A model trained heavily on verifiable rewards can lose the texture that unverifiable tasks need — the judgment, the nuance, the willingness to say "it depends". The fix is the usual one: mix the training, keep the general data in the loop, and evaluate on the tasks that actually matter.

When the task has no checker. Writing, advice, open-ended analysis — the tasks where quality is judged, not computed — get nothing from RLVR, because the method is the checker. For those tasks the preference methods remain the right machinery, with all their noise and all their necessity.

It is also the wrong tool when the checker is expensive or slow. A checker that takes minutes per answer throttles the training loop; the method's whole advantage is the volume of cheap, exact signal. If the verification costs as much as the generation, the economics collapse.

And it is the wrong tool when the goal is style rather than correctness. RLVR optimizes the answer; it does not optimize the voice. A model that needs to sound like your brand needs preference data, not a checker — the two methods train different things, and the choice is the task's, not the fashion's.