IN THIS BREAKDOWN · 10 SECTIONS
You can read 20% of this breakdown.
The paper in one minute
A common way to get better answers from a model on hard problems is to sample many solutions and let a reward model pick the best one. The reward model is only as useful as its training signal. The usual signal is the outcome: did the solution reach the correct final answer? OpenAI's paper compares that with process supervision, where people label each reasoning step.
The team fine-tuned a GPT-4 base model to write step-by-step solutions to MATH competition problems, one step per line. Labelers marked each step positive, neutral or negative and stopped at the first negative step. To spend labeling effort well, they were mostly shown solutions that the current process reward model rated highly but that reached a wrong final answer, so every label corrected a mistake the model was making. The result, PRM800K, contains about 800,000 step labels on 75,000 solutions to 12,000 problems. A process reward model (PRM) trained on it scores a solution as the product of its steps' probabilities of being correct.
Selecting from 1,860 samples per problem on a 500-problem MATH test subset, the PRM picked a correct solution 78.2% of the time, an outcome reward model (ORM) trained on far more automatically graded samples 72.4%, and majority voting 69.6%. The gap widened as more samples were compared. In a cheaper replication with small models, where the large PRM served as the labeler, process supervision beat outcome supervision at every data size. The PRM also led on 234 recent AP and AMC problems that the models could not have seen.
Research notes: the problem, approach and results
MATH problems solved by best-of-1860 selection with the process reward model, against 72.4% outcome (Figure 3)
step-level human labels on 75,000 solutions to 12,000 problems, released as PRM800K (Section 2.4)
data efficiency of active learning over uniform labeling in the small-scale study (Section 4.2)
Reward models that judge reasoning are usually trained on whether the final answer was right. That label says nothing about where a wrong solution failed and rewards right answers reached by wrong reasoning. An earlier study on grade-school math found that step-level labels did no better.
Have people label each step of model-written solutions as positive, neutral or negative, stopping at the first mistake, and train a process reward model on those labels. Choose which solutions to label with active learning: show labelers the solutions the current model likes that still reach a wrong answer.
With a GPT-4-based generator and reward models, best-of-1860 selection solved 78.2% of a 500-problem MATH test subset with the process model, 72.4% with the outcome model and 69.6% with majority voting. Active learning made labeling about 2.6 times more data-efficient. The 800,000 labels were released as PRM800K.
Why it matters to an FDE
Most enterprise reasoning tasks are multi-step: reconcile accounts, apply a policy, triage an incident. This paper is the clearest evidence on how to supervise a checker for that kind of work.
Check steps, not only outcomes, when outcomes are a weak signal. If most attempts at a task are wrong somewhere, "wrong" tells a reviewer model little. Step labels say where it went wrong. For a customer's workflow, that means a rubric per step (was the right record fetched, was the rule applied, was the amount computed) rather than a single pass or fail.
Spend reviewers on the cases the model gets wrong confidently. Labelers here mostly saw solutions the reward model liked that were nonetheless wrong. In the small-scale study this was about 2.6 times more efficient than labeling at random. The same idea applies to building a judge for a customer: send reviewers the outputs your current judge passes but a ground-truth check fails.
Right answer, wrong reasoning, is a real failure. Final-answer grading marks those as correct. For regulated decisions the reasoning is the record, so a checker that only sees the outcome will pass explanations you cannot defend.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 290 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
What process supervision adds, how scoring works, active learning, limits, and two misreadings
Free account - 04How it works
The generator, the labeling task, active learning, the two reward models and scoring
Premium - 05The numbers that matter
Large-scale best-of-N, the small-scale comparison, active learning, OOD exams and scoring
Premium - 06Try it yourself
Count problems, recheck Table 1, see product scoring's length bias, and run a best-of-N toy
Premium - 07Where it breaks
Cost of labels, unequal training sets, one domain, contamination, no generator training and slips
Premium - 08Using it on a deployment
Building a step-level checker for a customer workflow and spending reviewer time well
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium