Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

Rubrics as Rewards trains a model using criteria tailored to each task. Instead of asking a judge for an unstructured overall rating, it supplies a rubric describing what the answer should accomplish and which mistakes matter. The resulting feedback becomes the reward in an on-policy reinforcement-learning loop.

The paper compares two aggregation methods. Explicit aggregation scores criteria separately and combines them with chosen weights. Implicit aggregation gives the rubric to a judge and asks for a combined quality score. The latter performs best overall in the reported medicine and science experiments, while explicit aggregation provides more direct control over the arithmetic.

The approach depends on the quality of both rubric and judge. Most training rubrics are generated from reference answers, which serve as a proxy for expert guidance. Making criteria visible helps inspection, but it does not make model judgments infallible or transform subjective requirements into mathematically verified facts.

The useful idea extends beyond training: write down what a good response must satisfy before scoring it. Applying that idea to an enterprise evaluation is reasonable; claiming the paper proves a production agent is safe would go beyond its experiments. The released RaR-Medicine and RaR-Science datasets document the authors' training materials.

A customer may reject an answer that is factually correct but incomplete, unsupported or unsuitable for the requested workflow. A single “good answer” score hides those distinctions. A rubric makes the disagreement inspectable and gives the team a more useful starting point for evaluation or adaptation.

For example, a support-response rubric could distinguish identifying the issue, citing the relevant policy and avoiding disclosure of another tenant's data. Those conditions require different evidence. A model judge may help assess explanation quality, while authorization must remain enforced by the application.

Start with LLM-as-a-judge and a reviewed set of examples before attempting RL. If the rubric cannot reliably distinguish acceptable from unacceptable answers offline, using it as a training reward will not repair that weakness. It may instead teach the policy to satisfy the judge's mistakes more consistently.