STaR: Bootstrapping Reasoning With Reasoning

Models answer reasoning questions better when they write a rationale first, but teaching a model to do that by fine-tuning needs thousands of written rationales. Few-shot chain-of-thought prompting needs only a few, but for mid-sized models it is weak: GPT-J 6B scored 36.6% on CommonsenseQA with chain-of-thought examples, against 60.0% when fine-tuned to output answers directly.

STaR, the Self-Taught Reasoner, turns the few examples into a large training set. Prompted with about ten example rationales, the model writes a rationale and an answer for every question in a dataset that has answers but no rationales. Rationales that end in the correct answer are kept. The model is fine-tuned on them, and the improved model writes the next round of rationales. Each round restarts fine-tuning from the original pretrained model to avoid overfitting.

That loop stalls on questions the model never gets right, because they never enter the training set. The paper's second idea, rationalization, fixes this: for a missed question, the model is shown the correct answer as a hint and asked to write a rationale again. If it reaches that answer, the rationale is added to the training set as if the model had written it unaided. On CommonsenseQA, STaR with rationalization reached 72.5%, close to a GPT-3 175B fine-tuned on answers (73.0%). It also improved GSM8K and multi-digit addition. The authors note that the method needs a model with some few-shot reasoning ability to start and works poorly when answers can be guessed.

STaR is an early template for many reasoning-data pipelines that followed, including the rejection-sampling fine-tuning stages in later reasoning models. Three ideas from it apply directly to customer work.

Answers you already have can become reasoning data. Many customers have years of labeled outcomes (approved claims, resolved tickets, reconciled invoices) and no written reasoning. A STaR-style loop asks the model to explain how to reach each known outcome and keeps the explanations that do.

The filter is only as good as the check. STaR keeps a rationale if the final answer matches. With numeric answers, a lucky match is rare. With five options or yes/no, many wrong rationales reach the right answer by chance and get kept. Know your check's false-pass rate before you trust the kept set.

Rationalized data needs review. A rationale written after seeing the answer can argue backwards. The authors warn that the method amplifies whatever reasoning reaches the labels, including biased shortcuts, so sample and read the rationalized examples before training on them.