IN THIS BREAKDOWN · 10 SECTIONS
You can read 23% of this breakdown.
The paper in one minute
Applications ask models for JSON because free text is hard to parse reliably. There are three common ways to get it. JSON mode constrains decoding so the output is valid JSON. Format-restricting instructions ask for JSON, XML or YAML following a given schema, without enforcing it. NL-to-Format lets the model answer in natural language, then makes a second call to convert the answer into the format. The paper asks whether these restrictions change how well the model answers.
The authors test three reasoning tasks (GSM8K math, concatenating the last letters of words, tracking shuffled objects) and four classification tasks (medical diagnosis among 49 conditions in DDXPlus, financial text in MultiFin, sports plausibility, and a stereotype task known to be sensitive to formatting). Models are GPT-3.5 Turbo, Claude 3 Haiku, Gemini 1.5 Flash, LLaMA 3 8B Instruct and Gemma 2 9B Instruct. Each schema has two fields, reasoning and answer. Every setting uses nine prompt variants, and answers are extracted by an LLM so that a parser's fragility does not decide the score.
On reasoning tasks, stricter restriction usually meant lower accuracy, with JSON mode often worst. One cause was field order: in GPT-3.5's JSON-mode responses on Last Letter, the answer key came first every time, so the model answered before reasoning. NL-to-Format matched free text. Adding a schema to a format instruction lowered scores and raised variance for three of four models on GSM8K. On classification, JSON mode often helped, which the authors attribute to it narrowing the answer space. They also report that parse failures were not the main cause in general and could be fixed with a repair prompt, and that GPT-4o-mini's strict JSON Schema mode came close to free text.
Research notes: the problem, approach and results
GPT-3.5 JSON-mode responses on Last Letter that put the answer key before the reasoning key (Section 4.1)
Claude 3 Haiku on GSM8K, free text against JSON with a schema; 22.9-point spread across wordings (Table 1)
Haiku's GSM8K JSON score after a second call repaired unparseable output (Figure 5)
Production LLM features usually ask for structured output, JSON, XML or YAML, so code can parse the answer. Benchmarks of format following check whether the format is right, not whether asking for a format changes how well the model answers.
Compare four levels of restriction on the same tasks: constrained JSON mode, instructions to answer in a format with a schema, answering in natural language and then converting, and plain natural language. Average over nine prompt wordings and use an LLM to extract answers so parsing does not decide the score.
On reasoning tasks, stricter formats generally scored lower; JSON mode often scored worst, partly because it placed the answer before the reasoning. On classification tasks, JSON mode helped or matched. Answering freely and converting afterwards kept free-text accuracy.
Why it matters to an FDE
Order the schema so reasoning comes first. If a schema lists the answer before the reasoning, a model generating left to right commits to the answer first. Put a reasoning field before the answer field, and check that responses keep that order.
Count parse failures separately from wrong answers. A drop in accuracy after switching to JSON can be the model reasoning worse, or it can be output that does not parse. They have different fixes: prompt changes and repair calls for the second, schema design or a two-step flow for the first.
Consider answering first and formatting second. For reasoning-heavy tasks, a free-text answer followed by a cheap conversion call kept free-text accuracy in this study. It costs a second call, which is often cheaper than the accuracy lost.
Use constrained decoding where it narrows the answer. For classification into fixed labels, JSON mode or an enum in a schema can improve accuracy by ruling out invalid labels.
Evaluate the exact format you will ship. The paper shows results can move by tens of points between format instructions and between prompt wordings. Run your evaluation through the same output contract the product uses, over several wordings.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 290 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
The three restriction methods, results by task type, schema effects, parsing, key order and limits
Free account - 04How it works
The three methods, tasks and models, the schema, prompt variants, the LLM parser and the ablations
Premium - 05The numbers that matter
Table 1 with and without schemas, Table 2 for strict schema mode, Table 3 parse rates and Figure 5
Premium - 06Try it yourself
Put each gap in standard errors, separate parsing from reasoning, and ship a key-order guard
Premium - 07Where it breaks
Small, older models, a two-field schema, an LLM parser, prompt choices and slips
Premium - 08Using it on a deployment
Schema design, a two-step flow for reasoning, enums for labels, validation and repair
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium