IN THIS BREAKDOWN · 10 SECTIONS
You can read 16% of this breakdown.
The paper in one minute
A context file is a README written for a coding agent. Called AGENTS.md, CLAUDE.md or similar, it sits at the root of a repository and is loaded into the agent's context at the start of every session. Typical files describe the repository's layout, how to install and test it, and conventions to follow. Agent makers recommend them, several tools generate one with a single /init command, and the AGENTS.md project counts over 60,000 public repositories that include one.
Thibaud Gloaguen, Martin Vechev and colleagues at ETH Zurich and LogicStar.ai asked whether these files actually help agents complete tasks. Existing benchmarks could not answer that, since their repositories predate context files. So the authors built CTXBench: 138 Python tasks, bug fixes and features, drawn from 12 smaller repositories whose developers had committed their own context files. They also used the 300-task SWE-bench Lite set, generating context files there with each agent's own recommended command.
Four agents were tested: Claude Code with Sonnet 4.5, Codex with GPT-5.2 and with GPT-5.1 mini, and Qwen Code with Qwen3-30B-Coder. Each ran every task with no file, with an LLM-generated file, and, on CTXBench, with the developers' file. Context files did not significantly change success rates. They did increase the number of steps and the cost, because agents dutifully followed the extra instructions: running more tests, searching more files, and using the tools the file named.
Research notes: the problem, approach and results
points of average success from developer-written files on CTXBench, not significant (p = 0.21)
average cost per task on SWE-bench Lite with an LLM-generated context file, significant
uses of uv per task when the file mentioned it versus when it did not: the instructions were followed
Agent vendors recommend an AGENTS.md or CLAUDE.md file in every repository, and over 60,000 public repositories have one. Nobody had measured whether these files help agents finish real tasks.
Run four coding agents on 300 SWE-bench Lite tasks and on a new benchmark of 138 tasks from 12 repositories that already ship developer-written context files, with no file, an LLM-generated file, or the developers' own file.
LLM-generated files lowered average success slightly; developer-written files raised it by 2.4 points, neither significantly. Both raised steps and cost. Agents obeyed instructions like 'use uv'; repository overviews did not help them find the right files.
Why it matters to an FDE
Rolling out coding agents at a customer usually starts with someone running /init and committing the result. This paper says that file is likely costing money without improving outcomes, and it tells you what kind of content is worth keeping.
Do not ship an auto-generated context file by default. LLM-generated files lowered success slightly on average and raised cost by about a fifth. The authors recommend omitting them for now.
Write instructions, not tours. Agents followed specific instructions closely: when a file said to use a tool, agents used it. Repository overviews, the most common content, did not help agents reach the right files any sooner. Keep the file to what an agent cannot discover: non-standard commands, required conventions, things to never do.
Every line has a cost. The file is sent with every model call, and each instruction it contains can trigger extra work. Treat it as configuration with a budget, not documentation.
Measure before you standardize. If a team wants a context file template across repositories, test it on real tasks from those repositories, with and without the file, and compare success and cost.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 99 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦3 results tables
- ?5 interview questions with answers
- 03Key takeaways
How the benchmark was built, each finding with its evidence, and the misreadings to avoid
Free account - 04How it works
Building CTXBench, the three settings, the agents, and the statistics
Premium - 05The numbers that matter
Steps, cost, significance, and what the trace analysis showed
Premium - 06Try it yourself
Split the paper's cost increase into extra steps and dearer steps, then price the file itself
Premium - 07Where it breaks
What the benchmarks cannot show, and where the findings may not hold
Premium - 08Using it on a deployment
Writing, testing and budgeting a context file for a customer's repositories
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium