ContextBench: A Benchmark for Context Retrieval in Coding Agents

Coding agents are judged by whether their patch passes the tests. That hides how they got there. An agent may stumble on a fix after reading the wrong files, or read exactly the right function and still write a broken patch. Han Li, He Ye, Zhaoyang Chu and colleagues at Nanjing University and University College London built ContextBench to score the middle of the process: what code the agent retrieved.

They pooled 4,497 tasks from SWE-bench Verified, Multi-SWE-bench, SWE-PolyBench and SWE-bench Pro. After deduplication and a difficulty filter (tasks few agents solve, with patches touching many, scattered files), 1,136 tasks from 66 repositories in eight languages remained. For each one, expert annotators traced the code the reference fix depends on (calls, inheritance, data flow) into a compact gold context. A gold context counts as sufficient if GPT-5, given only that code, produces a test-passing patch in at least one of five tries; failures get up to two more annotation rounds.

At evaluation time, every file and line range the agent reads is logged, and before submitting, the agent must declare the code it used. Both are mapped with tree-sitter to files, definition-level blocks (functions, classes) and lines, and compared with gold: recall, precision and F1, plus how early gold is found, how often the agent re-reads, and how much gold it saw but left out of its declaration.

The authors report three headline findings. Elaborate scaffolds did not retrieve better than a bash-only baseline, which they call coding agents' Bitter Lesson. Models favored recall over precision. And agents often saw the right code and then dropped it.

When a customer's coding agent fails on their monorepo, the first question is whether it found the right code. ContextBench gives that question a method.

Separate finding from fixing. Log every read, compare with the code a human says matters, and you know whether to work on search (indexes, tools, repository maps) or on reasoning and patch generation.

Watch the gap between seen and used. Two models in this paper lost over 40% of relevant code they had already read. That points to context management, not search.

Retrieval scores are not outcomes. The best-retrieving scaffold was fourth of five on solve rate. Use retrieval metrics to diagnose a given agent, and keep the end-to-end test suite as the measure of success.

Building gold contexts is expensive. About 40 minutes of expert time per issue. For a customer, 30 to 50 annotated issues from their own repository is a realistic diagnostic set.