SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks

An Agent Skill is a folder an agent can discover and load while it works: a required SKILL.md file of instructions, plus optional scripts, templates and reference files. Anthropic introduced the format, and harnesses such as Claude Code, Codex CLI, Gemini CLI and OpenHands support it. A Skill sits between a prompt and a tool. It carries procedure ("how we do X here") and can ship executable code, without changing the model. By the authors' snapshot, community collections held about 2 million distinct Skills. Nobody had measured whether they help.

SkillsBench, a large community effort led by BenchFlow, measures it with paired runs. The authors gathered 400 task submissions from 142 contributors and kept 87 after automated checks and human review. Each task is real work in a container, such as security analysis, scientific signal processing, media conversion, finance or software engineering, with a human-written instruction, a reference solution and a deterministic test suite. Each comes with Skills written by domain experts independently of the task, carrying general expertise for that kind of problem, never the answer. The instruction never mentions the Skills; the agent has to find them.

Each task ran three times without Skills and three times with them, under 18 combinations of model and harness. Curated Skills raised the average pass rate from 33.9% to 50.5%. Every setup improved, from +4.1 to +25.7 points. Gains were biggest in domains models know least, such as natural science (+28.8) and media production (+24.1), and smallest in mathematics (+9.7) and software engineering (+11.6). One to three Skills per task worked better than four or more, and compact Skills beat comprehensive documentation, which added almost nothing. When agents wrote their own Skills before solving, results got worse.

Skills are a practical way to put a customer's procedures into an agent without fine-tuning: how they format reports, which internal API to call in what order, the quirks of their data. This paper is the first measurement of what works.

Write procedures, not manuals. The winning Skills were short and specific: executable scripts with sensible defaults, the authoritative data source and its parsing quirks, exact output-format constraints, invariants the answer must satisfy. Comprehensive documentation gained 0.7 points against about 19 to 21 for focused Skills. Put in what the agent cannot work out for itself and leave out the rest.

Fewer Skills per task. Tasks with four or more Skills gained about half as much as those with one to three. Bundling every Skill a team has written into every agent dilutes the guidance and can conflict.

Test every Skill with and without. Thirteen of 87 tasks did worse with Skills on average, typically because a Skill prescribed a heavy pipeline that displaced the agent's simpler default approach. Run the customer's tasks both ways before shipping a Skill library.

The harness matters. The same model gained different amounts in different harnesses: Claude Opus 4.7 reached 61.2% in Claude Code and 53.1% in OpenHands. Evaluate Skills in the harness the customer will actually run.

Do not let the agent write its own. Skills an agent generated for itself lowered scores by 8 to 12 points. A human expert's procedure is the asset.