NVIDIA SkillEvaluator Review: How to Measure Whether an Agent Skill Actually Helps

SkillEvaluator is NVIDIA's open-source tool for answering the question a security scan cannot: does this agent skill make the agent better? It runs Claude Code, Codex or OpenCode on the same tasks twice, once with the skill and once without, in isolated sandboxes, and reports the difference as Skill Lift. We analyzed NVIDIA's published results for 397 of its verified skills. The average outcome lift is large, about 36 points out of 100, and only 11 skills showed no gain. But 42% of those skills were measured on three tasks or fewer, mostly run once, and that group reports almost double the lift of skills with ten or more tasks. Our verdict: the method is right and worth copying; the catalog numbers are directional, and your own evaluations need more tasks than the defaults suggest.

What SkillEvaluator is

An agent skill is a folder of instructions, examples and scripts that a coding agent loads when a task matches, defined by the open Agent Skills specification. Teams now install dozens, and every one costs something: it sits in the agent's context, competes with other skills for attention, and can send the agent down its workflow when a simpler path existed. SkillEvaluator, released by NVIDIA under Apache 2.0 (version 0.8.0, 6 November 2026; about 850 GitHub stars this week), evaluates a skill in three independent tiers:

Tier 1 is the familiar part: schema checks, a static quality score with a passing threshold of 70, the SkillSpector security scanner, secret detection with Gitleaks, license checks and script linting. Tier 2 uses embeddings to find guidance that repeats inside one skill or overlaps with others in your catalog. Tier 3 is the new idea, and the subject of this review.

How Tier 3 measures a skill

Tier 3 hands each evaluation case to Harbor, an open-source agent evaluation framework, which runs the agent twice in separate sandboxes: once with the skill installed, once without, with the same prompt, model, inputs and grader. The skill is the only variable. Each run is graded on five dimensions:

Here is a real evaluation case, from the example skill shipped in the repository. The arithmetic in its ground truth checks out: the text has 20 words and 97 characters, and "the" appears five times.

{
  "id": "text-analyzer-001",
  "question": "Can you analyze this text and tell me the word count, line count, and most frequent words? Here is the text: 'The quick brown fox jumps over the lazy dog. The dog barked at the fox. The fox ran away quickly.'",
  "expected_skill": "text-analyzer",
  "expected_script": "analyze_text.py",
  "expected_behavior": [
    "The agent read the text-analyzer SKILL.md to understand how to use the skill",
    "The agent executed analyze_text.py with the --text flag passing the provided text",
    "The agent reported the word count (20 words) from the analysis results",
    "The agent reported the most frequent words, noting 'the' appears 5 times",
    "The agent included additional statistics like character count (97) or sentence count (3) in the response"
  ]
}

Look at the first two expected behaviors. Without the skill installed, the agent cannot read its SKILL.md or run its script, so the baseline arm cannot earn those points however good its answer is. NVIDIA is open about this for Discoverability and Efficiency, which it says should be read as "the skill gets used correctly" rather than as unaided agent performance. It is the reason to read Correctness and Effectiveness first, and to write cases whose grading is about the outcome.

create-eval-dataset --full generates explicit, implicit, contextual and negative cases from the skill. Negative cases, prompts the skill should not trigger on, are what catch a skill that loads too eagerly. In template mode without a model, negatives appear only if you write them into EVAL.md yourself.

Cost, worked through. The documentation's estimate of live trials is cases × agents × attempts × arms. Ten cases on two agents with three attempts each is 10 × 2 × 3 × 2 = 120 sandboxed agent runs, each with several model calls plus grader calls. SkillEvaluator does not estimate dollars. --skip-baseline halves the trials while you iterate on the dataset, but cannot produce Skill Lift, so turn the baseline back on for any number you will publish.

What NVIDIA's published results show

NVIDIA publishes per-skill results for its verified-skills catalog in benchmarks.json in the NVIDIA/skills repository. We analyzed the version of 12 November 2026: 451 skills, 432 with results, 4,254 rows across Claude Code and Codex.

Mean lift is large on the two outcome dimensions: Correctness rose from about 56 to 92, Effectiveness from 46 to 79. Security barely moves, which is the intended result: a skill should not make the agent less safe. Two details are easy to miss in the summary tables:

• Efficiency is often negative. 27% of Efficiency results show no gain and about one in five is below zero: with the skill, the agent took more steps. NVIDIA's own blog gives an example at each end: one Jetson skill cut tokens by 76.9%, while a cuOpt install skill increased them by 120.3%.
• The product matters more than the agent. Correctness lift averages 46 points on Claude Code and 27 on Codex, but by product it ranges from about +2 for FastGen to +68 for the TAO Toolkit. NVIDIA reached the same conclusion: the domain and the evaluation design matter more than the harness.

The sample-size problem in the catalog

The catalog's evaluations are small. Of the 432 skills with results, 316 ran one attempt per task. Of the 397 with a recorded task count, 167 (42%) were evaluated on three tasks or fewer, and only 57 on ten or more. Plot each skill's outcome lift against its number of tasks and the pattern is plain:

Skills with three or fewer tasks report a mean outcome lift of +47.2 points with a spread (standard deviation) of 24.7. Skills with ten or more report +27.0 with a spread of 13.8. The middle group, four to nine tasks, sits at +27.9. Either the most useful skills happen to get the smallest evaluations, or small evaluations inflate the number. The second is what you would expect when each task flips a score by a third.

A simulation makes the size of the effect concrete. Take a hypothetical skill with a true lift of 20 points, raising task success from 45% to 65%, and measure it many times with different evaluation sizes:

import random
random.seed(7)

def measured_lift(n_tasks, attempts, p_without=0.45, p_with=0.65):
    """Points of lift measured on n_tasks, each run `attempts` times per arm."""
    with_ = sum(random.random() < p_with for _ in range(n_tasks * attempts))
    without = sum(random.random() < p_without for _ in range(n_tasks * attempts))
    return 100 * (with_ - without) / (n_tasks * attempts)

for n, a in [(3, 1), (4, 1), (5, 1), (10, 1), (10, 3), (20, 3), (40, 3)]:
    runs = sorted(measured_lift(n, a) for _ in range(20_000))
    q = lambda f: runs[int(f * (len(runs) - 1))]
    zero = sum(x <= 0 for x in runs) / len(runs)
    print(n, a, q(.05), q(.5), q(.95), f"{zero:.1%}")

With three tasks run once, the middle 90% of measured lifts runs from −33 to +67 points, and the skill reads as no improvement 46% of the time. Twenty tasks with three attempts narrow that to +5 to +35, and the false "no gain" rate drops to 1.5%. NVIDIA's scorer uses graded rubrics rather than pass or fail, which smooths things a little, but the lesson carries over. NVIDIA's blog notes the same limit: most skills ran a single attempt, and it reports no confidence intervals for the catalog.

The research paper behind the method, "Evaluating Skills, Not Just Agents" (arXiv 2608.20614, August 2026), is more careful. On 947 scored paired cases from 58 of 64 production skills and four harnesses, mean composite Skill Lift was 0.2134 on a 0 to 1 scale, with a 95% confidence interval of 0.197 to 0.230, and positive in 72.8% of paired cases. The outcome-only lift, averaging accuracy and goal accuracy, was 0.18. That is still a solid gain, and a more honest picture: roughly one case in four did not benefit from the skill. The paper also found that static structural scores and LLM-judge rubric scores barely agree (Spearman correlation 0.14). A tidy SKILL.md and a useful skill are different things.

How to evaluate your own skills

1. Write the cases before you write the skill. NVIDIA's own first lesson: teams that define tasks, expected outputs and out-of-scope requests get sharper results. Generated cases are a start; review every one.
2. Grade outcomes, not usage. Keep behavior checks such as "ran the script" for Discoverability. Decide the go or no-go on whether the answer was right and the goal was met.
3. Use enough tasks. At least 10 reviewed cases, including negatives, with three attempts per case for anything you will act on. Report the spread, not just the mean.
4. Track tokens and time separately. A skill can raise correctness and double cost. That might be worth it; decide on purpose.
5. Re-run on model upgrades. A skill that helped one model can be redundant for the next, which already knows the workflow. Keep the cases as a regression suite.
6. Delete what does not earn its place. A skill with no measurable lift on a decent evaluation still costs context and attention. Remove it.

Pros and cons

The interview angle

"How would you know if this agent customization actually helped?" is now a standard agent design follow-up, whether the customization is a skill, a prompt, a tool or a plugin. The strong answer is a paired experiment: same tasks, same model, same grader, with and without the change, enough cases to see past the noise, and outcome metrics kept separate from usage metrics. It is the same discipline behind agentic evals, applied to the unit an FDE ships most often. Candidates who can explain why three tasks run once proves almost nothing are showing exactly the judgment the round tests.

Figures as of 12 November 2026: SkillEvaluator 0.8.0, NVIDIA/skills benchmarks.json at commit d5980550, NVIDIA Technical Blog post of 19 August 2026, arXiv 2608.20614. We did not run SkillEvaluator ourselves; the analysis uses NVIDIA's published results and documentation.