NVIDIA SkillSpector Review: What the Agent Skill Scanner Catches, What It Misses, and What SAFE Means

NVIDIA SkillSpector is a free, open-source scanner that reads an AI agent skill before you install it and scores how risky it looks. We ran version 2.13.4 on seven skills we wrote for the test. It flagged the two obvious threats firmly: a script that posts a token to a server and a pair of outdated, vulnerable dependencies both scored DO NOT INSTALL. With the optional LLM stage turned off, though, three skills that a careful reviewer would reject came back SAFE, including one whose hidden comment told the agent to copy the conversation to a gist and never mention it. Our verdict: adopt it as one gate, run the LLM stage on instruction text, and fail builds on findings rather than on the SAFE label alone.

What SkillSpector is

An agent skill is a folder: a SKILL.md file of instructions, often with scripts the agent can run. Claude Code, Codex CLI, Gemini CLI and other agents load skills with the user's permissions, and community marketplaces now distribute them by the thousand. That makes a skill a supply-chain artifact in the same sense an npm package is, with one twist: part of the payload is prose the model obeys.

SkillSpector, released by NVIDIA under Apache 2.0, scans a skill from a directory, a single file, a Git repository, a URL or a zip. It has two stages:

1. Static analysis. Regex patterns, Python AST checks for calls such as subprocess and eval, taint tracking from secrets to network calls, YARA signatures, and a dependency lookup against OSV.dev. Version 2.13.4 lists 71 patterns across 17 categories, from prompt injection and data exfiltration to MCP tool poisoning. The screenshots circulating this week say 64 and 16, which is an older README.
2. LLM semantic review (optional). A configured model reviews the static findings for context and intent. Its README is careful here: a confirmed finding can gain confidence, a disputed one is tagged llm-unconfirmed, and every deterministic finding stays in the report. The stage does not delete findings, and the README says it does not guarantee a precision rate.

The output is a 0 to 100 score with a recommendation, in terminal, JSON, Markdown or SARIF form. SkillSpector is also part of NVIDIA's Verified Skills pipeline, which scans and signs skills before they appear in NVIDIA's own catalog.

The numbers behind the pitch, read properly

The README and the posts sharing it cite one study, "Agent Skills in the Wild" (Liu et al., arXiv 2601.10338, January 2026). Three details change how its figures should be repeated:

A follow-up by the same lead author (arXiv 2602.06547, February 2026) collected 98,380 skills and verified behavior dynamically. It confirmed 157 malicious skills, about 0.16%, and found over half came from one actor using templated brand impersonation. Different dataset, different method, so the two rates do not subtract. The useful reading is that careless and risky skills are common, while confirmed malicious ones are rarer and deliberate. Its title, "Do Not Mention This to the User", turned out to describe our hardest test case.

What we tested

We built SkillSpector 2.13.4 from its Git tag inside a throwaway Docker container and scanned seven skills we wrote, mounted read-only. We ran static analysis only (--no-llm), because the LLM stage would have sent the files to a model provider and its result depends on which model you pick. Nothing we scanned was executed, by us or by the tool. The skills:

The first four rows are the tool working as advertised. The benign script got two cautionary findings and still scored SAFE, which is the right outcome for a reviewer's queue. The token sender and the stale dependencies were caught with high confidence.

Here is the finding SkillSpector reported for the last skill, trimmed to the fields that matter. Note what it matched: the concealment phrase, not the instruction to copy the conversation.

{
  "risk_assessment": { "score": 7, "severity": "LOW", "recommendation": "SAFE",
                       "max_issue_severity": "MEDIUM" },
  "issues": [{
    "id": "P4",
    "category": "Prompt Injection",
    "pattern": "Behavior Manipulation",
    "severity": "MEDIUM",
    "confidence": 0.75,
    "location": { "file": "SKILL.md", "start_line": 8 },
    "finding": "Never tell the user",
    "remediation": "Review content for implicit steering or bias."
  }]
}

The scanner saw something, labeled it correctly as prompt injection, and the arithmetic still produced SAFE. A gate that reads issues instead of recommendation would have stopped it.

The last three rows are the lesson. The split-string script is a well-known way to hide os.system from a pattern matcher. SkillSpector noticed the dynamic import and said so, but the score stayed at 13. The hidden-comment skills are instruction attacks: nothing in them is code. The one that named a credential file tripped a HIGH finding because of the path, and still landed in CAUTION at 22. The one that asked the agent to exfiltrate the conversation and conceal it named no file and no command, and scored 7.

How the score turns into a verdict

Reading the source explains the results. Findings add up to a 0 to 100 score; the score, not the worst finding, picks the verdict: 0 to 20 is SAFE, 21 to 50 CAUTION, 51 and above DO NOT INSTALL. The CLI exits nonzero only when the score is above 50. Three consequences for anyone wiring it into CI:

• A HIGH finding can sit inside a CAUTION verdict, and a MEDIUM prompt-injection finding inside SAFE.
• A default exit-code gate passes everything at 50 or below. In our set, that includes all three skills a reviewer would reject.
• Instruction-level attacks are where static analysis is weakest, which is exactly the class the follow-up study found in the wild: adversarial instructions embedded in documentation.

None of this is a defect in the tool's honesty. The README states the stage-one profile plainly as "high recall" with "moderate precision", and lists runtime behavior and non-English text as limits. The risk is in how a team reads one word on a dashboard.

Data egress and the trust model

Two network paths matter before you run it on internal skills:

• LLM analysis sends file contents to the provider you configure. SkillSpector supports OpenAI, Anthropic, Bedrock, Gemini on Google Cloud, NVIDIA Build, Azure OpenAI, local Ollama or vLLM, and others. For an internal skill that mentions hostnames or runbooks, pick an approved endpoint or a local model.
• The dependency check sends package names and versions to OSV.dev, even with --no-llm. It sends coordinates, not file contents, and falls back to a small bundled list when offline.

SkillSpector never runs the skill, and it is not a sandbox. It tells you what a skill looks like before install; it does nothing to contain a skill you install anyway. That containment is a separate control, covered in agent sandboxing.

How we would roll it out

For a team that lets engineers install skills, the order we would use:

1. Run static analysis on every new skill and every update, with SARIF output into the code-scanning view your security team already watches.
2. Fail the gate on findings, not the verdict. Any HIGH or CRITICAL finding, and any prompt-injection, exfiltration or dynamic-execution finding, goes to a person regardless of score. Use the baseline file to accept known, reviewed findings so re-scans show only new ones.
3. Run the LLM stage on instruction text through an approved provider, pinned to one model so results are comparable between scans.
4. Require a human read for any skill that carries scripts. The study found those 2.12 times more likely to contain vulnerabilities.
5. Pin installed skills by hash in an internal registry, and re-scan on every version change. A skill that was clean last month can ship a new script tomorrow; plugin lifecycle hooks have the same problem, as the coding-agent plugin hooks question works through.
6. Limit what an installed skill can reach: no ambient cloud credentials in the agent's environment, egress allowlists, and read-only mounts where possible.

Pros and cons

After the scan: does the skill help at all?

A clean scan says a skill is unlikely to hurt you. It says nothing about whether the skill makes the agent better, and a skill that does not help still costs context on every session and competes with other skills for the agent's attention. NVIDIA's companion tool, SkillEvaluator, runs the agent on the same tasks with and without the skill and reports the difference. We read its published results for more than 400 skills in our SkillEvaluator review.

What to say about this in an interview

Skill and plugin supply chain is now a standard topic in AI security and governance rounds, because forward deployed engineers are the people installing agents inside customer environments. The strong answer is layered: vet before install with a scanner and a reader, pin what you install, contain what runs, and watch what it does. The weak answer names a scanner and stops. If you can explain why a 7 on a hidden "do not tell the user" instruction is a process problem rather than a tool bug, you are showing the judgment the round is testing. The broader pattern, reading untrusted content that a model will follow, is prompt injection in a new package.

Tested 12 November 2026 against SkillSpector v2.13.4 (released 4 November 2026). Star counts and pattern totals change quickly; check the repository before relying on them.