NVIDIA has published SkillEvaluator, an Apache-2.0 framework that checks AI agent skills and then measures whether they actually change what an agent does. The repo is explicit about the two-step: run quality checks first, then compare runs with and without the skill. That second half is the part worth paying attention to. Most skill tooling stops at “does this folder parse,” and SkillEvaluator’s authors clearly think that is not enough.

The tool is organized into three independent tiers, each answering a different question. Tier 1 asks whether a skill is safe, well-formed, and harness-compatible. Tier 2 asks whether the guidance repeats or overlaps, using embeddings and chat. Tier 3 asks the question that matters most: does the skill help the agent? Tier 3 requires provider access, a supported agent, and a running sandbox, Docker by default, and it runs on Harbor, the open-source agent evaluation framework, for sandboxed agent runs.

What is actually new here

The headline mechanism is the before-and-after comparison. SkillEvaluator’s live evaluation measures how skills affect agent behavior by running the agent with and without the skill and comparing results. That is a behavioral test, not a documentation lint. It is the difference between “your SKILL.md has a valid frontmatter block” and “your SKILL.md changes the model’s output in a measurable direction.”

The static side is concrete and testable. The bundled text-analyzer reference skill scores 86.2/100 (a B) on the keyless quality-check path, exits 0, and writes reports/first-quality/skillevaluator-quality.json. The passing threshold defaults to 70. No API key, scanner, GPU, or Docker is needed for that run, which is a deliberate on-ramp: you can see a number before you commit to any infrastructure. The repo is honest that static quality is not the whole story. Publication evidence stays INCOMPLETE, security needs scanners, and agent impact needs live evaluation.

Tier 1 fetches pinned, verified SkillSpector and Gitleaks releases when they are missing, with Semgrep optional. SkillSpector is the specialized skill security scanner; the combination of a security scanner and a lint pass inside one CLI is the sort of thing that usually lives in three separate GitHub Actions. --no-llm does not remove scanner requirements, and a scanner that cannot run is marked INCOMPLETE and exits nonzero. That is the right default. A gate that silently passes when its scanner is missing is not a gate.

Tier 2 is the sleeper. Semantic overlap detection across skills, and inter-skill comparison when you pass --catalog ./skill-catalog.json, addresses a real failure mode: skill libraries that accumulate near-duplicate instructions until the agent is being told the same thing four ways. The repo notes that Tier 2 reports when inter-skill comparison did not run, so the absence of a result is itself reported rather than hidden.

The synthetic dataset question

Tier 3 ships create-eval-dataset, which generates synthetic evaluation datasets. The docs are careful about the limits. Standalone generation needs substantive evals/EVAL.md guidance, a --prompt file with task requests and outcomes, or matching retained traces. --force replaces the autopilot starter, --refine refines it, and explicit --grounding skill permits skill-only starters. The repo tells you to review cases and notes that templates omit unauthored negatives.

That last clause is the honest one. A synthetic eval set built from a skill’s own documentation will test whether the agent follows the skill, not whether the skill is correct. The autopilot “reuses sources or creates one skill-grounded case with limited coverage,” per the tier guide. Coverage is the thing synthetic generation is worst at, and NVIDIA says so in the README rather than in a footnote.

Read the support label

SkillEvaluator is marked Experimental: community-supported, best-effort, no SLA, no NVIDIA enterprise support entitlement. That is a meaningful signal for anyone wiring it into a release process. The CI path exists, skillevaluator ci init opens a setup PR on GitHub and there is a GitLab template, but an experimental project’s CI integration is a different bet than a supported one. The pinned-scanner behavior helps here, since the tool fetches verified releases rather than trusting whatever is on the runner.

The repo also carries the standard warning that it will download and install additional third-party open source software, and that you should review those licenses. Apache 2.0 covers SkillEvaluator itself, with NOTICE and THIRD_PARTY_NOTICES.md alongside.

Most skill tooling stops at “does this folder parse.” SkillEvaluator’s authors clearly think that is not enough.

What this means for builders

The skill format is converging on a folder of instructions plus supporting files, per the Agent Skills specification, and NVIDIA is positioning SkillEvaluator inside a “Verified Skills pipeline.” A vendor-backed verification pipeline for a format that other vendors also consume is a governance question as much as a tooling one. If “verified” becomes a badge, the criteria behind it matter more than the CLI ergonomics.

For now the practical read is narrower. If you maintain agent skills, Tier 1 is cheap to adopt and catches the boring failures: malformed skills, missing security scans, harness incompatibility. Tier 3 is where the interesting work is, and it is also where the setup cost lives: provider access, a supported agent, a running sandbox, and model and managed-sandbox usage that “can incur charges.” The doctor --env-mode docker command checks readiness before you run the full validate, which is the right order.

The open question is whether behavioral evaluation of skills becomes standard practice or stays a premium step that most teams skip. SkillEvaluator makes the before-and-after comparison a first-class command. Whether anyone runs it on every pull request, given the sandbox and provider requirements, is the part the framework cannot answer for you.