Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.
★ 2,090⑂ 286👁 3⑃ 509● 97▤ 11
+804
Daily
2,090
stars
286
forks
3
watchers
509
prs
97
issues
11
releases
README
SkillEvaluator !SkillEvaluator wordmark 📺 Livestream: Ask the Experts: Evaluating Agent Skills SkillEvaluator checks AI agent skills and measures their effect on task results: start with quality checks, then compare runs with and without a skill. Agent skills are folders of instructions and supporting files defined by the Agent Skills specification. SkillEvaluator is part of the NVIDIA Verified Skills pipeline. Get started Three ways in, one CLI, same results. 1. Run the CLI directly 2. Add it to your repo via CI 3. Wire it to your agent Install (or uv tool install): bash pip install skillevaluator Run all Tier 1 checks, no API key: bash skillevaluator validate \ ./my-skill --tiers 1 Or run from a clone: Quickstart. Check changed skills on every pull request. On GitHub, run: bash skillevaluator ci init It opens a setup PR. On GitLab, copy the CI template. Add the SkillEvaluator skill to Claude Code, Codex or any skill-aware agent: bash npx skills add \ NVIDIA/SkillEvaluator Then ask: "Is my skill any good?" No Node.js? Copy the folder. Guides: installation, CI setup (GitHub, GitLab, report examples) and agent skill. Quickstart Both paths use bundled text-analyzer. | Start here | Requires | First result | | --- | --- | --- | | Run without an LLM | Git and uv | A deterministic quality score and findings | | Use NVIDIA Build | Git, uv, and a Build API key | Model-backed rubric scores and feedback | Install uv and Git, then get the example: bash git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git cd SkillEvaluator uv provisions Python 3.13 and dependencies. For standalone installation, see Installation. Run without an LLM bash uv run --python 3.13 skillevaluator quality-check \ src/skillevaluator/tier3/reference_skills/text-analyzer \ -r cli -r json -o reports/first-quality No API key, scanner, GPU or Docker is needed. The example scores 86.2/100 (B), exits 0 and saves reports/first-quality/skillevaluator-quality.json. The passing threshold defaults to 70. This checks static quality; publication evidence stays INCOMPLETE. Security needs scanners; agent impact needs live evaluation. Replace the example path with your skill directory. See the keyless walkthrough. Use NVIDIA Build Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for free prototyping, subject to access terms and rate limits. 1. Sign in at build.nvidia.com and complete enrollment. 2. Open a model page, select Get API Key, and generate your key. 3. Set the provider and key in the same terminal, then run the rubric check: bash export SKILL_EVAL_LLM_PROVIDER=nv_build export NVIDIA_API_KEY='nvapi-...' uv run --python 3.13 --extra llm skillevaluator rubric-eval \ src/skillevaluator/tier3/reference_skills/text-analyzer \ -r cli -r json -o reports/first-build The command sends skill documentation to the model and saves feedback in reports/first-build/skillevaluator-rubric.json. N
Top issues
#462[FEA]: Recognize Node scripts (.mjs, .js) for skill type and the script checks5 comments · 2026-11-12#510[BUG]: `nv_build` with no `llm-model` returns HTTP 404 for our NVIDIA Build key, so `RUBRIC_EVAL` and the LLM security verification do not run3 comments · 2026-11-13#461[FEA]: Name the Tier 1 token count an estimate, and offer a tokenizer or a set ratio3 comments · 2026-11-12#495[BUG]: Tier 1 security on a plugin scans only its bundled skills; hooks and bin/ are never scanned and skill references to them stay INCOMPLETE2 comments · 2026-11-13#42ux: add consolidated Tier 1 scanner preflight and guided bootstrap2 comments · 2026-08-05#508[FEA]: Letter-grade quality score for plugins, hooks and mods1 comments · 2026-11-13#329[FEA]: Move the rest of Tier 1 configs out of code and into YAML files1 comments · 2026-11-03#41roadmap: scale Tier 2 overlap detection for registry-sized catalogs1 comments · 2026-08-05
Discussions
#490Field note: what the Tier 1 Discoverability score reads, and a question about Tier 30 comments · 2026-11-13#205Research collaboration inquiry: LLM judge calibration for Skill Lift0 comments · 2026-10-16#114Support GitHub Copilot CLI as a Tier 3 evaluation agent0 comments · 2026-08-28
Latest release
SkillEvaluator 0.9.0 v0.9.0 2026-11-13SkillEvaluator 0.9.0 SkillEvaluator 0.9.0 expands how you build evaluation datasets, compare skill versions, and understand why an agent passed or failed. It adds controls for evaluation budgets, reusable baselines, and clearer evidence across terminal, HTML, JSON, Markdown, and benchmark reports. For PyPI users upgrading from 0.7.2, this release also includes the changes shipped in the GitHub-only 0.8.0 release. Highlights - Build and review better evaluation datasets. Declare deterministic checks for script execution, workspace files, and final responses. Generate datasets grounded in s