NVIDIA Launches Open Source SkillEvaluator to Measure AI Agent Performance

NVIDIA SkillEvaluator is an open source tool for measuring how skills affect agent performance through static checks and real-world task runs with and without each skill. [1] NVIDIA verified Skills are packaged, signed capability descriptors that tell an agent exactly what an NVIDIA product does, when to invoke it, and how to call it. [1] The verified part is the measurement that determines it is ready. [1]

NVIDIA shared the first benchmark results for more than 300 verified skills across over 30 NVIDIA products. [1] Each skill was evaluated on two independent harnesses. [1] For each harness, Skill Lift was calculated by comparing scores from runs with and without the skill installed. [1]

Before a skill is published, it goes through three tiers of evaluation. [1] Tier 1 focuses on safety and structure, running static checks for schema and frontmatter validation, quality scoring, and security scanning for prompt injection and data exfiltration. [1] It also includes secret and PII detection, license checks, and script linting. [1] Tier 2 evaluates distinctiveness. [1]

NVIDIA's release of SkillEvaluator addresses a crucial bottleneck in AI agent deployment: verifying that agents use their tools effectively and safely. By open-sourcing the evaluation layer and providing a standardized way to measure "Skill Lift," NVIDIA is attempting to formalize how agent capabilities are benchmarked across its ecosystem. This move provides developers with concrete metrics on how adding specific instructions and tool guidance improves agent trajectories, rather than relying on trial and error. The structured evaluation tiers, particularly the focus on safety and data exfiltration, highlight the growing need for robust security protocols before deploying agents in production environments. What the sources don't address: How these verified skills perform across different underlying foundation models when subjected to the SkillEvaluator harnesses.