NVIDIA’s Framework for Evaluating AI Agents Beyond Tool Calls

Standard LLM evaluation harnesses were designed for static tasks, but agents operate across multi-step work by calling tools, handling errors, and observing results over many steps. [1] The Berkeley Function-Calling Leaderboard evaluates function selection and argument accuracy in single- and multi-turn scenarios. [1] NVIDIA says that call-level accuracy is necessary but insufficient because a valid refund call can still leave a task incomplete when required checks or updates are skipped. [1]

NVIDIA says full agent evaluation requires an execution environment that runs each tool call, tracks state across steps, and inspects the resulting world to determine whether the task was completed. [1] Step-level process scoring asks whether a call was valid, relevant, and useful given the state at that moment. [1] End-to-end outcome scoring instead checks the final state, such as whether a refund posted or a ticket was routed correctly, without judging the path taken. [1]

The practical distinction is between testing an agent’s vocabulary of actions and testing whether those actions produce the intended state. BFCL can reveal whether function selection and arguments are correct, while an execution environment exposes failures caused by omitted checks, updates, or error handling across steps. Teams therefore need both diagnostic views: process scoring to locate a bad decision and outcome scoring to verify task completion. What the sources don't address: how teams should weight process and outcome scores when they disagree or set pass thresholds for production deployment.