Sierra Open-Sources Hyper-τ-Bench to Evaluate AI Agents Building Agents

Sierra announced on September 8, 2026, that it is open-sourcing hyper-τ-bench to score whether AI coding agents can build a functional customer-service agent. [1] The strongest automated configuration achieved a 23.9% pass rate on held-out evaluation tasks, while a reference pairing of an engineer and a frontier model scored 82.2%. [1] The company stated that while its 2024 τ-bench tested whether a model could act as an agent, the new benchmark evaluates who builds the agent, work that is increasingly handled by models themselves. [1]

The benchmark, formally published as τ^τ-bench, is accompanied by a 41-page paper submitted to arXiv on September 4, 2026, and a public leaderboard. [1] The authors note that existing benchmarks fail to indicate whether an AI system can deliver an agent under real client engagement conditions. [1] Sierra described building these agents in practice as research involving scattered requirements across handbooks, support channels, spreadsheets, and the knowledge of frontline representatives. [1] The codebase is available to the public under an MIT license. [1]

The release of hyper-τ-bench shifts the evaluation frontier from an AI's ability to converse with customers—which Sierra now calls table stakes—to its ability to engineer complex software systems. The massive gap between automated configurations (23.9%) and human-assisted baselines (82.2%) highlights that autonomous agent construction remains a difficult challenge. What the sources don't address: how quickly automated configurations might close the gap with human engineers on these held-out tasks as frontier models evolve.