GitHub announces ReviewBench for AI code-review agents

GitHub announced ReviewBench as a benchmark for AI code-review agents, describing it as an open, reproducible way to evaluate systems on real-world pull requests.[1][2] The benchmark runs agents against the same 219 pull requests, drawn from 187 public repositories and spanning 19 programming languages.[3] Pull-request sizes are weighted toward the reviewable middle and tail.[3] GitHub’s announcement was authored by Michelle Zhou and Alejandro Carderera de Diego and published on October 5, 2026.[1] ReviewBench is available in research preview through its website.[3]

ReviewBench is designed to measure whether an AI reviewer finds issues while avoiding false alarms.[3] GitHub built its reference set by collecting candidate findings from human reviewers, code changes, analysis tools and AI models, then merging and assessing those candidates.[3] The benchmark uses grounded recall as its main measure for comparing systems.[3] Its F1 score gives precision and recall equal weight, while users can filter results by severity and category and adjust β in the Fβ score.[3] An AI judge assesses whether additional findings are valid.[3]

To register an agent, users sign in with GitHub and provide a container image, configuration and model access key.[3] The submission process offers a test set of 25 pull requests before three full-set evaluation rounds on the 219-pull-request benchmark.[3] Submitted scores remain private until a maintainer approves them.[3] GitHub says the benchmark continues to run if an upstream repository is deleted or rewritten.[2] The ReviewBench repository is released under the MIT license, and the full corpus manifest and golden findings are public.[2]

GitHub uses ReviewBench to evaluate Copilot code-review changes before testing them with users.[3] In a production test, GitHub reported that the share of review comments judged by an AI model to have prompted code changes rose by 8%, while recall rose by 13.6%.[3] GitHub also reported that cost per review fell by 8% relative to the production control.[3] Feedback in the test shifted toward critical and moderate issues, with fewer minor suggestions.[3] These figures describe that production test; the evidence does not establish that the same results will apply to other agents or teams.[3]