IBM Research Unveils Tool to Halve the 24-Point AI Agent 'Consistency Gap'

Most benchmark scores obscure the variability of AI agents by relying on averages instead of repeated reliability. [1]

On the AppWorld benchmark, a ReAct agent powered by GPT-4.1 achieved a success rate of 77.4 percent across five runs. [1] However, the same agent completed all five runs successfully for only 53.0 percent of the tasks, revealing a 24.4-point consistency gap. [1] This gap increases to 30 points for difficult tasks, highlighting a critical reliability issue for production workflows. [1]

To address this unreliability, IBM Research introduced consistency guidelines within the `altk-evolve` system, powered by a new Consistency Analyzer diagnostic tool. [1] The analyzer identifies "flip-prone" decision points by resampling an agent's recorded trajectory to see where it was one token-sample away from taking a different action. [1]

This method requires no ground truth and operates by resampling decision points with a single call requesting five completions, avoiding end-to-end task reruns. [1] Using these guidelines reduced the consistency gap by half, dropping it from 24.4 points to 12.0 points without decreasing average accuracy. [1]

By shifting the focus from average accuracy to repeated reliability, this diagnostic tool highlights a fundamental flaw in how the industry currently evaluates AI agents. The 24.4-point gap demonstrates that an agent's isolated success on a benchmark does not guarantee stability in a production environment. Unlike standard testing that requires complete end-to-end reruns, this approach efficiently isolates exact points of failure at the token-sample level. What the sources don't address: whether this diagnostic tool requires significant compute overhead when scaling beyond default k=5 completion requests for complex enterprise workflows.