Amazon Science Highlights Need to Discount Correlated LLM Judges

An article from Amazon Science questions whether we should believe large language model judges when they agree. [1] Panels of LLM judges are intended to reflect a true diversity of perspectives. [1] However, some LLM judges produce highly correlated outputs. [1] Discounting the opinions of these judges ensures the desired diversity of perspectives. [1]

This approach highlights a fundamental challenge in automated model evaluation: the risk of an artificial consensus or "echo chamber" when multiple evaluating models share similar underlying architectures or training data. By actively identifying and filtering out highly correlated outputs, AI practitioners can build more robust and genuinely independent evaluation panels, ensuring that consensus reflects actual quality rather than mirrored biases. What the sources don't address: the specific statistical thresholds or methods Amazon uses to determine when LLM outputs cross the line into being "highly correlated."