Databricks details RADAR system to catch silent 'gray failures' via anomaly detection

Gray failures are partial outages that slip past standard green dashboards, quietly costing organizations customers and revenue before being noticed. [1] To catch these issues early, Databricks uses a system called RADAR. [1] RADAR operates as a four-stage, metric-agnostic pattern that includes reliability metrics, anomaly detection, alerting, and root-cause analysis. [1] Databricks runs this system on itself, discovering failures 95% faster and achieving over 90% precision. [1]

Engineers can build the same system on Databricks for any metric, such as billing, conversion, or model performance. [1] The architecture relies on native components and an AI-agent scaffold to handle detection and alerting. [1]

RADAR highlights a shift from traditional infrastructure monitoring toward metric-agnostic anomaly detection for specific business workflows like billing or model performance. By utilizing native Databricks components and an AI-agent scaffold, reliability teams can proactively catch issues that typically rely on delayed customer support tickets to surface. While traditional monitoring might show completely healthy servers, this targeted approach uses AI agents to flag the partial failures that slip through. What the sources don't address: the specific computational overhead or cost associated with running continuous anomaly detection across high-cardinality business metrics.