Microsoft reports benchmark results for diagnostic AI system

Microsoft calls the system the Microsoft AI Diagnostic Orchestrator, or MAI-DxO.[1] It simulates a panel of virtual physicians that propose diagnoses, choose questions or tests, and challenge the panel’s emerging view.[1] The system coordinates several rounds of debate, updating its assessment as evidence arrives.[1] The study introduced the Sequential Diagnosis Benchmark, or SDBench, to test step-by-step diagnostic reasoning.[1] This makes the results a test of a particular diagnostic process, rather than a report of outcomes from patients receiving care.[1][2]

SDBench included 304 complex cases from New England Journal of Medicine case proceedings and 325 emergency-department-to-admission encounters from MIMIC-IV-ED.[2] It measured diagnostic accuracy and the estimated cumulative cost of tests ordered.[2] On the complex NEJM cases, MAI-DxO achieved 83.2% accuracy; physicians answering those cases without reference materials averaged 22.3%.[2] On emergency-department cases, MAI-DxO scored 80.3%, compared with 46.8% for generalist physicians.[2] The main MAI-DxO results used OpenAI’s GPT-5.[3]

The study’s cost estimates used CPT-based pricing derived from a major health system’s chargemaster, so they are estimates rather than a universal price list.[2] Without budget limits, MAI-DxO averaged $5,431 per complex case.[3] A configuration achieved 80.9% accuracy at an average cost of $2,241, illustrating a different accuracy-cost point in the reported results.[2] With GPT-5, the system asked an average of 25.5 questions before ordering its first test.[2] Across tested models, orchestration raised accuracy by an average of 7.9 percentage points on NEJM cases and 4.4 points on emergency cases.[2]

The study authors said the comparison does not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice, and the findings came from a controlled benchmark.[2] They reported that MAI-DxO’s errors typically involved a clinically related diagnosis at the wrong level of specificity.[2] Fifty-six cases were held out from engineering efforts during development.[2] Microsoft says it plans a prospective study in which patients interact directly with the system, with results compared against physicians’ independent assessments.[1] Microsoft also says it will open-source SDBench for select NEJM cases, along with benchmark and evaluation code.[1]