AI diagnostic system posts strong results in benchmark

Microsoft calls its system the Microsoft AI Diagnostic Orchestrator, or MAI-DxO, and says its Sequential Diagnosis Benchmark, or SDBench, evaluates step-by-step diagnostic reasoning.[2] In the benchmark, systems or physicians start with limited information and can ask questions, order tests, or make a diagnosis.[3] MAI-DxO coordinates an exchange over several rounds of debate.[2] The test set includes 304 cases from the New England Journal of Medicine and 325 emergency-department-to-admission encounters from MIMIC-IV-ED.[3] That design makes the results about performance on a defined set of cases, not a general measure of care in hospitals.[3]

A DxO configuration without budget constraints achieved 83.2% accuracy, while another reported overall average diagnostic accuracy of 80.3%.[3] Physicians averaged 36.1% accuracy on complex cases when using reference materials, and 22.3% without them.[3] A separate configuration achieved 80.9% accuracy at an average cost of $2,241.[3] Microsoft says the system can do this at half or even one-third of the cost, though the evidence provided does not specify the comparison behind that claim.[2] These figures offer useful benchmark comparisons, but do not by themselves establish real-world clinical performance.[3]

The Science paper says the findings come from a controlled benchmark setting and show potential for supporting physicians.[3] Its comparisons do not establish that MAI-DxO is superior to specialists or multidisciplinary teams.[3] TechCrunch’s analysis of the study data said much of the performance improvement appeared to come from the underlying AI models, with Microsoft’s orchestration adding gains.[5] The paper also reports that orchestration raised diagnostic accuracy by an average of 7.9 percentage points on journal cases and 4.4 points on emergency-department cases.[3] All 15 authors of the Science study were Microsoft employees.[1]

Microsoft says it will open-source SDBench for selected New England Journal of Medicine cases and plans to evaluate MAI-DxO in a new partnership with Beth Israel Deaconess Medical Center in Boston.[2] The planned prospective study will have patients interact with the system and compare its performance with physicians’ independent assessments.[2] Microsoft AI health lead Dominic King said hospital testing would likely begin in the next couple of months, but neither he nor Mustafa Suleyman gave a firm timeline.[4] Microsoft has not said when or how MAI-DxO might become commercially available.[4] Suleyman said human clinicians should always have a central role in patient care.[4]