Nadella calls for emergency brakes and controls on advanced AI

Nadella said AI systems should separate the supply of intelligence from authority over it.[2] He argued against treating systems as nested black boxes whose recommendations, answers and actions users simply accept or reject.[2] Instead, he called for nondeterministic AI models to operate within deterministic system design, with human controls and reliable operating procedures.[2] He also said organizations should independently determine what a model can access and what actions it can take.[2] That approach makes control of the surrounding system—not just confidence in a model’s output—a central part of deployment planning.

Nadella said frontier closed and open-weight models should be treated like insider risks, while clarifying that this does not mean the models are necessarily malicious.[2] His rationale was that capable actors with access to important systems can make mistakes or be compromised.[2] He called for systems to assume a model is compromised and contain it from the outset.[2] An authorized person, he said, should always be able to pause or shut down a model in the middle of a task.[2] He added that more advanced models will require more advanced containment technologies, which should be standardized.[2]

Nadella proposed model diversity, saying no single model should be the sole dependency for an important outcome or verify its own work.[2] He also said no single model should control both a system’s behavior and the evidence used to assess whether it followed its original intent.[2] Every meaningful model action, he argued, should leave tamper-proof, human-readable evidence, and systems should make it possible to reproduce how an outcome was achieved without relying on the model’s own account.[2] He called for continuous testing of the entire system, including failures, attacks, edge cases and system changes.[2]

Nadella called for industry standards where existing standards are insufficient.[2] He also urged timely disclosure to affected people when AI systems fail or are compromised.[2] He said incident disclosures should explain what went wrong and which controls failed, while helping prevent recurrence and sharing lessons industrywide.[2] The comments came amid warnings from technology executives and researchers about inadequate AI safeguards and the pace of development.[3] The article also said President Donald Trump opposes slowing U.S. AI development, while his administration had launched a safety-focused task force.[3]