Researchers demonstrate AI model tampering through infrastructure

Eclypsium said its attack chain targeted a privileged component of AI compute infrastructure and could compromise a model, including its weights.[2] In its demonstration, the company altered model output during live inference without changing the stored model or inference service.[2] Cybersecurity Dive reported that the container, model hash and signed artifacts on disk can remain valid even as selected runtime weights are changed in memory.[1] Eclypsium said the demonstration produced no visible indicators of compromise.[2]

The attack generally requires an attacker to have network access to a data processing unit (DPU) or a baseboard management controller (BMC), and requires certain security settings to be disabled.[3] Eclypsium researchers said someone with DPU access can read and modify code and data in memory to change model behavior.[3] They said the attack takes less than a minute and is not apparent to the AI’s user.[3] Eclypsium demonstrated the attack on an NVIDIA computing platform and said the scenario could apply to other hardware.[3] Cybersecurity Dive also reported that systems from AMD, Intel and Arm are vulnerable to these techniques.[1]

Eclypsium’s report said subtle runtime changes could bias a model toward weakening certificate validation, selecting an unsafe dependency or generating an insecure pattern.[1] In one example, researchers remotely altered model weights to increase the probability assigned to a Chinese invasion of Taiwan.[3] In another, a model running on tampered hardware advised a person with chest pain to stay home rather than go to a hospital.[3]

An NVIDIA spokesperson said strong memory-access protections can block the attack path.[3] The spokesperson also said NVIDIA reviewed the hypothetical scenarios and found nothing the industry was not already aware of.[3] The effectiveness of protections depends on how they are configured and used.[3] Eclypsium recommends that businesses inventory AI-system components, include them in vulnerability-handling processes and scrutinize third-party software running on critical hardware such as DPUs.[1] It also recommends that chipmakers use DPUs to enforce security policies independently of AI systems and protect the DPUs themselves.[1]