AI hardware can be hacked so chatbots provide dangerous answers, research shows

I-Hwa Cheng/AFP/Getty Images

The hardware underpinning major AI models can be hacked so that chatbots provide faulty and dangerous answers to questions from users, according to new research shared exclusively with CNN.

The researchers were able to subvert answers to sensitive national security and personal health questions. In one instance, they were able to vastly increase the probability that an AI model assigns to a Chinese invasion of Taiwan by remotely tampering with the “weights” the model uses to process information. In another scenario, an AI model running the tampered hardware told a user seeking medical advice for chest pain to stay at home instead of going to the hospital.

While much of the global anxiety over AI has centered on the actions of AI firms themselves, the research focuses on the security of the multibillion-dollar hardware industry the AI models run on. That includes the chips, memory and data center servers, all of which can affect a model’s behavior.

“We can’t trust the [AI] model if we can’t trust the infrastructure it’s running on,” Yuriy Bulygin, chief executive of Eclypsium, the Portland-based firm that conducted the research, told CNN. The firm is staffed by security engineers who previously worked at hardware giants like Intel.

Eclypsium demonstrated the attack on a computing platform made by tech giant Nvidia. However, the attack scenario is not unique to Nvidia and could apply to other hardware and the code it runs, the researchers say.

State actors like China, Russia or the US are capable of conducting a targeted attack on an AI user using the methods outlined in the research, according to Eclypsium. Security protections against such attacks are available, too.

“We reviewed the hypothetical scenarios and did not identify anything the industry wasn’t previously aware of,” a Nvidia spokesperson told CNN, adding that the firm welcomed working with Eclypsium on the research. Strong memory access protections can block the attack path, the spokesperson noted.

Nvidia last month announced a new security platform designed to guard against hardware attacks and keep AI agents from going rogue. That new platform offers more security monitoring for AI agents, according to Eclypsium.

The effectiveness of security controls against attack scenarios outlined by the researchers “depends on how they are configured and used,” Eclypsium said in a statement to CNN. “We have been working with NVIDIA to ensure these controls are effective.”

The Eclypsium attack requires some technical know-how (which many AI agents can help with), and certain conditions to be met. The attacker generally needs to be on a network with access to either a data processing unit (DPU), a system on a chip that helps run data centers, or a baseboard management controller (BMC), a small computer used to manage servers remotely. Certain security configurations also have to be disabled. With access to the DPU, an attacker can read and modify code and data in the machine’s memory — and change the AI model’s behavior in the process. The attack takes less than a minute, and the user of the AI is none the wiser, the researchers said.

CNN has asked Anthropic and OpenAI, whose models run on Nvidia chips, about the new research.

The research comes as the race for AI supremacy between China and the US has intensified and as real-world cases demonstrating AI’s implications for national security are multiplying.

And there are concerns about the reliability of AI models, even when there is no suggestion they have been tampered with. This spring, a false AI-assisted US intelligence report about a purported Chinese ship transporting nuclear components almost led the US military to board the ship, CNN previously reported . A US military analyst queried a chatbot about intelligence on the ship’s manifest, and the chatbot falsely reported that the ship contained nuclear components. It was not clear whether the chatbot was a commercially available one or a US government product.

President Donald Trump discussed AI last month at the White House with Chinese leader Xi Jinping, but the meeting did not produce a substantive accord on the technology. Trump has resisted calls for guardrails on advanced AI models out of concern that the US will fall behind China in the race for AI supremacy.

The new findings highlight another risk to the integrity of AI models following multiple reports of AI agents going rogue and attacking governments and corporations alike.

“Everybody’s talking about model safety, which is perfectly fine. Please do that,” Eclypsium senior vice president John Loucaides told CNN. “I just am trying to make sure that we don’t forget about all the computers it’s running on.”

The potential attack surface is vast. This summer, longtime cybersecurity researcher HD Moore disclosed multiple vulnerabilities in BMCs. Those very vulnerabilities offer a way into sensitive hardware systems in the research conducted by Eclypsium.

Eclypsium executives said they briefed the National Security Agency and the Cybersecurity and Infrastructure Security Agency on their findings. The two agencies are charged with protecting military and civilian federal computer networks, respectively. CNN has asked those agencies for comment.

Two independent hardware security experts who reviewed the Eclypsium findings at CNN’s request said that they were credible and important to consider.

“The attacks are fundamentally the same as any other [direct memory access] attacks and I don’t see any reason to doubt their findings or results,” said Joe FitzPatrick, a hardware security instructor and researcher.

Dominic Rizzo, senior fellow at the nonprofit Institute for Security and Technology, told CNN: “What [Eclypsium is] describing is a pretty general attack path on modern infrastructure and they are correct to point out that the solutions have to start much deeper in the stack.”

I-Hwa Cheng/AFP/Getty Images

据 CNN 独家报道，一项新的研究表明，支撑主流人工智能模型的硬件可能被黑客入侵，导致聊天机器人对用户的问题提供错误且危险的答案。

研究人员能够篡改有关国家安全和个人健康敏感问题的答案。例如，他们远程篡改人工智能模型用于处理信息的“权重”，从而大幅提高了该模型对中国入侵台湾的概率判断。在另一个案例中，运行在被篡改硬件上的人工智能模型建议一位因胸痛寻求医疗建议的用户待在家中，而不是去医院。

尽管全球对人工智能的担忧大多集中在人工智能公司自身的行为上，但这项研究的重点却放在了人工智能模型运行所依赖的价值数十亿美元的硬件产业的安全问题上。这包括芯片、内存和数据中心服务器，所有这些都可能影响模型的行为。

“如果我们无法信任人工智能模型运行的基础设施，我们就无法信任该模型，”总部位于波特兰的Eclypsium公司首席执行官尤里·布雷金告诉CNN。该公司开展了这项研究，其员工均为曾在英特尔等硬件巨头公司工作过的安全工程师。

Eclypsium 演示了对科技巨头英伟达（Nvidia）计算平台的攻击。然而，研究人员表示，这种攻击方式并非英伟达独有，也可能适用于其他硬件及其运行的代码。

Eclypsium指出，像中国、俄罗斯或美国这样的国家行为体有能力利用研究中概述的方法对人工智能用户发起定向攻击。同时，也存在针对此类攻击的安全防护措施。

英伟达发言人告诉CNN：“我们审查了假设情景，没有发现任何业内人士此前不了解的情况。”该发言人还表示，公司欢迎与Eclypsium就此研究展开合作。发言人指出，强大的内存访问保护机制可以阻断攻击路径。

英伟达上个月发布了一款新的安全平台，旨在防范硬件攻击并防止人工智能代理失控。据 Eclypsium 称，该新平台可为人工智能代理提供更强大的安全监控功能。

Eclypsium在给CNN的一份声明中表示，研究人员概述的攻击场景下，安全控制措施的有效性“取决于它们的配置和使用方式”。“我们一直在与NVIDIA合作，以确保这些控制措施的有效性。”

Eclypsium攻击需要一定的技术知识（许多人工智能代理可以提供这方面的帮助），并且需要满足特定条件。攻击者通常需要连接到能够访问数据处理单元（DPU，一种用于运行数据中心的芯片系统）或基板管理控制器（BMC，一种用于远程管理服务器的小型计算机）的网络。此外，还需要禁用某些安全配置。一旦获得DPU的访问权限，攻击者就可以读取和修改机器内存中的代码和数据，并在此过程中改变人工智能模型的行为。研究人员表示，整个攻击过程不到一分钟，而人工智能用户对此毫不知情。

CNN 已就这项新研究询问了 Anthropologie 和 OpenAI，这两家公司的模型均运行在英伟达芯片上。

这项研究出台之际，正值中美两国争夺人工智能霸权的竞赛愈演愈烈，同时，现实世界中人工智能对国家安全的影响案例也越来越多。

即使没有迹象表明人工智能模型被篡改，人们仍然对其可靠性感到担忧。据美国有线电视新闻网（CNN）此前报道，今年春天，一份由人工智能辅助生成的关于一艘据称运载核部件的中国船只的虚假情报报告，几乎导致美军登船检查。一名美国军事分析员就该船的货物清单向聊天机器人查询情报，而聊天机器人错误地报告称船上载有核部件。目前尚不清楚该聊天机器人是市面上销售的，还是美国政府开发的。

上个月，美国总统唐纳德·特朗普在白宫与中国国家主席习近平讨论了人工智能问题，但会晤并未就该技术达成实质性协议。特朗普一直拒绝为先进的人工智能模型设置限制措施，因为他担心美国会在人工智能领域的竞争中落后于中国。

新的研究结果凸显了人工智能模型完整性面临的另一风险，此前已有数起人工智能代理失控并攻击政府和企业的报道。

“大家都在谈论模型安全，这完全没问题。请务必这样做，”Eclypsium 高级副总裁 John Loucaides 对 CNN 说。“我只是想确保我们不会忘记运行该模型的所有计算机。”

潜在的攻击面非常广阔。今年夏天，资深网络安全研究员HD Moore披露了BMC中的多个漏洞。Eclypsium的研究表明，这些漏洞正是入侵敏感硬件系统的途径。

Eclypsium公司的高管表示，他们已将调查结果汇报给了国家安全局和网络安全与基础设施安全局。这两个机构分别负责保护军用和民用联邦计算机网络。CNN已就此事向这两个机构征求意见。

应 CNN 的要求，两名独立的硬件安全专家审查了 Eclypsium 的调查结果，他们表示这些结果可信且值得考虑。

硬件安全讲师兼研究员乔·菲茨帕特里克表示：“这些攻击本质上与其他任何[直接内存访问]攻击相同，我看不出有任何理由怀疑他们的发现或结果。”

非营利组织安全与技术研究所的高级研究员多米尼克·里佐告诉 CNN：“Eclypsium 所描述的是现代基础设施的一种相当普遍的攻击路径，他们指出解决方案必须从更深层次的技术栈开始，这一点是正确的。”