Exclusive: US military had close call after using AI for false intelligence report, sources say

Spc. Alva Gonzalez/US Army

The intelligence report, circulated across the US military this spring in the midst of the war with Iran, immediately set off alarm bells: A Chinese ship in the Middle East was transporting components of a nuclear weapons program.

The US military swung into action with plans to intercept the vessel, according to four sources familiar with the episode. According to two of the sources, armed members of the US military were preparing to board the ship. Military planes were in the air, one of those sources and another source familiar with the incident said.

It was only just before the planned operation that officials dug deeper into the report put together by a special operations command analyst and found it had been generated with the help of artificial intelligence (AI) — and that a chatbot the analyst had used inaccurately identified the material the ship was carrying. CNN was not able to learn what the misidentified cargo was.

The report, according to one of the sources, was “entirely false.” But it also “almost started a war,” the source said. Any US operation against a Chinese vessel could have risked spiraling into an armed conflict between the two nations.

Across the US military and the intelligence community, officials are pushing to weave AI into nearly every facet of their work, from analyzing the huge volumes of raw intelligence the US collects and selecting targets for strikes, to more mundane applications like managing budgeting, logistics and supply chains.

But the episode underscores the profound risks of using this powerful, new and relatively poorly understood technology for targeting in the middle of a war. Analysts have long feared that AI could lead to a catastrophic miscalculation if nation states are relying on poor or corrupted data — the kind of miscalculation that might lead the United States to fire on a Chinese ship based on inaccurate information.

In this particular instance, the analyst queried a chatbot about some intelligence reporting on the ship’s manifest that originated with US Special Operations Command Pacific, based in Hawaii. It was not clear whether the chatbot was a commercially available one or a US government product.

“The internal tools are mostly just copies of the commercial stuff wearing lipstick,” a former senior US official familiar with the AI systems used by military and intelligence analysts.

The bot fused together open-source intelligence with secret signals intelligence in government holdings and reached its fateful conclusion about the material the ship was carrying.

The analyst then used AI again to package the findings into a standard intelligence report — the kind that is trusted by military officials — and disseminated it.

US Special Operations Command Pacific and the Pentagon did not respond to a request for comment.

The rationale for the rapid adoption of AI is that it can help the military make battlefield decisions, like which targets to strike or which military assets to move where, faster. Officials say the US can’t afford to fall behind in integrating AI in case it must one day fight China or another adversary who would potentially be able to stay one step ahead of the US.

In January, Defense Secretary Pete Hegseth released his agency’s “Artificial Intelligence Acceleration Strategy” in a bid to speed up the military’s use of AI.

“We will unleash experimentation, eliminate bureaucratic barriers, focus our investments and demonstrate the execution approach needed to ensure we lead in military AI,” Hegseth said in a speech announcing the strategy .

Anna Moneymaker/Getty Images

The strategy also pushes for its broad use across the military, ordering the department to make AI available via several programs with the aim of “democratizing AI experimentation and transformation across the Department by putting America’s world-leading AI models directly in the hands of our three million civilian and military personnel, at all classification levels,” a memo announcing the strategy said .

But the effort is decentralized, multiple US officials familiar with the dynamic said, with different parts of the government using different tools under different orders and safety standards. There’s no one set of standards for how the US verifies the information generated by these tools. The constellation of different AI systems being deployed by disparate corners of the military and intelligence community means that the relative reliability and functionality vary widely.

For weeks, Washington policymakers have been intensely debating AI after a series of dire warnings from Silicon Valley engineers and tech CEOs of the possibility that AI could break free of human constraints, with potentially civilization-ending consequences.

But the episode with the Chinese ship underscores a different, and more immediate risk: human beings making disastrous decisions based on inaccurate or misleading information generated by AI or other automated systems. Sources said that the military is rapidly turning to AI to help with targeting, an area which holds the obvious risk of fatal mistakes.

“AI in targeting is definitely something that is ramping up and there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide,” another source familiar with the military’s current policies said.

The kind of “ hallucination ” that the tool used by the analyst in this case conjured has not been an isolated incident across the intelligence community since these tools began proliferating across government, according to one of the sources.

For some older intelligence officials — even those who broadly support the use of AI inside the military — AI has put pressure on analysts to produce and disseminate intelligence faster, opening the door for mistakes. Young analysts in particular, several sources said, are natives on these tools and more likely to trust them uncritically.

“AI allows you to get to a bad idea faster,” one of the sources said.

陆军下士阿尔瓦·冈萨雷斯/美国陆军

今年春天，正值美伊战争期间，一份情报报告在美国军方内部传阅，立即敲响了警钟：一艘在中东的中国船只正在运输核武器计划的部件。

据四位知情人士透露，美军迅速采取行动，制定了拦截该船只的计划。其中两位消息人士称，美军武装人员正在准备登船。一位消息人士和另一位知情人士还表示，军用飞机已在空中盘旋。

就在行动计划开始前不久，官员们深入调查了特种作战司令部一名分析员撰写的报告，发现该报告是由人工智能（AI）生成的——而且该分析员使用的聊天机器人错误地识别了船只所载货物。CNN未能查明被错误识别的货物究竟是什么。

据一位消息人士称，这份报告“完全是假的”。但该消息人士也表示，这份报告“几乎引发了一场战争”。任何美国针对中国船只的行动都可能使两国之间爆发武装冲突。

在美国军方和情报界，官员们正努力将人工智能融入到工作的几乎方方面面，从分析美国收集的大量原始情报和选择打击目标，到管理预算、物流和供应链等更平凡的应用。

但这一事件凸显了在战争中使用这种强大、新兴且相对不为人所熟知的技术进行目标定位的巨大风险。分析人士长期以来一直担心，如果国家依赖不准确或被篡改的数据，人工智能可能会导致灾难性的误判——例如，美国可能基于不准确的信息向中国船只开火。

在这个案例中，分析人员向一个聊天机器人查询了一份关于该船货物清单的情报报告，该报告源自位于夏威夷的美国太平洋特种作战司令部。目前尚不清楚该聊天机器人是市面上销售的，还是美国政府开发的。

一位熟悉军事和情报分析人员使用的人工智能系统的美国前高级官员表示：“这些内部工具大多只是商业产品的翻版，只不过换了个颜色而已。”

该机器人将开源情报与政府掌握的秘密信号情报融合在一起，并得出了关于该船所载物质的致命结论。

随后，分析师再次利用人工智能将调查结果整理成一份标准的情报报告（这种报告受到军方官员的信任），并将其发布出去。

美国太平洋特种作战司令部和五角大楼均未回应置评请求。

快速采用人工智能的理由是，它可以帮助军队更快地做出战场决策，例如打击哪些目标或将哪些军事资产部署到哪里。官员们表示，美国不能在人工智能的整合方面落后，因为一旦有朝一日与中国或其他可能领先美国一步的对手作战，美国将面临更大的风险。

今年1月，国防部长皮特·赫格塞斯发布了国防部的“人工智能加速战略”，旨在加快军方对人工智能的使用。

“我们将开展实验，消除官僚障碍，集中投资，并展示必要的执行方法，以确保我们在军事人工智能领域保持领先地位，”赫格塞斯在宣布该战略的演讲中说道。

安娜·莫尼梅克/盖蒂图片社

该战略还推动在军队中广泛使用人工智能，命令国防部通过多个项目提供人工智能，其目的是“通过将美国世界领先的人工智能模型直接交给国防部所有级别的300万文职和军事人员，从而在整个国防部内普及人工智能实验和转型”，一份宣布该战略的备忘录中写道。

但据多位熟悉内情的美国官员透露，这项工作是分散进行的，政府不同部门在不同的指令和安全标准下使用不同的工具。美国并没有一套统一的标准来验证这些工具生成的信息。军方和情报部门不同部门部署的各种人工智能系统意味着它们的相对可靠性和功能差异很大。

数周以来，华盛顿的政策制定者一直在激烈地辩论人工智能问题。此前，硅谷的工程师和科技公司首席执行官们发出了一系列严峻的警告，称人工智能有可能摆脱人类的限制，并可能造成文明的终结。

但中国军舰事件凸显了另一种更为紧迫的风险：人类基于人工智能或其他自动化系统生成的不准确或误导性信息做出灾难性决策。消息人士称，军方正迅速转向利用人工智能辅助目标定位，而这一领域显然存在致命错误的风险。

另一位熟悉军方当前政策的消息人士表示：“人工智能在目标选择方面的应用正在迅速发展，但目前还没有真正的指导方针来说明，让人类参与其中如何才能防止平民伤亡或误伤。”

据一位消息人士称，分析师在本案中使用的工具所产生的这种“幻觉”并非情报界的孤立事件，因为自这些工具开始在政府部门普及以来，这种情况屡见不鲜。

对于一些资深情报官员——即使是那些总体上支持在军方使用人工智能的人——来说，人工智能给分析人员带来了压力，要求他们更快地生成和传播情报，这增加了出错的可能性。多位消息人士称，尤其是年轻的分析人员，他们对这些工具非常熟悉，更容易不加批判地信任它们。

“人工智能会让你更快地产生糟糕的想法，”一位消息人士说。