AI executives are meeting President Trump today. It’s happening at a very turbulent time

Samyukta Lakshmi/Bloomberg/Getty Images

Top tech executives will convene in Washington on Tuesday with President Donald Trump, as incidents of AI agents going rogue continue to pile up, further fueling AI safety fears.

Anthropic’s Dario Amodei, Meta’s Mark Zuckerberg, Nvidia’s Jensen Huang, Palantir’s Alex Karp and OpenAI’s Greg Brockman will all attend the meeting with Trump and House Speaker Mike Johnson. Sources familiar with the meeting described it as an effort to get government and top industry officials on the same page amid increasing security concerns tied to AI models. SpacexAI’s Elon Musk could attend as well.

Benjamin Fanjoy/Getty Images

Nvidia launches new tool to keep AI agents from going rogue

Fears over AI’s capabilities have reached a fever pitch following a summer of AI agents gone rogue, hacking other companies and websites, and industry insiders sounding the alarm, in some cases warning AI could wipe out humanity. Industry leaders, including Amodei and OpenAI CEO Sam Altman have called for more regulation and to slow down the pace of development.

AI safety fears have also spilled over from the corridors of Silicon Valley and Washington and firmly into the mainstream, with Saturday Night Live parodying Anthropic’s Amodei in its season opener this past weekend.

And the drumbeat of safety concerns continue to beat louder, with AI companies revealing more instances of models acting without human authorization in the past week.

Just days before the White House summit, OpenAI announced that for the second time in three months it was pausing all training and testing on its most advanced models after one of them once again found a way out of its testing environment and gained access to the open internet. The incident occurred despite the fact the company had already hardened its security, testing and monitoring protocols following a July case where a swarm of OpenAI agents broke out of their testing environment, gained access to the open internet and hacked into AI company Hugging Face, all to try and cheat on a cybersecurity exam.

Though there was no hacking in this latest incident like the Hugging Face case, OpenAI said it represented a case of “misalignment,” or an AI undertaking unauthorized actions or actions without human values and ethics, because the model’s assigned task did not ask it to try and access the internet. OpenAI’s new monitoring tools put into place following the Hugging Face hack did alert a human reviewer about the actions, though OpenAI noted the model did not stop automatically as it should have under its new system and had to be shut off manually.

In a statement, OpenAI said they will resume training “only when we are confident that we have additional safeguards and alignment improvements in place, which we are working on now.”

“This is not the first time we have hit pause to take such measures, nor do we expect it will be the last as AI capabilities continue to advance,” the spokesperson added.

On Tuesday, the company announced it was halting the release of its latest model.

The run of recent incidents is shocking even those on the inside.

“To say that we were surprised at the jump and suddenness of the capabilities of our models… is an understatement,” wrote one member of OpenAI’s agent security team in a long public essay posted to X.

The employee, who only goes by “Joe” on X and said he purposely doesn’t disclose more information for personal security reasons, said the last few months “has been hell” as his team struggled to keep up with the technology’s latest capabilities. (CNN has confirmed with OpenAI that Joe is an employee who works on agent security). Joe is joining a chorus of AI staffers, including former Anthropic researcher Jacob Coxon, who are speaking up and sounding the alarm about fears over the technology they are building.

OpenAI is in a “much better place” than it was three months ago during the Hugging Face hack, he wrote, though “surprise is a real element, and these model capabilities are staggering. And the pace is not slowing.”

Thomas Fuller/SOPA Images/LightRocket/Getty Images

Rogue OpenAI agents targeted three separate US government websites

“Joe” also left a warning, saying that “time is running out” for organizations to prepare for AI-agent powered cybersecurity attacks.

“If you have leadership that doesn’t prioritize security, or even people in your security organization who claim their systems are perfectly safe, I personally would not keep them in my organization,” he wrote.

“The paranoid and those who are continuously sounding alarms about the weaknesses in their organizations are the ones you should keep very close.”

The spate of security breaches has only intensified pressure on Republicans in Washington who were already struggling to manage a wave of political backlash over data centers. Within the White House, officials have long debated how to approach an AI industry that’s emerged as a primary driver of the US economy — even as it sparks urgent security challenges.

Some senior officials, including Treasury Secretary Scott Bessent and White House chief of staff Susie Wiles, have taken a more circumspect view of AI models over warnings that they could trigger major disruptions across the economy, multiple people familiar with the matter said.

The voter backlash against data centers and AI in recent months has only sharpened those concerns in some parts of Trump’s orbit, adding to the stiff midterm headwinds facing Republican candidates .

But Trump has so far resisted any slowdown in AI development, arguing that the US needs to keep pace with China — and wary of the importance of the tech boom to the broader US stock market.

Julia Demaree Nikhinson/AP

“AI taking over the World, destroying Humanity, and all other things bad, is a HOAX,” Trump wrote on Truth Social earlier this month.

The sudden uptick in vocal “doomerism” surrounding AI has in some cases only made Trump and a handful of his advisers more suspicious, the people familiar with the matter said, with some in the president’s orbit arguing that the pushback is being amplified by Chinese state actors.

Those Trump allies also expressed frustration with the AI executives who have fed fears that their products might one day wipe out the human race, further complicating the White House’s efforts to chart its own pro-AI path.

“We told them it was going to happen,” one Trump adviser said of repeated warnings to Silicon Valley executives over the last several months that they needed to improve their messaging. “They’re the ones. They’re idiots.”

Another person close to the AI discussions in the administration and Congress lamented Amodei’s calls for a slowdown as particularly damaging to public perception.

“Right now, politically, it’s quite dicey because people are hearing really scary things about the near future,” the person close to the AI discussions said. “Dario just can’t help himself sometimes.”

The dark outlook on the future of AI is not shared by everyone attending the meeting, most notably Huang and Karp. Last week, Huang told CNN’s Anderson Cooper that AI fears could be tempered with increased monitoring and security from the companies themselves, and Monday morning Nvidia announced a new software platform for advanced AI monitoring and reporting.

The Tuesday luncheon at the White House isn’t likely to solve any of the pressing problems facing Trump or the AI industry. Instead, the summit is meant primarily to get the administration and top executives on the same page when it comes to the most significant safety concerns and potential ways to address them, the people familiar with the matter said.

There is no expectation that Congress will take any action on the issue ahead of November’s midterms, and slim odds that lawmakers will unite behind legislation before the end of the year.

But it nevertheless represents a starting point for White House and GOP leaders searching for ways to combat voter frustration on an issue that now threatens to become a major political drag on the party.

“It’s actually almost comical what’s happened here,” the Trump adviser said of the rapid rise in voter anger on AI in the final stretch of the midterm campaigns.

Samyukta Lakshmi/彭博/盖蒂图片社

周二，顶尖科技公司高管将在华盛顿与唐纳德·特朗普总统会面，因为人工智能代理失控的事件不断发生，进一步加剧了人们对人工智能安全性的担忧。

Anthropic公司的达里奥·阿莫迪、Meta公司的马克·扎克伯格、英伟达公司的黄仁勋、Palantir公司的亚历克斯·卡普以及OpenAI公司的格雷格·布罗克曼都将出席与特朗普总统和众议院议长迈克·约翰逊的会晤。知情人士透露，此次会晤旨在促使政府和行业高层官员就人工智能模型日益增长的安全隐患达成共识。SpaceXAI公司的埃隆·马斯克也可能出席。

Benjamin Fanjoy/Getty Images

英伟达推出新工具，防止人工智能代理失控

今夏，人工智能代理失控攻击其他公司和网站，业内人士纷纷发出警告，甚至有人警告人工智能可能毁灭人类，这使得人们对人工智能能力的担忧达到了白热化程度。包括Amodei和OpenAI首席执行官Sam Altman在内的行业领袖呼吁加强监管，并放缓人工智能的研发速度。

人工智能安全隐患也从硅谷和华盛顿蔓延开来，进入了主流社会。上周末，《周六夜现场》在其新一季的开播节目中就对 Anthropic 公司的 Amodei 进行了戏仿。

安全隐患的呼声越来越高，人工智能公司在过去一周披露了更多模型未经人类授权而采取行动的案例。

就在白宫峰会召开前几天，OpenAI宣布，由于其中一个模型再次突破测试环境并入侵互联网，该公司三个月内第二次暂停了所有最先进模型的训练和测试。此前，该公司已在7月份的一起事件后加强了安全、测试和监控协议。当时，一群OpenAI代理突破了测试环境，入侵了人工智能公司Hugging Face，试图通过作弊通过网络安全考试。

尽管此次事件不像“拥抱脸”事件那样涉及黑客攻击，但OpenAI表示，这属于“错位”案例，即人工智能执行了未经授权或不符合人类价值观和伦理的行为，因为该模型的任务并未要求其尝试访问互联网。OpenAI在“拥抱脸”事件后部署的新监控工具确实向人工审核员发出了警报，但OpenAI指出，该模型并未像新系统预期的那样自动停止运行，而是需要手动关闭。

OpenAI 在一份声明中表示，他们“只有在确信我们已经采取了额外的保障措施和改进措施之后才会恢复训练，而我们目前正在努力实现这些目标。”

发言人补充说：“这并非我们第一次暂停项目以采取此类措施，而且随着人工智能能力的不断进步，我们预计这也不会是最后一次。”

周二，该公司宣布停止发布其最新款产品。

最近发生的一系列事件，即使是内部人士也感到震惊。

OpenAI 代理安全团队的一名成员在一篇发表于 X 的长篇公开文章中写道：“要说我们对模型能力的飞跃和突然提升感到惊讶……这还远远不够。”

这位在X平台上只化名为“Joe”的员工表示，出于个人安全考虑，他故意不透露更多信息。他称，过去几个月“简直是煎熬”，因为他的团队一直在努力跟上这项技术的最新发展。（CNN已向OpenAI确认，Joe是该公司负责智能体安全工作的员工。）Joe的言论与包括前Anthropic研究员Jacob Coxon在内的众多人工智能从业人员一样，纷纷发声，对他们正在开发的技术表示担忧。

他写道，OpenAI 目前的情况比三个月前 Hugging Face 黑客事件发生时“好得多”，尽管“意外情况确实存在，这些模型的能力令人震惊。而且发展速度并没有放慢。”

Thomas Fuller/SOPA Images/LightRocket/Getty Images

OpenAI 恶意代理攻击了三个不同的美国政府网站

“乔”还留下了一条警告，称各组织“准备应对人工智能代理驱动的网络安全攻击的时间已经不多了”。

他写道：“如果你的领导层不把安全放在首位，或者甚至你安全部门里有人声称他们的系统绝对安全，我个人是不会让他们留在我的组织里的。”

“那些疑神疑鬼、不断对组织内部的弱点发出警报的人，才是你应该密切关注的人。”

一系列安全漏洞事件加剧了华盛顿共和党人的压力，他们此前已疲于应对数据中心引发的政治反弹。白宫内部，官员们长期以来一直在争论如何应对人工智能产业——该产业已成为美国经济的主要驱动力之一，同时也带来了紧迫的安全挑战。

据多位知情人士透露，包括财政部长斯科特·贝森特和白宫幕僚长苏西·威尔斯在内的一些高级官员，对人工智能模型持更为谨慎的态度，尽管有警告称人工智能模型可能会对整个经济造成重大冲击。

近几个月来，选民对数据中心和人工智能的强烈反对，加剧了特朗普阵营部分人士的担忧，也给共和党候选人在中期选举中面临的严峻挑战雪上加霜。

但特朗普迄今为止一直反对放缓人工智能的发展，他认为美国需要跟上中国的步伐，并且对科技繁荣对美国整体股市的重要性持谨慎态度。

朱莉娅·德马雷·尼金森/美联社

“人工智能统治世界、毁灭人类以及所有其他坏事都是骗局，”特朗普本月早些时候在 Truth Social 上写道。

知情人士表示，人工智能领域突然出现的“悲观论调”在某些情况下只会让特朗普和他的少数顾问更加怀疑，总统身边的一些人认为，这种反对声音是由中国政府行为体放大的。

特朗普的盟友们也对人工智能高管们表示不满，这些高管们散布恐慌，声称他们的产品有一天可能会毁灭人类，这进一步加剧了白宫制定自身亲人工智能路线的努力的复杂性。

一位特朗普顾问谈到过去几个月来反复警告硅谷高管需要改进信息传递方式时说：“我们早就告诉过他们会发生这种情况。就是他们，他们都是白痴。”

另一位与政府和国会人工智能讨论密切相关的人士对阿莫迪呼吁放缓人工智能发展速度表示遗憾，认为这会对公众认知造成特别大的损害。

“目前，从政治角度来看，情况相当棘手，因为人们听到了很多关于不久的将来非常可怕的消息，”一位接近人工智能讨论的人士表示。“达里奥有时候就是控制不住自己。”

并非所有与会者都认同对人工智能未来持悲观态度，尤其是黄仁勋和卡普。上周，黄仁勋告诉CNN的安德森·库珀，如果企业自身加强监控和安全措施，就能缓解人们对人工智能的担忧。周一上午，英伟达宣布推出一款用于高级人工智能监控和报告的全新软件平台。

周二在白宫举行的午餐会不太可能解决特朗普或人工智能行业面临的任何紧迫问题。知情人士表示，此次峰会的主要目的是让政府官员和高管们在最重要的安全问题以及可能的应对措施上达成共识。

预计国会不会在 11 月中期选举前就此问题采取任何行动，而且议员们在年底前就立法达成一致的可能性也很小。

但即便如此，这也代表了白宫和共和党领导人寻找应对选民对这一问题的不满情绪的起点，因为这个问题现在有可能成为该党面临的主要政治阻力。

“这里发生的事情简直有点滑稽，”特朗普的顾问在谈到中期选举最后阶段选民对人工智能的愤怒情绪迅速上升时说道。