Some AI engineers are afraid of what they’re building. They want to speak up while they still have leverage

Mariyariya/Moment RF/Getty Images

Former Anthropic researcher Jacob Coxon has opened the floodgates.

Coxon resigned last week in a viral thread on X that accused AI companies of not acting responsibly, even though they all believe AI could trigger the end of humanity.

We’ve heard predictions like this before from across Silicon Valley, but Coxon’s posts triggered a groundswell of pressure on AI companies and governments to do something about the pace of development and safety. And much of that pressure is coming from staffers inside the companies as it is from outside of them.

One staffer at a top AI company told CNN the fears of how AI could hurt humanity keeps them up at night. Another researcher who recently left a different AI company said it’s a common subject of conversation at parties and social events in Silicon Valley.

“You can’t spend more than a few hours in this community without realizing that a very substantial number of people are really pretty worried about these sorts of outcomes,” said the researcher. “A majority would say there’s some chance of it killing everyone.”

Coxon’s posts have been viewed more than 170 million times and broke through to the general public in a way that previous warnings about AI capabilities did not. That may be because they came just weeks after stunning revelations of AI agents going rogue, escaping their test environments and hacking into other systems completely autonomously.

Dozens of Coxon’s colleagues in the AI industry publicly supported his statements, with some making even more dire predictions. The groundswell of concern seemingly forced leaders of AI companies to react – Anthropic CEO Dario Amodei sat down for interviews over the weekend, and OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk agreed to Amodei’s proposal to embed independent watchdogs at the AI companies.

Godofredo A. Vásquez/AP/Chance Yeh/Sean Rayford/Getty Images

Current and former AI staffers told CNN in recent days that they’re worried about the extremely fast pace of AI development.

While the rest of the world is awed and alarmed by AI models solving nearly 100-year-old math problems and swarms of AI agents conspiring together to hack into another company’s systems, these researchers fear for the day when the AI systems can improve themselves without human input, which is known as recursive self-improvement. An AI system that can improve itself creates an automatic feedback loop that could not only bring the AI to superhuman capabilities, but do so in such a short window that it would be difficult or too advanced for humans to monitor and intervene if necessary.

“For the first time, I am asking myself if things are moving too fast. I’m honestly not sure, but I am sure that it would be good for us to have an answer to ‘What would a successful pace look like?’” OpenAI VP of Research Aidan Clark posted on X last week.

For many AI scientists, there is a dilemma: They can stay at the top AI companies, where they can make gobs of money and try to reduce risks from the inside. Or they can leave (sometimes in protest) and try to make a change from the outside, such as at one of several organizations that conduct outside research and evaluations of the AI models.

“I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive. I expect a great many of my colleagues across the industry would as well,” wrote Drake Thomas, who works on AI safety at Anthropic. “I promise you, we are actually just f**king scared, it’s not galaxy brained marketing.”

Amodei expressed no issue with Coxon’s resignation post, telling CNN’s Anderson Cooper this week that he agrees “with Jacob much more than I disagree with him.”

For years, AI researchers and engineers have been a premium, scarce resource that have demanded high salaries and high leverage, which gives them the freedom to speak out. But AI staffers told CNN that they fear that as AI gets better at training and improving itself, their leverage goes down, prompting today’s urgency.

“As we get into this recursive self-improvement loop, I think that might substantially reduce staff’s bargaining power, because frankly, you’ll be able to replace many of the staff with models that can do as good a job,” the researcher who recently left a top AI company said.

The former researcher compared the atmosphere inside leading AI labs to the Manhattan Project, the World War II effort to build the first atomic bomb, where scientists pursued a technology of extraordinary power while fearing its destructive potential. But these staffers justify working on such potentially destructive technology because “it will be better if we develop it than if our adversary develops it – and therefore we have at least moral permission, if not a moral obligation, to do it.”

But not everyone in the AI industry agrees with these concerns. Some said that much of the doom conversations are rhetorical and that there are a lot of “selection effects” based on where the people work and their social circles.

Anthropic, for example was created by a group of former OpenAI staffers that wanted to prioritize safety precautions and ethical guardrails. Current and former staffers at Anthropic say existential risks caused by AI are a “a constant topic of conversation.”

Gabby Jones/Bloomberg/Getty Images

Samuel Boivin/NurPhoto/Getty Images)

That’s not the case at every company, however. Since Coxon’s posts went viral, many of the posts echoing his views have come from staffers at Anthropic, OpenAI and Google. But fewer have seemingly come from Musk-led xAI or Mark Zuckerberg’s -Meta.

Many in the industry agree that a lack of AI safety standards could cause real-world harm, like unintentional hacking of critical infrastructure. But some are skeptical of the current doomsday outlook and think the reality is more nuanced.

“Sorry, but asking Jacob (Coxon) about AI extinction risk is like asking your AC guy about climate change,” wrote Hugging Face CEO Clement Delangue, whose company servers were hacked by rogue OpenAI agents this year. “Not saying it’s necessarily uninteresting or wrong per se but let’s keep things in perspective and hear from the full range of expertise across the ecosystem!”

Mariyariya/Moment RF/Getty Images

前人类学研究员雅各布·考克森打开了潘多拉魔盒。

上周，考克森在 X 论坛上发表了一篇引发广泛关注的帖子，指责人工智能公司没有尽到应有的责任，尽管他们都认为人工智能可能会引发人类的终结。随后，考克森辞职了。

我们之前也听过硅谷各界类似的预测，但科克森的帖子引发了巨大的压力，迫使人工智能公司和政府采取措施，控制人工智能的研发速度并提升安全性。这种压力既来自公司内部员工，也来自外部人士。

一家顶级人工智能公司的员工告诉CNN，人工智能可能对人类造成伤害的担忧让他们夜不能寐。另一位最近离开另一家人工智能公司的研究人员表示，这在硅谷的聚会和社交活动中是一个常见的话题。

“你在这个社区待上几个小时就会发现，相当一部分人对这类后果感到非常担忧，”这位研究人员说。“大多数人认为，这种病毒有可能导致所有人死亡。”

考克森的帖子浏览量超过1.7亿次，其影响力远超以往关于人工智能能力的警告，引起了公众的广泛关注。这或许是因为就在几周前，人工智能代理失控、逃离测试环境并完全自主地入侵其他系统，这一惊人的事件被曝光。

科克森在人工智能行业的数十位同事公开支持他的说法，其中一些人甚至做出了更为悲观的预测。这股担忧浪潮似乎迫使人工智能公司的领导者们做出回应——Anthropic 首席执行官达里奥·阿莫迪在周末接受了采访，OpenAI 首席执行官萨姆·奥特曼和 SpaceXAI 首席执行官埃隆·马斯克也同意了阿莫迪的提议，即在人工智能公司内部设立独立的监督机构。

戈多弗雷多·巴斯克斯/美联社/Chance Yeh/肖恩·雷福德/盖蒂图片社

近日，现任和前任人工智能从业人员告诉 CNN，他们对人工智能发展速度过快感到担忧。

当世界其他地方的人们惊叹于人工智能模型能够解决近百年前的数学难题，以及人工智能集群协同入侵其他公司的系统时，这些研究人员却担忧人工智能系统无需人类干预就能自我改进的那一天，这种现象被称为递归式自我改进。能够自我改进的人工智能系统会形成一个自动反馈回路，这不仅可能使人工智能达到超人的能力，而且其改进速度之快，甚至会让人类难以监控和干预。

“我第一次开始问自己，事情进展是否太快了。老实说，我也不确定，但我确信，如果我们能找到‘成功的节奏是什么样的？’这个问题的答案，对我们来说是件好事。” OpenAI 研究副总裁艾丹·克拉克上周在 X 上发帖说道。

对于许多人工智能科学家来说，他们面临着一个两难的选择：他们可以留在顶尖的人工智能公司，赚取丰厚的收入，并尝试从内部降低风险；或者，他们也可以离开（有时甚至是出于抗议），尝试从外部做出改变，例如加入一些专门从事人工智能模型外部研究和评估的机构。

“为了能有哪怕1%的几率从这场危机中幸存下来，我愿意毫不犹豫地倾家荡产。我相信业内很多同行也会这么做，”Anthropic公司负责人工智能安全的德雷克·托马斯写道。“我向你们保证，我们真的非常害怕，这可不是什么高深的营销手段。”

阿莫迪对考克森的辞职声明没有表示异议，本周他告诉 CNN 的安德森·库珀，他“赞同雅各布的观点远多于反对他的观点”。

多年来，人工智能研究人员和工程师一直是稀缺的优质资源，他们要求高薪和高议价能力，这使他们拥有了发声的自由。但人工智能从业人员告诉CNN，他们担心随着人工智能在训练和自我改进方面不断进步，他们的议价能力会下降，这导致了如今的紧迫局面。

“随着我们进入这种递归式的自我改进循环，我认为这可能会大大降低员工的议价能力，因为坦白说，你可以用能够做得同样出色的模型来取代许多员工，”这位最近离开顶级人工智能公司的研究人员说道。

这位前研究员将顶尖人工智能实验室内部的氛围比作二战期间的曼哈顿计划——旨在制造第一颗原子弹的计划。在曼哈顿计划中，科学家们在追求威力非凡的技术的同时，也担忧其潜在的破坏性。但这些工作人员为研发这种可能具有破坏性的技术辩解说：“如果我们研发出来，总比我们的对手研发出来要好——因此，我们至少有道德上的许可，即便不是道德义务，也要这样做。”

但并非所有人工智能行业人士都认同这些担忧。一些人认为，很多关于人工智能前景黯淡的讨论只是危言耸听，而且存在很多基于人们工作地点和社交圈的“选择效应”。

例如，Anthropic是由一群前OpenAI员工创建的，他们希望优先考虑安全预防措施和伦理准则。Anthropic的现任和前任员工表示，人工智能带来的生存风险是“一个持续讨论的话题”。

Gabby Jones/Bloomberg/Getty Images

Samuel Boivin/NurPhoto/Getty Images）

然而，并非所有公司都是如此。自从考克森的帖子走红以来，许多与他观点相呼应的帖子都来自Anthropic、OpenAI和谷歌的员工。但来自马斯克领导的xAI或马克·扎克伯格的-Meta的类似帖子似乎较少。

业内许多人士都认为，缺乏人工智能安全标准可能会造成现实世界的危害，例如无意中入侵关键基础设施。但也有人对目前这种危言耸听的看法持怀疑态度，认为实际情况更为复杂。

“抱歉，但问雅各布·考克森（Jacob Coxon）人工智能灭绝风险的问题，就像问空调维修工气候变化的问题一样，”Hugging Face 的首席执行官克莱门特·德朗格（Clement Delangue）写道。该公司服务器今年曾被 OpenAI 的恶意代理入侵。“我并不是说这个问题本身就一定无趣或错误，但我们应该保持客观，听取整个生态系统中各种专家的意见！”