‘Gambling with our lives’: Another AI employee quits over safety concerns

Jonathan Raa/NurPhoto/Getty Images

“The people building AI earnestly believe that it could kill us all by the end of the decade.”

That blunt admission from former Anthropic employee Jacob Coxon made waves Tuesday, as the just-quit 27-year-old AI researcher spilled the beans on his way out the door in a resignation thread on X.

As alarming and serious as Coxon’s revelations sound (“These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources”), they’re also not new. Anthropic got its start in exactly this way, as OpenAI employees concerned about the company’s safety protocols left Sam Altman’s startup to launch their own AI company, arguing their approach was more responsible.

The concerns are becoming commonplace. A number of AI employees have loudly quit their companies over safety concerns over the past several months and years. Others have publicly voiced worries that the technology is moving too quickly and will one day outpace humans’ ability to control it.

In July, nearly 1,400 AI company employees signed an open letter urging the US government to regulate the technology to rein in Big Tech and slow the pace of AI to ensure its safety. And two days ago, OpenAI Chief Scientist Jakub Pachocki warned that AI capabilities are advancing faster than researchers’ ability to reliably monitor and control them.

Martin Lelievre/AFP/Getty Images

‘It feels like early Covid’: The messy scramble to regulate AI

Now, a relatively junior researcher has become the latest to sound the alarm.

“They are racing straight to self-improving superintelligence and gambling with our lives,” Coxon posted.

Coxon’s post gained attention after a more senior Anthropic employee, Evan Hubinger, commented that Coxon was correct.

“We really do earnestly believe AI could kill all humans!” Hubinger posted on X . He said he personally believes the chance is under 10% over the next decade, and despite Anthropic’s good intentions it doesn’t have a plan to avoid so-called superintelligence that could exceed humans’ ability to control AI. Superintelligence , a state at which AI surpasses human capabilities, is a widely debated theoretical milestone with no benchmark that some AI researchers believe has already been achieved and others believe will never come.

Hubinger shortly after clarified his statement in a follow-up post that Anthropic’s own Risk Report acknowledges the concern but says present AI poses little chance of gaining such power. In a response to a request for comment, Anthropic pointed CNN to Hubinger’s follow-up post.

Anthropic CEO Dario Amodei has also repeatedly warned that the race to move fast on the technology could unintentionally result in a catastrophic human-caused error that causes a company to lose control of its AI.

“This is a complex engineering problem and I think something will go wrong with someone’s AI system. Hopefully not ours,” Amodei told the New York Times in February .

Despite the constant stream of warnings, regulation seems far off. The Trump administration has actively worked to undermine state AI regulations and Congress has so far been unwilling to rein in the technology. A common refrain: Any effort to regulate AI represents a capitulation to China, which is under no such constraints.

Except China has introduced AI regulation, particularly aimed at risk management and safety. Last year, the government required AI companies to label AI-generated content to make it transparent and traceable. However, China has resisted the kind of stringent regulation that would prevent the kind of problems Silicon Valley researchers have raised.

In the meantime, in the US at least, AI companies are more or less responsible for policing themselves.

The White House has moved toward a voluntary framework to review certain AI models before launch, sources familiar with a meeting last month between the Trump administration and several top AI companies told CNN.

Among the representatives were those from OpenAI, Anthropic, Google and Meta, according to multiple sources familiar with the situation. The details of the framework will not be released publicly and many of the standards set within it will be classified, according to a June executive order on the framework.

In July, Altman suggested it may be time to slow AI development, saying in the Invest Like The Best podcast that the recent testing incidents involving advanced models were raising “long-term questions” about how AI companies should manage rapid advances in capabilities.

“We may have to pace the rate of AI development to give ourselves enough time for society to harden around these new capability levels,” Altman said.

Jonathan Raa/NurPhoto/Getty Images

“人工智能的开发者们真心相信，到本十年末，人工智能可能会毁灭我们所有人。”

前 Anthropic 员工 Jacob Coxon 周二的这番直言不讳的承认引起了轩然大波，这位刚刚离职的 27 岁人工智能研究员在 X 论坛的辞职帖中透露了真相。

考克森的爆料听起来令人震惊且十分严重（“这些系统很快就会发展成超人般的系统，它们可以入侵任何系统，一夜之间彻底改变任何领域，并获得真正的权力和资源”），但这些并非新鲜事。Anthropic 的创立正是如此：OpenAI 的一些员工对公司的安全协议感到担忧，于是离开了萨姆·奥特曼的这家初创公司，创办了自己的 AI 公司，并声称他们的做法更加负责任。

这些担忧正变得越来越普遍。过去几个月甚至几年里，许多人工智能员工因安全顾虑而公开辞职。另一些人则公开表达了担忧，认为这项技术发展过快，终有一天会超越人类的控制能力。

今年7月，近1400名人工智能公司员工签署了一封公开信，敦促美国政府对这项技术进行监管，以约束大型科技公司并减缓人工智能的发展速度，确保其安全性。两天前，OpenAI首席科学家雅库布·帕乔基警告说，人工智能能力的提升速度已经超过了研究人员对其进行可靠监控和控制的能力。

Martin Lelievre/AFP/Getty Images

“感觉像是新冠疫情初期”：人工智能监管的混乱局面

现在，一位资历尚浅的研究人员也发出了警告。

“他们正朝着自我提升的超级智能方向飞速发展，拿我们的生命冒险，”考克森发帖说。

Coxon 的帖子引起了人们的关注，因为 Anthropic 公司的一位资深员工 Evan Hubinger 评论说 Coxon 的说法是正确的。

“我们真的非常相信人工智能可能会毁灭全人类！”Hubinger在X上发帖说道。他表示，他个人认为未来十年内这种可能性低于10%，而且尽管Anthropic公司的初衷是好的，但它并没有制定任何计划来避免所谓的“超级智能”出现，而超级智能可能会超越人类对人工智能的控制能力。超级智能是指人工智能超越人类能力的状态，这是一个备受争议的理论里程碑，目前尚无明确的衡量标准。一些人工智能研究人员认为超级智能已经出现，而另一些人则认为它永远不会出现。

随后，Hubinger在后续帖子中澄清了他的声明，称Anthropic公司自己的风险报告也承认存在这种担忧，但表示目前的人工智能不太可能获得如此强大的力量。在回应CNN的置评请求时，Anthropic公司建议CNN参考Hubinger的后续帖子。

Anthropic 首席执行官 Dario Amodei 也曾多次警告说，在人工智能技术领域快速发展的竞赛可能会无意中导致灾难性的人为错误，使公司失去对其人工智能的控制。

“这是一个复杂的工程问题，我认为某些人工智能系统会出问题。希望不会是我们的，”阿莫迪在二月份接受《纽约时报》采访时表示。

尽管警告不断，但监管似乎遥遥无期。特朗普政府一直积极削弱各州的人工智能监管，而国会迄今为止也不愿对这项技术加以约束。一种常见的说法是：任何监管人工智能的努力都等同于向中国投降，因为中国目前不受任何此类约束。

但中国已经出台了人工智能监管政策，尤其侧重于风险管理和安全。去年，政府要求人工智能公司对人工智能生成的内容进行标注，以提高透明度和可追溯性。然而，中国一直抵制那种能够预防硅谷研究人员所提出的问题的严格监管措施。

与此同时，至少在美国，人工智能公司或多或少要对自身进行监管。

据CNN报道，知情人士透露，上个月特朗普政府与几家顶级人工智能公司举行了一次会议，白宫已着手制定一项自愿框架，在推出某些人工智能模型之前对其进行审查。

据多位知情人士透露，出席会议的代表来自 OpenAI、Anthropic、谷歌和 Meta 等公司。根据六月份发布的一项关于该框架的行政命令，该框架的细节不会公开，其中许多标准也将被列为机密。

7 月，奥特曼在“像最优秀的人一样投资”播客节目中表示，现在或许应该放慢人工智能的发展速度，因为最近涉及高级模型的测试事件引发了人们对人工智能公司应该如何管理能力快速提升的“长期问题”。

“我们或许需要放慢人工智能的发展速度，以便给社会足够的时间来适应这些新的能力水平，”奥特曼说。