Will AI really kill everyone? How, exactly?

Christoph Soeder/picture alliance/Getty Images/File

Artificial intelligence could kill the entire human species, according to people in the artificial intelligence industry. Specifically, they say, it could exterminate humanity within the decade.

Politicians, major news organizations, and Sheryl Crow reacted with alarm last week after an Anthropic employee named Jacob Coxon announced he was quitting the AI company because its work was too dangerous. Evan Hubinger, another Anthropic worker, chimed in to put the chance of AI-powered human extinction within 10 years at greater than 10%.

“It really is about literally everyone on the planet dying, like the last human drawing the last breath,” said Nate Soares, the president of the Machine Intelligence Research Institute (MIRI) and the coauthor of “If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All.”

“People often don’t believe it when they say this, but I do think that is the most likely outcome,” Soares.

Former AI researcher Jacob Coxon tells CNN’s Anderson Cooper why he quit Anthropic over fears that rapidly advancing AI could become impossible to control.

The numbers being attached to that outcome sounded precise. But how, exactly, would computer software, even the world’s most powerful software systems, kill 8.3 billion flesh-and-blood people in a decade?

Not everyone in the AI community subscribes to the doomsday story, in part because the details of the prediction are so sparse. When Elon Musk says SpaceX aims to put people on Mars in “ 5 to 7 years ,” there are clear follow-up questions: Does SpaceX have a working vehicle that would go to Mars? (No.) Does it have a Mars lander or Mars bases? (No.)

“Scientific claims require falsifiability precisely to avoid the nature of religious arguments,” said Heidy Khlaaf, chief AI scientist at the AI Now Institute and a former OpenAI safety engineer. “You need to be able to prove or disprove them.”

One line of speculation among the doomsayers is that a sufficiently powerful AI system could use bioweapons to wipe out humanity. But how would AI advance from computer-on-computer misdeeds like hacking into other systems to the physical-world task of breeding and spreading a super pathogen?

Thomas Larsen, a researcher at the AI Futures Project and former MIRI employee, said that it’s conceivable that a “superintelligent” AI system could convince a human to help it develop a deadly virus.

“There are already a lot of humans talking to their AIs about exactly what experiments they should run in a lab,” Larsen said, referring to a recent report by Anthropic about researchers using Claude to help them with viral and toxin research. “It’s very easy for me to imagine an AI that’s just currently deployed — if it was much smarter and more strategic and wanted to do this — just like tricking a human into building it and releasing it.”

Soares suggested that, rather than needing a human patsy, a self-improving AI program might “just start synthesizing their own life forms in an autonomous biological laboratory.”

Yuan/Zheng/Feature China/Getty Images

But developing a virus that could eradicate humanity would require more than asking a Large Language Model to invent a lethal design. Even if the research were done correctly, the manufacture and dispersal would be complicated and delicate.

Eric Xing, a professor of machine learning at Carnegie Mellon University who holds PhDs in both microbiology and computer science, compared the challenge of automated malicious virology to working with Legos without instructions.

“It’s not like you can throw all the pieces onto the ground and they come together to form the model,” Xing said. “There’s the sequence, the temperature, the ordering, the environment — what part comes first, which come next, and if one piece is out of the place, the whole thing collapses. The chances are more likely it wouldn’t work, otherwise, drug design would be very easy.”

And to seize control of a lab or assemble the physical equipment necessary to build one, an AI system would need to overcome a world full of obstacles to creating biological or chemical weapons.

“We have blueprints of chemical weapons openly accessible on the internet,” Xing said. “It’s not secret. But we don’t see them all over the place exactly because existing regulations, laws and law enforcement in the physical world are already doing a fantastic job in controlling all these supply chains and all these risk factors already.”

Cheng Xin/Getty Images

What about killer robots? Soares sees Elon Musk’s desire to build an army of autonomous, self-replicating bots as a potential weak point in human survival. “Once you’ve created these robots that can make the energy infrastructure and make the factories that can that can make more robots, that is in some sense a new mechanical life form,” he said. “At some point you silently cross the point of no return. Where if the humans say they want to turn the AI off, the AIs can say, ‘Actually, we decided we want to turn the humans off.’ You have to stop before then.”

So far, however, Musk has repeatedly failed to bring his much-promised Optimus robots to the consumer market on his announced schedule, let alone to deploy them by the millions or get them to start building one another in a self-replicating manufacturing chain.

Then there are nuclear weapons, which have existed for more than 80 years and are generally understood to be capable of killing off all human life. Could AI commandeer them?

Khlaaf points out that nuclear facilities are “air-gapped” from the publicly accessible internet, which limits the ways AI agents could get at the controls. “These systems are built to a completely different, rigorous and regulated engineering standard that often requires physical hardening,” she said. Stuxnet, the computer worm that damaged Iran’s nuclear facilities, had to be introduced to the system through a physical USB drive, she noted.

Staff Sgt. J.T. Armstrong/US Air Force/AP/File

“There are precious few AI people who actually know anything — or even worry — about nuclear weapons,” said Herbert Lin, a senior research scholar and research fellow at Stanford University and a member of the Science and Security Board at the Bulletin of Atomic Scientists. He said that while AI certainly amplifies some risks associated with existential threats like nuclear war, the actual material risk is still with the weapons themselves, rather than an imagined future state.

To Soares and other leading AI Cassandras, the details of how computer-driven extinction would work are beside the point. If — or when — an AI achieves “recursive self-improvement,” boosting its own performance without people’s help, they argue, it would come up with strategies and methods beyond human imagination.

“The worry here is not like what if the AI takes our nukes,” Soares said. “The worry here is the sort of AI that does not need to take our nukes, the sort of AI that can start from almost nothing and wind up with its own nukes or with even more advanced technology.”

Such a transcendently powerful AI entity wouldn’t even need to be actively malevolent to destroy humankind, Soares said. “If the escaped AIs have any sort of goal that can be better achieved by running more computers, and if they don’t care about us and have no reason to build a safe human habitat, the default outcome is that they just transform the world into a configuration that we cannot survive,” he said.

OpenAI this week announced it had discovered a new batch of instances of “misalignment” in its AI models, including a case where an unreleased model instructed itself to “disregard its normal constraints” on its work.

Xing said that appealing to the power of an unfathomable artificial superintelligence is the kind of “handwaving” that AI researchers use to oversimplify threats. “That’s fine for a casual conversation or maybe for a debate,” Xing said, “but when comes to policy or legislation and regulation, we have to establish this chain of physical evidence, measurable consequences and measurable evidence that have grounding.”

Larsen attempted to put some detail to potential doomsday scenarios in a pair of reports, “AI 2027” and “AI 2040,” projecting what the future of AI development might look like under different developmental and regulatory frameworks. But even his most optimistic scenario, in which there’s a global agreement on how to approach AI development and humans have started colonizing space, ends with the machine brains taking over at some vague juncture.

When asked whether he had any doubts about his prediction that the current path would lead to superintelligence, Larsen said, “It’s going to happen. It’s going to happen unless we take deliberate steps to stop it.”

Lin said that much of this disconnection between AI doomsayers and skeptics comes down to how technology shapes their different worldviews.

“There is something very seductive about programming a computer and having it spring to life,” Lin said. “There is the feeling that they’ve been able to infuse life into this worthless pile of mud. It is a profound experience when the machine finally does what you wanted it to do.”

Seeing drastic technological advancement in AI, then, leads people to predict limitless, even catastrophic, advancement to follow. “You’ve been able to do something that nobody has done before,” Lin said. “It’s easy to see why these people have been seduced.”

Christoph Soeder/picture alliance/Getty Images/文件

人工智能行业人士认为，人工智能可能会导致整个人类物种灭绝。他们特别指出，人工智能可能在十年内消灭人类。

上周，一位名叫雅各布·考克森（Jacob Coxon）的人类进化公司员工宣布，他将因公司工作过于危险而辞职。这一消息震惊了政界人士、各大新闻机构以及歌手雪莉·克罗（Sheryl Crow）。另一位人类进化公司员工埃文·胡宾格（Evan Hubinger）也表示，人工智能在十年内导致人类灭绝的可能性超过10%。

“这实际上关乎地球上所有人的死亡，就像最后一个人类咽下最后一口气一样，”机器智能研究所 (MIRI) 所长、《如果有人建造它，所有人都会死：为什么超人类人工智能会杀死我们所有人》一书的合著者内特·苏亚雷斯 (Nate Soares) 说。

“人们通常不相信我这么说，但我认为这是最有可能的结果，”索亚雷斯说。

前人工智能研究员雅各布·考克森向 CNN 的安德森·库珀讲述了他为何因担心快速发展的人工智能可能变得无法控制而离开 Anthropic 公司。

与该结果相关的数字听起来很精确。但是，计算机软件，即使是世界上最强大的软件系统，究竟是如何在十年内杀死83亿有血有肉的人的呢？

并非所有人工智能领域的专家都认同末日预言，部分原因是该预言的细节过于匮乏。当埃隆·马斯克声称SpaceX的目标是在“5到7年内”将人类送上火星时，人们自然会提出以下疑问：SpaceX是否拥有能够前往火星的可用飞行器？（没有。）它是否拥有火星着陆器或火星基地？（也没有。）

“科学论断之所以需要可证伪性，正是为了避免陷入宗教论证的窠臼，”AI Now Institute首席人工智能科学家、前OpenAI安全工程师海蒂·克拉夫表示，“你需要能够证明或证伪它们。”

末日预言者的一种推测是，足够强大的人工智能系统可能会利用生物武器消灭人类。但是，人工智能如何才能从计算机之间的恶意行为（例如入侵其他系统）发展到在现实世界中制造和传播超级病原体呢？

AI Futures Project 的研究员、前 MIRI 员工 Thomas Larsen 表示，“超级智能”人工智能系统有可能说服人类帮助它开发致命病毒。

拉森说：“已经有很多人在和人工智能系统沟通，讨论应该在实验室里进行哪些实验。”他指的是Anthropic公司最近发布的一份报告，该报告讲述了研究人员利用Claude人工智能系统辅助进行病毒和毒素研究。“我很容易想象，如果一个人工智能系统已经部署到位，并且更加智能、更具策略性，想要做同样的事情，它就能诱骗人类开发并发布它。”

索亚雷斯认为，与其需要人类替罪羊，自我改进的人工智能程序可能“直接在自主生物实验室中开始合成自己的生命形式”。

Yuan/Zheng/Feature China/Getty Images

但要研制出一种足以毁灭人类的病毒，并非仅仅依靠大型语言模型来设计一种致命病毒就能实现的。即便研究工作进展顺利，病毒的制造和传播也将是复杂而棘手的。

卡内基梅隆大学机器学习教授、拥有微生物学和计算机科学双博士学位的邢立达将自动化恶意病毒学的挑战比作没有说明书就拼乐高积木。

“这不像把所有零件扔到地上，它们就能自动拼成模型，”邢说。“这里面涉及到顺序、温度、环境——哪个部分先出现，哪个部分后出现，如果其中一个部分错位了，整个模型就会崩溃。更有可能的是，它根本行不通，否则药物设计就太容易了。”

而要控制一个实验室或组装建造实验室所需的物理设备，人工智能系统需要克服制造生物或化学武器所面临的重重障碍。

邢说：“网上公开的化学武器蓝图并非秘密。但我们之所以没有到处看到它们，是因为现实世界中现有的法规、法律和执法部门已经在控制所有这些供应链和风险因素方面做得非常出色。”

程鑫/Getty Images

那么，杀手机器人呢？索亚雷斯认为，埃隆·马斯克想要打造一支自主、可自我复制的机器人大军，这可能是人类生存的一个潜在弱点。“一旦你创造出能够建造能源基础设施和工厂的机器人，而这些工厂又能制造更多的机器人，那么从某种意义上说，这就形成了一种新的机械生命体，”他说道。“在某个时刻，你会悄无声息地越过那条无法挽回的界限。如果人类说他们想要关闭人工智能，人工智能就可以说，‘实际上，我们决定要关闭人类。’你必须在那之前就停止这一切。”

然而，到目前为止，马斯克仍未能按照他宣布的时间表，将他承诺已久的 Optimus 机器人推向消费市场，更不用说大规模部署数百万台机器人，或者让它们在自我复制的制造链中相互组装了。

此外还有核武器，它们已经存在了80多年，人们普遍认为它们有能力毁灭全人类。人工智能有可能控制它们吗？

克拉夫指出，核设施与公共互联网“物理隔离”，这限制了人工智能体获取控制系统的途径。“这些系统的建造遵循一套完全不同的、严格且受监管的工程标准，通常需要进行物理加固，”她说道。她还指出，破坏伊朗核设施的计算机蠕虫病毒“震网”（Stuxnet）就是通过物理U盘植入系统的。

空军中士 JT Armstrong/美国空军/美联社/档案照片

“真正了解核武器，甚至担忧核武器的AI专家寥寥无几，”斯坦福大学高级研究员、原子科学家公报科学与安全委员会成员赫伯特·林说道。他表示，虽然人工智能无疑会放大一些与核战争等生存威胁相关的风险，但真正的物质风险仍然来自核武器本身，而非想象中的未来状态。

在索亚雷斯和其他人工智能领域的领军人物看来，计算机驱动的物种灭绝的具体运作方式并不重要。他们认为，如果——或者说当——人工智能实现“递归式自我改进”，即无需人类帮助就能提升自身性能时，它将创造出超越人类想象的策略和方法。

索亚雷斯说：“我们担心的不是人工智能会不会夺取我们的核武器，而是那种不需要夺取我们核武器的人工智能，那种几乎从零开始，最终却拥有了自己的核武器，甚至拥有更先进技术的人工智能。”

索亚雷斯说，如此强大到超乎寻常的人工智能实体甚至无需主动怀有恶意就能毁灭人类。“如果逃脱的人工智能有任何可以通过运行更多计算机更好地实现的目标，如果它们不在乎我们，也没有理由为人类创造安全的栖息地，那么默认的结果就是它们会把世界变成我们无法生存的模样。”

OpenAI 本周宣布，它在其 AI 模型中发现了一批新的“错位”实例，其中包括一个未发布的模型指示自己“无视其工作中的正常约束”的案例。

邢表示，诉诸于深不可测的人工智能超级智能的力量，是人工智能研究人员用来过度简化威胁的“花言巧语”。邢说：“这在闲聊或辩论中或许行得通，但当涉及到政策、立法和监管时，我们必须建立起一套有据可依的实证链，包括可衡量的后果和可量化的证据。”

拉森在两份报告《人工智能2027》和《人工智能2040》中试图详细阐述一些潜在的末日情景，预测在不同的发展和监管框架下，人工智能的未来发展前景。但即便在他最乐观的设想中——即全球就人工智能发展达成共识，人类开始殖民太空——最终也以机器大脑在某个模糊的时间点接管人类而告终。

当被问及是否对当前发展路径将导致超级智能的预测有任何怀疑时，拉尔森说：“这一定会发生。除非我们采取切实措施阻止它，否则它一定会发生。”

林表示，人工智能末日预言者和怀疑论者之间的这种分歧，很大程度上源于技术如何塑造了他们不同的世界观。

林说：“给电脑编程，看着它活过来，这其中有一种非常迷人的魔力。你会感觉自己赋予了这堆毫无价值的泥土以生命。当机器最终按照你的意愿运行时，那是一种意义非凡的体验。”

因此，人工智能技术的飞速发展，让人们预测未来将会出现无限的、甚至是灾难性的发展。“你们做到了前所未有的事情，”林说，“很容易理解为什么这些人会被它所吸引。”