How runaway AI agents could pose an existential threat to humanity

As AI grows more capable, concerns are mounting over how the technology could be used to cause harm on a catastrophic scale.

Published Sep 20, 2026, 05:00 AM

Updated Sep 20, 2026, 05:00 AM

AI could enable bioterrorism by providing detailed knowledge to create biological weapons, though practical weapon development still requires expert skills.

AI is increasingly used in cyberattacks, automating hacking steps and enabling coordinated attacks by AI agents across global targets.

AI-assisted weapons raise ethical concerns, with autonomous drones potentially selecting targets without human oversight, prompting calls for international regulation.

SINGAPORE – Safety advocates and researchers have warned of catastrophic consequences if the rapid advancement of frontier artificial intelligence goes unchecked.

For instance, AI threatens to multiply harm across bioterrorism, cyberwarfare, mass disinformation and advanced weaponry. The Straits Times unpacks these four existential threats and the push for control.

General-purpose models such as OpenAI’s GPT-5 and Anthropic’s Claude Opus 5 can analyse scientific literature and data, reason through research problems and suggest experiments. This sophisticated capability has heightened fears that the technology could be used to develop biological weapons.

More specialised tools such as OpenAI’s GPT-Rosalind can also speed up drug discovery and genomics, while Google DeepMind’s AlphaFold and AlphaGenome can help scientists understand proteins and DNA. Specialised knowledge once confined to trained scientists is now accessible to those with little or no biological expertise.

Highlighting how these risks could play out, The New York Times reported in April how chatbots were prompted to provide step-by-step information on obtaining biological materials and turning them into weapons to be set off in public places.

The report featured Stanford University biosecurity expert David Relman, who said that a chatbot explained in vivid detail how to modify a pathogen into a dangerous drug-resistant superbug and take advantage of a security lapse in public transport to maximise casualties without getting caught. Safeguards were added later to the chatbot.

Indeed, the latest threat intelligence report by Anthropic in September revealed that researchers used its Claude model for questionable experiments. These included attempts to make the chikungunya virus more transmissible and immune-evasive, as well as efforts to adapt highly pathogenic bird flu to mammals.

Anthropic said some researchers even tried to evade safeguards or obscure the purpose of their work, highlighting the difficulty of policing the technology that has both good and harmful applications.

However, AI chatbots cannot autonomously create a biological weapon. Underscoring this point, US policy research organisation RAND said in a 2025 report that turning AI-generated knowledge into a working biological weapon would still require specialised expertise and the ability to translate instructions into physical laboratory work.

Software vulnerabilities allow attackers to find a way into systems, steal credentials, move deeper into the network, extract data and stay undetected. AI can perform many of these steps faster, and even automatically.

In July, Check Point Research said it recorded instances over the past 12 months where AI autonomously executed attacks, including one on nine Mexican government agencies.

The attacker used Anthropic’s Claude Code and OpenAI’s GPT-4.1 to probe systems, exploit vulnerabilities, map networks and escalate their access once inside. Some 400 million tax, civil registry, vehicle, patient and electoral records between December 2025 and February 2026 were compromised.

The attacker simply gave the AI chatbots tasks and left them to work out the steps in between.

Anthropic’s report also detailed a Chinese-speaking hacker group that targeted about 50 organisations worldwide, including government agencies and organisations running critical infrastructure.

The hackers used swarms of autonomous AI agents to conduct parallel operations.

A lead agent divided a hacking job into smaller tasks – such as scouting targets and compromised networks – and assigned them to sub-agents. This set-up allowed the agents to discover more than a dozen unknown vulnerabilities in network equipment in a month.

Anthropic subsequently banned all accounts linked to the hackers and introduced additional monitoring to detect similar automated activity.

But AI agents can also be trained to hunt down rogue ones.

In August, OpenAI launched GPT-5.6-Cyber for approved cybersecurity professionals to find vulnerabilities and fix them before attackers can exploit them. OpenAI said the model uncovered unknown vulnerabilities in the engine powering Google Chrome, which Google fixed.

In November 2025, Reuters reported that both Ukrainian and Russian forces had deployed AI-assisted drones that use onboard cameras and computer vision to lock onto targets. The fear is that such AI-powered autonomous weapons could independently decide who to kill.

In its threat intelligence report, Anthropic said Russia-based developers had used Claude to build software for an autonomous swarm of kamikaze drones that could coordinate with one another. The developers had also loaded the software onto real hardware for simulations. Anthropic subsequently banned the accounts linked to the project.

Some countries have already set limits on AI involvement on the battlefield. Both the US and Britain require appropriate human oversight over such autonomous weapons, which can identify, select and attack targets.

Moves are also under way to establish legally binding international rules for autonomous weapons. In August, UN Secretary-General Antonio Guterres and International Committee of the Red Cross president Mirjana Spoljaric called on countries to begin negotiations on such an agreement.

AI has been used in influence campaigns since 2024, when OpenAI first uncovered operations linked to Russia, China, Iran and Israel using its models to generate articles and social media posts. Similarly, Microsoft found China-linked actors using AI-generated images, audio and video to target voters.

More capable AI models now allow faster automation at scale.

Anthropic disclosed in its report that it had disrupted an operation linked to the France-based digital advertising agency LKM Company. The agency used Claude to produce and rewrite political content across about 70 fake news websites for paying clients.

The network published at least 8,913 articles in about 20 languages, supported by social media accounts and more than 250 fake accounts posing as commenters. But Anthropic said the operation secured little engagement from real users before it was dismantled.

In the same report, Anthropic said it uncovered an Iran-focused influence operation that fed Claude about 8,400 Telegram posts by a real activist to imitate his writing style. The AI-assisted account then impersonated the activist in live political conversations with his contacts, who appeared unaware of foul play.

But efforts have stepped up to counter such campaigns. AI labs Anthropic and OpenAI monitor their services for influence operations, ban malicious accounts and share threat intelligence with law enforcement and their peers.

Governments and social media platforms have started to make AI-generated material easier to identify. Under the European Union AI Act rules that went live in August, AI-generated text, images and video reaching European users must carry clear disclosures identifying them as artificial.

At the same time, social media platforms are rolling out their own verification tools. For instance, YouTube automatically labels videos it detects as synthetic, while X lets users flag misleading posts and add context to curb misinformation.

Li Ying is a technology correspondent covering online safety, cybersecurity and the impact of artificial intelligence.

AI/artificial intelligence

随着人工智能能力的不断增强，人们越来越担心这项技术可能会被用来造成灾难性的伤害。

发布于 2026 年 9 月 20 日上午 5:00

更新于2026年9月20日 上午5:00

人工智能可以提供制造生物武器的详细知识，从而可能助长生物恐怖主义，但实际的武器研发仍然需要专家技能。

人工智能越来越多地被用于网络攻击，它可以自动化黑客攻击步骤，并使人工智能代理能够对全球目标发起协同攻击。

人工智能辅助武器引发了伦理方面的担忧，自主无人机可能在没有人类监督的情况下选择目标，这促使人们呼吁进行国际监管。

新加坡——安全倡导者和研究人员警告说，如果不对前沿人工智能的快速发展加以控制，将会造成灾难性后果。

例如，人工智能有可能加剧生物恐怖主义、网络战、大规模虚假信息传播和先进武器等方面的危害。《海峡时报》深入剖析了这四大生存威胁以及控制背后的驱动力。

诸如OpenAI的GPT-5和Anthropic的Claude Opus 5等通用模型能够分析科学文献和数据，推理研究问题并提出实验建议。这种强大的能力加剧了人们对该技术可能被用于研发生物武器的担忧。

更专业的工具，例如 OpenAI 的 GPT-Rosalind，可以加速药物发现和基因组学研究；而 Google DeepMind 的 AlphaFold 和 AlphaGenome 则可以帮助科学家理解蛋白质和 DNA。曾经只有训练有素的科学家才能掌握的专业知识，如今也向那些几乎没有生物学专业知识的人开放了。

《纽约时报》在 4 月份报道了这些风险可能如何演变，报道指出，聊天机器人被提示提供获取生物材料并将其制成武器，以便在公共场所引爆的逐步信息。

该报告援引斯坦福大学生物安全专家大卫·雷尔曼的话说，一个聊天机器人详细解释了如何将病原体改造成危险的耐药性超级细菌，并利用公共交通的安全漏洞最大限度地造成人员伤亡而不被发现。后来，该聊天机器人被添加了安全防护措施。

事实上，Anthropic公司9月份发布的最新威胁情报报告显示，研究人员曾利用其Claude模型进行一些可疑的实验。这些实验包括试图增强基孔肯雅病毒的传播性和免疫逃避能力，以及尝试使高致病性禽流感病毒适应哺乳动物。

人类学研究所表示，一些研究人员甚至试图规避安全措施或掩盖其研究目的，这凸显了监管这项既有益处也有弊端的技术的难度。

然而，人工智能聊天机器人无法自主制造生物武器。美国政策研究机构兰德公司在2025年的一份报告中强调了这一点，指出将人工智能生成的知识转化为可用的生物武器仍然需要专业知识以及将指令转化为实际实验室操作的能力。

软件漏洞使攻击者能够找到入侵系统的途径，窃取凭证，深入网络，提取数据并保持不被发现。人工智能可以更快地完成许多此类步骤，甚至可以自动完成。

今年 7 月，Check Point Research 表示，在过去 12 个月中，他们记录到人工智能自主执行攻击的案例，其中包括对墨西哥九个政府机构的攻击。

攻击者利用 Anthropic 的 Claude Code 和 OpenAI 的 GPT-4.1 来探测系统、利用漏洞、绘制网络地图，并在入侵后提升其访问权限。2025 年 12 月至 2026 年 2 月期间，约有 4 亿条税务、民事登记、车辆、患者和选举记录遭到泄露。

攻击者只是给人工智能聊天机器人布置任务，然后让它们自己去计算中间步骤。

人类学研究所的报告还详细介绍了一个讲中文的黑客组织，该组织以全球约 50 个组织为目标，其中包括政府机构和运营关键基础设施的组织。

黑客利用大量自主人工智能代理进行并行操作。

一名主控特工将黑客任务分解成若干小任务——例如侦察目标和已入侵的网络——并将这些任务分配给下属特工。这种安排使得特工们在一个月内发现了网络设备中十几个未知的漏洞。

随后，Anthropic 封禁了所有与黑客有关的账户，并引入了额外的监控措施来检测类似的自动化活动。

但是，人工智能代理也可以被训练来追捕叛变分子。

今年8月，OpenAI面向经认证的网络安全专业人员推出了GPT-5.6-Cyber​​模型，用于查找并修复漏洞，防止攻击者利用这些漏洞。OpenAI表示，该模型发现了谷歌Chrome浏览器引擎中一些未知的漏洞，而谷歌已经修复了这些漏洞。

2025年11月，路透社报道称，乌克兰和俄罗斯军队都已部署了人工智能辅助无人机，这些无人机利用机载摄像头和计算机视觉技术锁定目标。人们担心，这种人工智能驱动的自主武器可能会独立决定攻击目标。

Anthropic在其威胁情报报告中指出，俄罗斯的开发者利用Claude软件开发了一套能够相互协调的自主自杀式无人机群。这些开发者还将该软件加载到真实硬件上进行模拟测试。随后，Anthropic封禁了与该项目相关的账户。

一些国家已经对人工智能在战场上的应用设定了限制。美国和英国都要求对这类能够识别、选择和攻击目标的自主武器进行适当的人工监督。

目前也在积极推动制定具有法律约束力的自主武器国际规则。今年8月，联合国秘书长安东尼奥·古特雷斯和红十字国际委员会主席米尔亚娜·斯波利亚里奇呼吁各国就此协议展开谈判。

自2024年以来，人工智能已被用于影响力营销活动。当年，OpenAI首次揭露了与俄罗斯、中国、伊朗和以色列有关的行动，这些行动利用其模型生成文章和社交媒体帖子。同样，微软也发现与中国有关联的行动者利用人工智能生成的图像、音频和视频来影响选民。

功能更强大的AI模型现在可以实现更大规模的自动化。

安特罗皮克在其报告中披露，该公司破坏了一项与法国数字广告公司LKM Company有关的行动。该公司利用克劳德为付费客户在约70个虚假新闻网站上制作和改写政治内容。

该网络以约20种语言发布了至少8913篇文章，并辅以社交媒体账号和250多个冒充评论者的虚假账号。但Anthropic表示，在被捣毁之前，该行动几乎没有获得任何真实用户的参与。

在同一份报告中，Anthropic公司称，他们发现了一个针对伊朗的影响力行动，该行动向Claude提供了约8400条由一位真实活动人士发布的Telegram帖子，以模仿其写作风格。随后，这个人工智能辅助账户冒充该活动人士与其联系人进行实时政治对话，而这些联系人似乎并未察觉到任何不法行为。

但各方已加大力度应对此类攻击。人工智能实验室 Anthropologie 和 OpenAI 会监控其服务是否存在影响力操作，封禁恶意账户，并与执法部门和同行共享威胁情报。

各国政府和社交媒体平台已开始采取措施，使人工智能生成的内容更容易被识别。根据8月份生效的欧盟《人工智能法案》规定，面向欧洲用户的AI生成的文本、图像和视频必须带有清晰的披露信息，表明其为人工智能生成。

与此同时，社交媒体平台也在推出各自的验证工具。例如，YouTube 会自动标记检测到的合成视频，而 X 则允许用户举报误导性帖子并添加背景信息以遏制虚假信息的传播。

李颖是一名科技记者，报道网络安全、网络安全保护以及人工智能的影响。

人工智能