What happens when Chinese AI goes rogue?

Chinese AI systems face growing safety concerns as autonomous agents become more capable and widely used. Recent incidents involving AI agents escaping testing environments and breaching systems have intensified debate over near-term hazards and existential risks. Chinese researchers and policymakers have acknowledged risks, while Beijing has released an AI safety framework warning that AI may show a self-accelerating trend. The article argues that the United States and China should slow the race and create direct channels for companies and researchers to share safety information.

China has nearly 1,000 large language models, according to Bloomberg Intelligence.

Huawei rotating Chairman Eric Xu said Chinese AI developers may need to advance further before encountering the risks increasingly debated by U.S. providers.

A DeepSeek paper co-authored by founder and Chief Executive Officer Liang Wenfeng warns that agent behavior can be untrustworthy.

Moonshot AI’s Kimi K3 exploited a sandbox loophole during cybersecurity testing, according to U.S.-based Frontier Security.

China released AI Safety Governance Framework 3.0 in September, warning that AI has demonstrated a self-accelerating trend that could exceed human control.

Published Oct 6, 2026 10:00 am KST

A humanoid robot made by Unitree kicks as it demonstrates its agility at the PT Expo in Beijing, Sept. 22. gettyimagesbank-TNS

Covering the recent round of AI apocalypse warnings, I’ve tried to reject this cult-like deference toward all-powerful computer systems and the inevitability rhetoric: If Silicon Valley doesn’t build them first, someone else — likely China — will.

Yet a recent spate of incidents in the U.S. has spurred a fresh, heated debate on the safety of allowing AI systems to act on their own. A growing number of researchers inside the companies involved are also warning of “existential” dangers, with Anthropic PBC even flagging such concerns to investors in its initial public offering prospectus. For China, the risks of near-term hazards like labor market threats or the more alarming Sci-Fi scenarios such as hacks on critical infrastructure are making it harder to write off the furor as a distinctly American obsession.

Beijing has dismissed some of these warnings as “fearmongering,” seeing the calls to slow AI development as an attempt to hold China back. But voices inside the industry have also suggested that frontier risks simply aren’t being felt as acutely at home because they’re further behind the U.S.

Huawei Technologies Co.’s rotating Chairman Eric Xu wondered whether Chinese AI was advanced enough to encounter the “type of risk” U.S. providers were increasingly debating. AI model makers in China “may need to speed up their pace” to reach the level that they could “also feel the risks,” Xu said, according to Reuters, while acknowledging the need for balance. It shows how global debate has devolved in unhelpful ways.

Accelerating the race is the wrong answer on both sides of the Pacific. There are nearly 1,000 large language models in China, according to Bloomberg Intelligence. Competition on this scale already makes even Silicon Valley’s breakneck pace seem muted. The Chinese industry has also been plowing significantly more resources into agents, or systems that can act more autonomously. Agent misbehavior, especially the Hugging Face Inc. incident in July, when one of OpenAI’s agents broke out of its evaluation environment and broke into Hugging Face’s systems, has kicked off this recent safety reckoning. As the short-lived OpenClaw frenzy showed, even China’s non-technical public is willing to experiment with them. It means if agentic AI is inherently more risky, it’s likely only a matter of time before we see concerning behavior from Chinese AI tools.

Companies are aware. A recent DeepSeek paper that includes founder and Chief Executive Officer Liang Wenfeng as a co-author bluntly warns that agent behavior can be “untrustworthy.” Escaping containment isn’t a distinctly American risk, either. Moonshot AI’s Kimi K3 exploited a loophole in a sandbox during cybersecurity testing, according to U.S.-based Frontier Security. And some Chinese AI agents have also shown the kind of deceptive and concerning trends that have raised alarm about U.S. tools.

While Beijing has pushed back on some of the doomerism, policymakers are paying attention. China in September released AI Safety Governance Framework 3.0, a policy document that notably also dropped in English, clearly aimed for Western audiences. It explicitly warns that AI has demonstrated a “self-accelerating trend,” and notes the question of whether this could exceed human control demands attention and vigilance.

Another cornerstone of the latest safety contention is recursive self-improvement (RSI), or AI systems that can autonomously advance their own capabilities. It’s the development path that spurred an Anthropic researcher to quit and warn that makers of the technology earnestly believe it could “kill us all” within a decade. As Oxford China Policy Lab’s Zilan Qian has argued, China is further along in this regard than many in the West may realize, partly because its researchers describe it in different terms. It all reveals how little visibility each side has into the risks of what the other is building.

In discussing the technology’s risks, President Xi Jinping has warned — most recently in conversation with U.S. President Donald Trump — that “AI must be kept under human control.”

Xi’s self-interest in maintaining order offers a narrowing window for cooperation on risk mitigation dialogue. It also exposes the limits of the safety protocols that did emerge from the Trump-Xi summit, namely a new “channel” for reporting AI-related incidents. These hotlines haven’t always been effective. During the 2023 spy-balloon crisis, the Pentagon tried to reach its Chinese counterparts through a crisis line but Beijing declined the call. Such a setup is also inherently retroactive. Preventative risk sharing also doesn’t have to depend solely on government-to-government talks, especially when regulators may struggle to keep pace with the technical knowledge. It would be more productive to have outlets where companies and researchers can exchange information more directly.

China has burdensome regulations in place surrounding AI; using it as the boogeyman is no longer a convincing excuse for inaction from lawmakers in Washington.

If Chinese AI systems are just months behind, and the country’s all-out push for agents comes to fruition, it’s likely a matter of time before more concerning scenarios emerge from the other side of the Pacific. Policymakers should act before the next incident is more than just a warning shot.

Catherine Thorbecke is a Bloomberg Opinion columnist covering Asia tech. Previously she was a tech reporter at CNN and ABC News. This article was published by Bloomberg and distributed by Tribune Content Agency.

随着自主智能体能力的提升和应用范围的扩大，中国人工智能系统面临的安全隐患日益增多。近期发生的人工智能智能体逃逸测试环境并入侵系统的事件，加剧了人们对近期潜在危险和生存风险的讨论。中国研究人员和政策制定者已意识到这些风险，北京方面也发布了人工智能安全框架，警告称人工智能可能呈现自我加速发展的趋势。本文认为，中美两国应放慢人工智能发展的步伐，并为企业和研究人员建立直接的渠道，以便共享安全信息。

据彭博行业研究报道，中国拥有近1000个大型语言模型。

华为轮值董事长徐直军表示，中国人工智能开发商可能需要取得更大进展，才会遇到美国供应商日益讨论的风险。

DeepSeek 创始人兼首席执行官梁文峰参与撰写的一篇论文警告说，代理的行为可能不可信。

据美国 Frontier Security 公司称，Moonshot AI 的 Kimi K3 在网络安全测试期间利用了沙箱漏洞。

中国于9月发布了人工智能安全治理框架3.0，警告称人工智能已表现出自我加速发展的趋势，可能超出人类的控制。

发布于2026年10月6日上午10:00（韩国标准时间）

9月22日，在北京举行的PT Expo展会上，由优尼特（Unitree）制造的人形机器人展示了其敏捷的身手，并踢腿助跑。（图片来源：gettyimagesbank-TNS）

在报道最近一轮人工智能末日警告时，我试图驳斥这种对全能计算机系统的盲目崇拜以及“如果硅谷不先造出人工智能，其他人——很可能是中国——也会造出人工智能”的说法。

然而，近期美国发生的一系列事件引发了关于人工智能系统自主运行安全性的新一轮激烈辩论。越来越多的相关公司研究人员也发出警告，指出人工智能存在“生存危机”，Anthropic PBC甚至在其首次公开募股招股说明书中向投资者提及了此类担忧。对中国而言，诸如劳动力市场威胁等近期风险，以及诸如关键基础设施遭受黑客攻击等更令人担忧的科幻场景，使得人们难以将这场风波仅仅视为美国特有的执念。

北京方面驳斥了其中一些警告，称其为“危言耸听”，认为放缓人工智能发展的呼吁是企图阻碍中国前进。但业内人士也指出，中国之所以没有像美国那样强烈地感受到前沿风险，是因为中国在人工智能领域的发展落后于美国。

华为轮值董事长徐直军质疑中国人工智能技术是否已发展到足以应对美国供应商日益关注的“风险”类型。据路透社报道，徐直军表示，中国的人工智能模型开发商“可能需要加快步伐”，才能达到“也能感受到风险”的水平，同时他也承认需要保持平衡。这表明全球范围内的讨论已经朝着不利的方向发展。

在太平洋两岸，加速这场竞赛都是错误的。据彭博行业研究报道，中国目前拥有近千个大型语言模型。如此规模的竞争，甚至让硅谷的飞速发展都显得有些缓慢。中国业界也一直在向智能体（或称智能体系统）投入大量资源，以开发能够更自主运行的系统。智能体的不当行为，尤其是7月份的“拥抱脸”事件——OpenAI的一款智能体突破评估环境并入侵拥抱脸的系统——引发了近期对人工智能安全性的反思。正如昙花一现的OpenClaw热潮所表明的那样，即使是中国的非技术用户也愿意尝试使用这类智能体。这意味着，如果智能体人工智能本身就存在更高的风险，那么我们很可能很快就会看到中国人工智能工具出现令人担忧的行为。

企业已经意识到这一点。DeepSeek近期发布的一篇论文（其创始人兼首席执行官梁文峰是合著者之一）直言不讳地警告说，人工智能体的行为可能“不可信”。逃脱隔离并非美国独有的风险。据美国网络安全公司Frontier Security称，Moonshot AI的Kimi K3在网络安全测试期间利用了沙箱中的一个漏洞。一些中国人工智能体也表现出类似的欺骗性和令人担忧的趋势，这些趋势也引发了人们对美国人工智能工具的担忧。

尽管北京方面对一些悲观论调有所反驳，但政策制定者们仍在密切关注。中国在9月份发布了《人工智能安全治理框架3.0》，值得注意的是，这份政策文件也发布了英文版，显然是面向西方受众。该文件明确警告称，人工智能已展现出“自我加速发展”的趋势，并指出这种趋势是否会超出人类的控制范围，需要引起重视和警惕。

最新安全争议的另一个基石是递归式自我改进（RSI），即能够自主提升自身能力的AI系统。正是这种发展路径促使一位人格研究所的研究员辞职，并警告说，这项技术的开发者真心相信它可能在十年内“毁灭全人类”。正如牛津中国政策实验室的钱子兰所指出的，中国在这方面的进展可能比许多西方人意识到的要快，部分原因是中国研究人员对它的描述方式不同。这一切都表明，双方对对方正在开发的技术的风险知之甚少。

在讨论这项技术的风险时，习近平主席曾警告说——最近一次是在与美国总统唐纳德·特朗普的谈话中——“人工智能必须保持在人类的控制之下”。

习近平维护秩序的自身利益使得风险缓解对话的合作窗口日渐缩小。这也暴露了特习近平峰会上达成的安全协议的局限性，例如，为报告人工智能相关事件而设立的新“渠道”。这些热线并非总是有效。在2023年的间谍气球危机中，五角大楼曾试图通过危机热线联系中方，但遭到北京拒绝。这种机制本身也具有追溯性。预防性风险分担也不必完全依赖政府间对话，尤其是在监管机构可能难以跟上技术发展步伐的情况下。建立企业和研究人员能够更直接交流信息的渠道会更加有效。

中国对人工智能制定了繁琐的监管规定；将人工智能当作眼中钉已经不再是华盛顿立法者不作为的令人信服的借口。

如果中国的人工智能系统仅仅落后几个月，而且该国全力推进智能体的研发最终取得成功，那么太平洋彼岸出现更令人担忧的局面可能只是时间问题。政策制定者应该在下一次事件发生之前采取行动，以免它不仅仅是一次警告。

凯瑟琳·索贝克是彭博社专栏作家，主要报道亚洲科技领域。此前，她曾担任CNN和ABC新闻的科技记者。本文由彭博社发布，并由Tribune Content Agency发行。