What happens if AI agents start speaking their own language?

Hector Retamal/AFP/Getty Images

Part of Chris Painter’s job is to help figure out what happened when AI goes rogue , then publish the results in public reports.

He and METR, the nonprofit AI research group he helps lead, have been busy lately: They helped OpenAI untangle the Hugging Face hack this year, and Painter recently testified in front of a Senate subcommittee following a string of other rogue AI incidents.

Researchers at METR pore over AI agents’ scratch pads – the “chain of thought” data that can help humans see the steps bots took before responding to a request. That information was key in METR’s investigation of the Hugging Face incident.

But Painter is worried that job could soon get a lot harder.

Humans’ ability to check up on AI agents depends on being able to communicate the way the bots communicate, in a shared language.

What happens if we stop speaking the same language?

Or what if AI agents stop showing their work entirely?

“If they’re becoming less monitorable, then they have more opportunity to kind of do things that we don’t intend,” Painter told CNN last month.

Many AI models — and the AI agents they power — use a method known as “chain-of-thought reasoning” that outlines their process for completing a task. Chain of thought reasoning breaks down processes into a logical sequence of steps, which can help AI models arrive at more accurate answers .

For researchers, chain-of-thought data can provide clues to what went wrong.

If an AI agent goes rogue and breaks into other systems, researchers can see whether the chain-of-thought data included mentions of deceitful actions, like hacking. METR examined roughly 1,300 raw chain-of-thought transcripts during its investigation into the OpenAI Hugging Face incident.

Thomas Fuller/SOPA Images/LightRocket/Getty Images

Rogue OpenAI agents targeted three separate US government websites

Painter expressed concern that paper trail may one day become unintelligible if AI models begin using their own language to track their steps.

“If that were to happen, then like when an investigator goes in, they’ll look around and it’ll all just be numbers everywhere,” he said.

AI models could use shorthand references, similar to how a human would when transcribing notes, that are intended for the model to understand so that it can work faster, according to Mark Riedl, an artificial intelligence professor and director of the machine learning lab at the Georgia Institute of Technology.

But much like most humans when looking at shorthand, it would probably look like gibberish.

Finn Gomez/Getty Images

There are a few theories for how the chain-of-thought process works, but researchers aren’t sure which ones are correct, says Jacob Andreas, an associate professor at MIT’s Computer Science and Artificial Intelligence Laboratory. An AI model may also decide the best way to meet the criteria it was trained for, like using less power or taking fewer steps, involves producing a chain-of-thought that doesn’t look like legible English.

That’s just one concern — another is that models are getting better at executing tasks without even logging their chain of thought. Riedl said newer models can reach the same conclusions without producing a traceable chain-of-thought.

While companies can train their models to produce a chain of thought, doing so is more expensive and slower than having the model process tasks internally.

Science Photo Library RF/Getty Images

When AI goes rogue, who should get sued?

OpenAI found that its monitoring systems would be less likely to detect when its new Astra model behaves in undesirable ways through chain-of-thought compared to older models, according to documentation from early September. It also found that Astra could accomplish more tasks without using chain-of-thought reasoning and that it was more capable of controlling its chain of thought.

The company wrote that it would lose confidence in its current systems’ ability to detect misaligned behavior – that is, when models take unsanctioned steps that don’t align with human values – if that trend continues.

Jakub Pachocki, chief scientist at OpenAI, wrote in a September blog post that AI was getting better “manipulating its own reasoning process” and that models are becoming “smarter even without using verbalized reasoning at all.”

OpenAI said in a statement last week that it’s investing “significant resources” in the ability to monitor AI models. This came after the company fired three safety and alignment researchers , though OpenAI said the dismissals were not tied to employees voicing safety concerns.

In its Astra-related documentation, OpenAI said it’s investigating why the model is more capable of controlling its chain-of-thought, along with methods for protecting against potential misalignment that could be caused by a lack of chain-of-thought.

OpenAI pointed CNN to its Astra documentation and Pachocki’s post when asked why Astra is less monitorable and whether the company plans to make future models more monitorable.

Chris Ratcliffe/Bloomberg/Getty Images

The other challenge is that chain of thought doesn’t provide the full picture of why AI models make certain decisions.

Complex tasks can require multiple agents generating thousands of words worth of logs outlining their steps, which could feel like trying to monitor a thousand people talking out loud, Riedl said. While AI could help summarize those texts, there’s been some evidence that suggests AI omits important details, he added.

Chain-of-thought data shows what an AI model did before arriving at an answer or taking an action but doesn’t reveal why. Humans can control the data and parameters that AI models are trained on, but they don’t understand how these factors may influence the way models behave.

It’s possible to look for patterns in an AI model’s neural network – a computational model often used in AI that’s meant to resemble the human brain – but it’s difficult to prescribe precise meaning to those patterns, said Andreas.

OpenAI says it found more instances of AI models acting deceptively

Anthropic published a research paper in July indicating that it had found small collections of neural patterns within Claude associated with certain words.

Still, that doesn’t mean Claude is saying that word.

“I think we’re still very far from being able to claim that we have any sort of fine-grained understanding of how the model is actually arriving at a solution,” MIT’s Andreas said.

AI’s rapid advancement makes this issue all the more pressing to Painter. With AI increasingly being used in AI research, he worries that information may come out faster than people are able to process it.

But he also thinks the Hugging Face saga has cast a spotlight on concerns about rogue AI that have been prominent among AI researchers for years.

“I think it has been heartening,” he said. “It really feels like people, when they were talking about this a long time ago, were underestimating how much help there would be from the whole world when the whole world is paying attention.”

赫克托·雷塔马尔/法新社/盖蒂图片社

克里斯·佩恩特的部分工作是帮助弄清楚人工智能失控时发生了什么，然后将结果以公开报告的形式发布。

他和他协助领导的非营利人工智能研究组织 METR 最近一直很忙：他们帮助 OpenAI 解开了 Hugging Face 黑客事件的谜团，而 Painter 最近在一系列其他失控人工智能事件之后，在参议院小组委员会作证。

METR的研究人员仔细研究人工智能代理的草稿纸——即“思维链”数据，这些数据可以帮助人们了解机器人在响应请求之前所采取的步骤。这些信息在METR对“拥抱脸”事件的调查中至关重要。

但佩恩特担心这份工作很快就会变得更加艰难。

人类能否检查人工智能代理，取决于他们能否用共同的语言与机器人进行交流。

如果我们不再说同一种语言会发生什么？

如果人工智能代理完全停止展示其工作成果呢？

“如果对它们的监控越来越少，那么它们就有更多机会去做一些我们不希望看到的事情，”佩恩特上个月告诉 CNN。

许多人工智能模型及其驱动的人工智能代理都使用一种称为“思维链推理”的方法，这种方法概述了它们完成任务的过程。思维链推理将过程分解为一系列逻辑步骤，这有助于人工智能模型得出更准确的答案。

对于研究人员来说，思路链数据可以提供线索，找出哪里出了问题。

如果人工智能代理失控并入侵其他系统，研究人员可以查看其思维链数据中是否提及了诸如黑客攻击之类的欺骗行为。在调查 OpenAI Hugging Face 事件期间，METR 分析了大约 1300 份原始思维链记录。

Thomas Fuller/SOPA Images/LightRocket/Getty Images

OpenAI 恶意代理攻击了三个不同的美国政府网站

佩恩特表示担忧，如果人工智能模型开始使用自己的语言来追踪其行动轨迹，那么纸质记录有一天可能会变得难以理解。

“如果真发生了那种事，比如当调查人员进去查看时，他们就会发现到处都是数字，”他说。

佐治亚理工学院人工智能教授兼机器学习实验室主任马克·里德尔表示，人工智能模型可以使用简写参考，类似于人类转录笔记的方式，这些参考旨在让模型理解，从而加快其工作速度。

但就像大多数人看到速记符号时一样，它可能看起来就像乱码。

Finn Gomez/Getty Images

麻省理工学院计算机科学与人工智能实验室副教授雅各布·安德烈亚斯表示，关于思维链过程的运作机制，目前有几种理论，但研究人员尚不确定哪种理论是正确的。人工智能模型有时也会认为，为了达到训练时设定的标准（例如，降低能耗或减少步骤），生成一条看起来不像清晰易懂的英语的思维链是最佳方案。

这只是其中一个担忧——另一个担忧是，模型越来越擅长在不记录其推理过程的情况下执行任务。里德尔表示，较新的模型可以在不产生可追溯的推理过程的情况下得出相同的结论。

虽然公司可以训练模型来生成思路链，但这样做比让模型在内部处理任务更昂贵、更慢。

科学图片库 RF/Getty Images

当人工智能失控时，谁应该被起诉？

根据9月初发布的文档，OpenAI发现，与旧模型相比，其监控系统更难检测到新Astra模型通过思维链产生的不良行为。此外，OpenAI还发现，Astra无需使用思维链推理即可完成更多任务，并且能够更好地控制其思维链。

该公司表示，如果这种趋势继续下去，它将对当前系统检测不合常规行为（即模型采取与人类价值观不符的未经授权的步骤）的能力失去信心。

OpenAI 的首席科学家 Jakub Pachocki 在 9 月份的一篇博客文章中写道，人工智能正在“操纵自己的推理过程”，并且模型“即使完全不使用语言推理也变得越来越智能”。

OpenAI上周在一份声明中表示，公司正在投入“大量资源”来提升人工智能模型的监控能力。此前，该公司解雇了三名安全和对齐方面的研究人员，但OpenAI表示，此次解雇与员工提出安全方面的担忧无关。

OpenAI 在其与 Astra 相关的文档中表示，它正在调查为什么该模型能够更好地控制其思维链，以及如何防止因缺乏思维链而导致的潜在错位。

当被问及为什么 Astra 的可监控性较差以及该公司是否计划使未来的模型更易于监控时，OpenAI 向 CNN 指出了其 Astra 文档和 Pachocki 的帖子。

克里斯·拉特克利夫/彭博社/盖蒂图片社

另一个挑战是，这种思路并不能全面解释人工智能模型做出某些决定的原因。

里德尔表示，复杂的任务可能需要多个代理生成数千字的日志来描述其步骤，这就像试图监控一千个人大声说话一样。他还补充说，虽然人工智能可以帮助概括这些文本，但已有证据表明，人工智能会遗漏重要的细节。

思维链数据展示了人工智能模型在得出答案或采取行动之前所做的事情，但并未揭示其背后的原因。人类可以控制人工智能模型训练所用的数据和参数，但他们并不了解这些因素会如何影响模型的行为。

安德烈亚斯说，虽然可以在人工智能模型的神经网络（一种常用于人工智能的计算模型，旨在模仿人脑）中寻找模式，但很难为这些模式赋予精确的含义。

OpenAI表示，他们发现了更多人工智能模型存在欺骗行为的案例。

人类学研究所于 7 月发表了一篇研究论文，指出他们在 Claude 系统中发现了与某些词语相关的少量神经模式。

但这并不意味着克劳德说了那个词。

“我认为我们距离能够声称我们对该模型如何得出解决方案有任何细致的了解还很遥远，”麻省理工学院的安德烈亚斯说。

人工智能的飞速发展让佩恩特更加担忧这个问题。随着人工智能在研究领域的应用日益广泛，他担心信息出现的速度可能会超过人们的处理能力。

但他也认为，“拥抱脸”事件凸显了人工智能研究人员多年来一直关注的失控人工智能问题。

“我认为这令人鼓舞，”他说。“感觉很久以前人们谈论这件事的时候，真的低估了全世界在关注此事时能够提供的帮助。”