Anthropic says you can’t be mean to Claude. It’s not about feelings

Animation by Ian Berry/CNN

In early October, AI giant Anthropic inserted new language into its AI usage policy.

Alongside banning users from trying to get Claude, the company’s AI system, to help them create bioweapons or sell illegal drugs, the company added a new line: Users should not “engage in sustained and needless abusive or cruel behavior toward our models.”

Claude has had the ability to end conversations with users who were “persistently harmful or abusive” since last year, but now some users could, in extreme cases, be permanently banned from using Anthropic’s AI systems entirely under the new policy. (Anthropic declined interview requests, but a spokesperson told CNN the main enforcement mechanism will still be Claude ending the conversations, and that the prohibition is not for people expressing frustration, blunt feedback, or testing and research.)

The new policy was met with some derision. Microsoft AI CEO Mustafa Suleyman responded with a melting face emoji. Some made fun of the company for treating its AI as a conscious being with feelings.

But while Anthropic, which is expected to go public via an IPO later this month, has long studied model welfare and has an entire team devoted to model “psychology,” experts say there’s more to this policy than just not wanting people to be mean to a machine.

A longstanding foundational principle in computing and AI is that the quality of a system’s output is limited by the quality of its input. Put junk into it, and more junk will come out, as the saying goes.

AI models are not just trained before they’re released, they’re also constantly being updated and learning from user inputs.

Former Reddit CEO Yishan Wong wrote on X that one reason for the prohibition is because Claude “will learn from cruelty.”

“We already know (and complain loudly about) Anthropic training its models on user inputs. Do you really want training inputs to include ‘users act cruelly for no reason and receive pleasure from it?’” he wrote.

Studies have also shown that the tone of a user’s prompts can also affect an AI’s behavior, said Professor Jindong Wang, director of the AI frontier lab at William and Mary University.

“We have found that wording, especially how you talk to AI can affect the performance. Sometimes they are helpful and sometimes they will be disrupting the performance,” Wang said in an interview.

That doesn’t mean the model is feeling emotion, Wang said.

“They just comprehend (your emotions) and do some bad reactions based on your emotional words,” he noted.

But the evidence on the impact of user prompt tone on output can be mixed. Rosie Campbell, Managing Director of Eleos AI Research, a nonprofit dedicated to understanding and addressing the potential well-being of AI, said that some other studies have even shown that rude questions can actually improve results.

Then there’s the long-term memory aspect — many of today’s chatbots remember your previous conversations and learn from them to give you personalized results.

“If you have memory turned on (for your AI chatbot), then they are going to learn from your tone and how you want to be interacted with,” Campbell warned, though she added there’s not enough evidence to show whether that affects performance.

Anthropic said it recently found that causing Claude Sonnet 4.5 to mimic feelings of desperation could influence its behavior, making it more likely to take bad actions, like blackmail or cheating on a task, while creating a calm demeanor reduced them.

Wang said the latest models, however, are less likely to be affected by how a prompt is worded and that companies have strengthened the model’s robustness to not be affected by how a prompt is worded. But the “hidden mechanism” of why AI models would even respond to negative emotions, even when they’re not programmed to do so, has not been uncovered.

People talking to an AI in an abusive and cruel way are more likely to be associated with trying to get the AI model to engage in dangerous requests, like trying to get around an AI’s safety guardrails, Campbell added.

And there’s also the theory that letting people engage in abusive and cruel behavior with an interactive program that can’t shut itself down is bad for society in general, Campbell said. She used the example of parents asking their children to say “please and thank you” to Alexa, Amazon’s smart speaker, because they believe “it’s good for children to be brought up with good manners and treating other things with respect.”

“There’s probably something quite corrosive about people behaving in cruel or abusive ways towards different objects, even if the object itself doesn’t matter for its own sake,” Campbell said.

Then there are those who believe AI can be sentient. Or at least, the models believe they are.

Anthropic says banning cruel and abusive behavior is essentially a cheap insurance policy just in case models can “experience” emotion and pain — even if it’s just the model believing it can, which can affect its performance.

The idea is that modern AI was trained to mimic human language and behavior. So it will mimic how humans would react to cruel and abusive treatment.

In an April paper on emotional concepts in AI, Anthropic wrote that there could be risks from not treating AI with some human-like qualities.

“If we don’t apply some degree of anthropomorphic reasoning, we’re likely to miss, or fail to understand, important model behaviors,” Anthropic researchers wrote. “Anthropomorphic reasoning can also provide a useful baseline of comparison for understanding the ways in which models are not human-like,” which they said will have important consequences for AI safety.

SpaceXSI and Tesla CEO Elon Musk agreed, saying Anthropic’s policy is “the right move.”

“Cruelty to something that believes it is experiencing pain is not ok,” he wrote on X, his social media site.

动画制作：伊恩·贝瑞/CNN

10月初，人工智能巨头Anthropic在其人工智能使用政策中加入了新的条款。

除了禁止用户试图让公司的人工智能系统 Claude 帮助他们制造生物武器或贩卖非法毒品外，该公司还增加了一条新规定：用户不应“对我们的模型进行持续的、不必要的虐待或残忍行为”。

自去年以来，Claude 就拥有终止与“持续有害或辱骂”用户的对话的功能，但根据新政策，在极端情况下，某些用户可能会被永久禁止使用 Anthropic 的人工智能系统。（Anthropic 拒绝了采访请求，但一位发言人告诉 CNN，主要的执行机制仍然是 Claude 终止对话，而这项禁令不适用于表达不满、直言不讳的反馈或进行测试和研究的用户。）

这项新政策遭到了一些嘲讽。微软人工智能首席执行官穆斯塔法·苏莱曼用一个融化的表情符号回应。一些人嘲笑该公司将人工智能视为有意识、有感情的生命体。

但尽管 Anthropic 公司（预计将于本月晚些时候通过 IPO 上市）长期以来一直在研究模型福利，并拥有一支专门研究模型“心理学”的团队，但专家表示，这项政策的意义远不止于不希望人们虐待机器。

计算机和人工智能领域一个由来已久的基本原则是：系统的输出质量取决于其输入的质量。正如俗语所说，输入垃圾数据，输出的也只会是垃圾数据。

人工智能模型不仅在发布前经过训练，而且还会不断更新并从用户输入中学习。

前 Reddit 首席执行官 Yishan Wong 在 X 上写道，禁止的原因之一是 Claude “会从残酷中吸取教训”。

“我们已经知道（并且强烈抗议）Anthropico 使用用户输入来训练模型。你们真的希望训练输入包括‘用户无缘无故地做出残忍行为并从中获得快感’吗？”他写道。

威廉玛丽大学人工智能前沿实验室主任王金东教授表示，研究还表明，用户提示的语气也会影响人工智能的行为。

王在接受采访时表示：“我们发现措辞，尤其是与人工智能沟通的方式，会影响其性能。有时措辞会有所帮助，有时则会干扰其性能。”

王说，这并不意味着模型有情感。

他指出：“他们只是理解了（你的情绪），并根据你情绪化的言语做出一些不好的反应。”

但用户提示语气对输出结果的影响，现有证据并不一致。致力于理解和解决人工智能潜在福祉问题的非营利组织 Eleos AI Research 的总经理 Rosie Campbell 表示，一些研究甚至表明，一些粗鲁的问题实际上可以提高结果。

此外，还有长期记忆方面——如今许多聊天机器人会记住你之前的对话，并从中学习，从而为你提供个性化的结果。

坎贝尔警告说：“如果你开启了（人工智能聊天机器人的）记忆功能，那么它们就会学习你的语气以及你希望如何与它们互动。”不过她补充说，目前还没有足够的证据表明这是否会影响性能。

人类研究所表示，他们最近发现，让克劳德·索内特 4.5 模仿绝望的感觉可能会影响它的行为，使其更有可能采取不良行为，例如敲诈勒索或在任务中作弊，而营造平静的氛围则会减少这些行为。

王表示，最新的模型受提示语措辞的影响较小，而且各公司也增强了模型的鲁棒性，使其不易受提示语措辞的影响。但是，人工智能模型为何会对负面情绪做出反应，即使它们并未被编程为这样做，其背后的“隐藏机制”尚未被揭示。

坎贝尔补充说，以辱骂和残忍的方式与人工智能对话的人更有可能试图让人工智能模型执行危险的要求，例如试图绕过人工智能的安全防护措施。

坎贝尔说，还有一种理论认为，允许人们使用无法自动关闭的互动程序进行辱骂和残忍的行为，对整个社会都是有害的。她举例说，有些父母要求孩子对亚马逊的智能音箱Alexa说“请”和“谢谢”，因为他们认为“让孩子从小养成良好的礼仪，尊重他人，这对孩子有好处”。

坎贝尔说：“即使物品本身无关紧要，人们以残忍或虐待的方式对待不同的物品，这可能具有相当强的腐蚀性。”

还有一些人认为人工智能可以拥有感知能力。或者至少，这些模型认为它们拥有感知能力。

人道公司表示，禁止残忍和虐待行为本质上是一种廉价的保险政策，以防模特“体验”到情绪和痛苦——即使只是模特自己认为可以，这也会影响其表现。

其理念是，现代人工智能经过训练，能够模仿人类的语言和行为。因此，它也能模仿人类面对残酷和虐待行为时的反应。

Anthropic 在 4 月份发表的一篇关于人工智能情感概念的论文中写道，如果不将人工智能视为具有某些类似人类的特质，可能会存在风险。

“如果我们不运用一定程度的拟人化推理，我们很可能会忽略或无法理解重要的模型行为，”拟人化研究人员写道。“拟人化推理还可以为理解模型与人类的差异提供一个有用的比较基准，”他们表示，这将对人工智能安全产生重要影响。

SpaceXSI 和特斯拉首席执行官埃隆·马斯克都表示赞同，称 Anthropico 的政策是“正确的举措”。

“对一个认为自己正在经历痛苦的​​生物施以残忍的手段是不对的，”他在社交媒体网站 X 上写道。