When AI goes rogue, who should get sued?

Science Photo Library RF/Getty Images

The AI industry has been buzzing with concern about potentially catastrophic risks to humanity — a hum of anxiety that’s grown louder as Anthropic prepares to make its stock market debut next month.

But companies building AI models like Claude and ChatGPT face a more immediate and prosaic risk: They’re likely to get sued. A lot.

AI companies have so far avoided major lawsuits over recent incidents in which they’ve failed to keep AI agents — systems capable of working autonomously with little oversight — contained to their testing environments. But some legal experts and policymakers say it’s pretty clear: The companies can and should be held responsible

“The question of ‘how is the law going to apply to this?’ is a lot easier to answer if you see (AI) as it is, and not as this hypothetical thing that many in the industry are proposing it might someday be,” said Kristen Thomasen, associate professor at University of Windsor and an expert in robotics law. “The law has dealt with new technologies for as long as it has existed.”

The AI industry often refers to the breakout incidents as the technology going “rogue” — language that some critics say is designed to amplify the companies’ claims that their technology is so powerful that even its creators struggle to contain it.

‘Extreme concern’ over first known AI hack of a government system

It may also be a tactic to try to shield themselves from responsibility and future legal claims, Thomasen said.

Still, she said, the liability questions around AI models have a simple answer: “It’s a thing that this company is putting out into the world. If it does harm, there are laws that regulate that.”

So far, the real-world harms done by rogue bots have been far from calamitous but nonetheless costly for the organizations they’ve infiltrated. Last week, the nonprofit that operates Wikipedia said it had found evidence of unauthorized OpenAI agents attempting to make “potentially malicious edits” to some pages and pummeling it with so much traffic that they may have triggered a partial outage.

The investigation “stopped work inside key parts of the organization for us to focus on this,” Eric Mill, product manager for safety and security at Wikimedia, told CNN. “We put our very best people on it… and it’s a pretty messy process.”

Wikimedia launched its investigation this summer as global alarm grew over the Hugging Face attack and dozens of other instances of internal AI models breaking out of their testing environments to hack other computer systems. Meta and Anthropic also disclosed that their test models have hacked into other systems without authorization.

The Wikimedia probe is significant in part because of what didn’t happen: There was no catastrophic, existential threat to humanity of the sort that AI executives often allude to. But the cost — more than two weeks of work carried out by the leading engineers at a global, $200 million-plus organization — was real.

Wikimedia isn’t pushing any one policy or technological solution, and Mill said the organization is in communication with OpenAI about its findings. (OpenAI didn’t respond to CNN’s request for comment.) But the nonprofit hopes these incidents are a wake-up call for AI developers.

“AI companies are not doing enough to secure their systems and protect the public from the harm they cause,” Wikimedia said in its blog post last week. “That burden is falling onto everyone else, including smaller organizations.”

Anthropic, which is preparing to go public next month in one of the largest IPOs in history, disclosed in a regulatory filing that because the legal framework around AI is “unsettled,” those agents’ actions could expose the company to “significant and unpredictable” lawsuits, according to a copy of the document viewed by Reuters.

The potential litigation onslaught isn’t just a headache for company lawyers or a quandary for academics — it could also become a major drag on Anthropic’s bottom line. Litigation costs are a financial drain for any company, but it could become especially problematic for Anthropic, which is burning through cash faster than it can bring it in. The company tore through $42 billion last year while bringing in just $4.6 billion in revenue, according to the document seen by Reuters.

Chance Yeh/Anna Moneymaker/Getty Images

The CEOs of both Anthropic and OpenAI have argued for new regulations tailored to the industry. Dario Amodei recently penned an essay arguing that companies “pace the frontier” and foster international cooperation on safety protocols.

But the rival AI firms have diverged slightly on the liability question.

OpenAI CEO Sam Altman earlier this year favored an Illinois bill that would limit when AI labs can be held liable, even when their products cause “critical harm.” (Anthropic came out against it.)

And while Altman endorsed Amodei’s “pace the frontier” proposal, he also told POLITICO earlier week that his worldview isn’t perfectly aligned with his rival’s: OpenAI believes “the world should accept some bad things happening for the benefits of this technology,” he said. Altman also told the news site that “if something goes wrong with our models during training, there’s gonna be some version of that we need to be responsible for.”

An Anthropic spokesperson told CNN the company sees liability laws as a backstop, rather than a substitute, for safety regulations . Anthropic has never proposed exemptions from existing liability laws, the spokesperson added.

Policymakers on the left and the right have sought to cut through some of the more nuanced questions of future regulation and underscore the legal liabilities AI is already subject to.

Lina Khan, the former Federal Trade Commission chair under President Joe Biden, has become a prominent challenger to AI executives, arguing on X that “we shouldn’t let discussions about new legal regimes distract from the fact that there’s no AI exemption from laws already on the books.”

Meanwhile, Republican Senator Josh Hawley, alongside Democratic counterpart Chris Murphy, introduced a bill last week that seeks to ensure “AI agent operators and developers are held liable for hacking incidents.”

“What these companies need is not more power… but some long-overdue accountability,” Hawley wrote in a Washington Post op-ed . “One way to give that to them is to let the people they harm sue them in court.”

Legal experts who spoke to CNN said that while product liability and consumer protection issues aren’t new, there will inevitably be some nuanced aspects of the technology that will ultimately fall on the courts to rule on.

“We’ve seen these sorts of issues with gun safety or with cigarettes or other issues, and it’s really interesting — what analogy will the courts seize onto?” said Frank Pasquale, a Cornell law professor and an expert on AI law.

One of the bigger questions Pasquale expect courts to grapple with is design. For example, he said, imagine if a manufacturer made a car with sharp swords at the end of it, and then someone drove that car into a pedestrian. The logical response is pretty simple: Don’t design something with swords attached to it.

It’s similar with AI products, he said: Don’t design something that tries a password 50 times or more. Make sure it contains high level instructions that would override any user’s effort to misuse it.

“I think it’s very hard for (AI companies) to say, ‘Oh, it can do a billion different things but we can’t control it.’” Pasquale said. “That doesn’t make sense to me because if it can do all these things, then you can also have meta-rules, or rules for the for the rules, that would that would prevent it from doing certain bad things.”

—CNN’s Hadas Gold contributed reporting.

科学图片库 RF/Getty Images

人工智能行业一直对可能给人类带来灾难性风险的担忧不已——随着 Anthropic 准备下个月在股票市场上市，这种焦虑的嗡嗡声越来越大。

但像 Claude 和 ChatGPT 这样开发人工智能模型的公司面临着一个更直接、更实际的风险：它们很可能会被起诉。而且很可能是很多次。

尽管近期发生多起人工智能公司未能将人工智能代理（即能够在极少监管下自主运行的系统）限制在测试环境中的事件，但迄今为止，这些公司都避免了重大诉讼。然而，一些法律专家和政策制定者表示，事实非常明确：这些公司可以而且应该承担责任。

“如果你能正确看待人工智能，而不是把它看作业内许多人所设想的未来某种假设性的东西，那么‘法律将如何适用于人工智能？’这个问题就更容易回答了，”温莎大学副教授、机器人法专家克里斯汀·托马森说道。“法律自诞生以来就一直在处理新技术。”

人工智能行业经常将突破性事件称为技术“失控”——一些批评人士表示，这种说法旨在强化这些公司的说法，即他们的技术非常强大，甚至连其创造者都难以控制它。

首次发现人工智能入侵政府系统，引发“极度担忧”。

托马森表示，这或许也是一种试图逃避责任和未来法律诉讼的策略。

不过，她表示，关于人工智能模型的责任问题其实很简单：“这是这家公司向世界推出的产品。如果它造成了伤害，那么有法律可以规范。”

迄今为止，恶意机器人造成的实际危害远未达到灾难性的程度，但对于被其渗透的组织而言，仍然代价不菲。上周，运营维基百科的非营利组织表示，他们发现有未经授权的OpenAI代理试图对某些页面进行“潜在的恶意编辑”，并向网站发起大量流量攻击，可能导致部分服务中断。

维基媒体安全产品经理埃里克·米尔告诉CNN，调查“导致公司关键部门停工，以便我们集中精力处理此事”。“我们派出了最优秀的人员参与……这是一个相当棘手的过程。”

今年夏天，随着“拥抱脸”（Hugging Face）攻击事件以及其他数十起内部人工智能模型突破测试环境入侵其他计算机系统的事件引发全球恐慌，维基媒体启动了调查。Meta和Anthropic也披露，他们的测试模型未经授权就入侵了其他系统。

维基媒体调查之所以意义重大，部分原因在于它并未造成什么严重后果：并没有像人工智能高管们经常提到的那种对人类构成灾难性生存威胁的情况发生。但代价却是实实在在的——这家全球性、年收入超过2亿美元的机构的顶尖工程师们为此耗费了超过两周的时间。

维基媒体并未力推任何特定的政策或技术解决方案，米尔表示，该组织正就调查结果与OpenAI进行沟通。（OpenAI未回应CNN的置评请求。）但这家非营利组织希望这些事件能给人工智能开发者敲响警钟。

维基媒体上周在其博客文章中指出：“人工智能公司在保护其系统安全和防止公众遭受其造成的危害方面做得远远不够。这一负担最终落在了包括小型组织在内的所有人身上。”

据路透社看到的一份监管文件副本显示，Anthropic公司准备下个月上市，这将是历史上规模最大的IPO之一。该公司在一份监管​​文件中披露，由于人工智能的法律框架“尚未确定”，这些代理人的行为可能会使公司面临“重大且不可预测的”诉讼。

潜在的诉讼攻势不仅令公司律师头疼，也令学术界困惑，更可能严重拖累Anthropic公司的盈利。诉讼成本对任何公司来说都是一笔不小的开支，但对Anthropic公司而言，问题可能尤为严重，因为该公司现金消耗速度远超收入增长速度。据路透社看到的资料显示，该公司去年支出高达420亿美元，而收入仅为46亿美元。

Chance Yeh/Anna Moneymaker/Getty Images

Anthropic 和 OpenAI 的首席执行官都呼吁制定针对该行业的新法规。Dario Amodei 最近撰写了一篇文章，认为企业应该“引领潮流”，并促进安全协议方面的国际合作。

但这两家人工智能竞争对手在责任问题上略有分歧。

OpenAI首席执行官萨姆·奥特曼今年早些时候支持伊利诺伊州的一项法案，该法案将限制人工智能实验室承担责任的时机，即使他们的产品造成了“严重伤害”。（Anthropic公司对此表示反对。）

尽管奥特曼赞同阿莫迪提出的“引领前沿”方案，但他本周早些时候也告诉POLITICO，他的世界观与竞争对手并不完全一致：OpenAI认为“为了这项技术的益处，世界应该接受一些不好的事情发生”，他说。奥特曼还告诉该新闻网站，“如果我们的模型在训练过程中出现问题，我们将不得不承担某种程度的责任。”

Anthropic公司的一位发言人告诉CNN，该公司将责任法视为安全法规的补充，而非替代品。该发言人补充说，Anthropic从未提出过任何豁免现有责任法的申请。

左翼和右翼的政策制定者都试图厘清未来监管中一些更微妙的问题，并强调人工智能已经承担的法律责任。

曾任乔·拜登总统时期联邦贸易委员会主席的莉娜·汗已成为人工智能高管们的一位杰出挑战者。她在 X 网站上辩称，“我们不应该让关于新法律制度的讨论分散人们对人工智能不受现有法律约束这一事实的注意力。”

与此同时，共和党参议员乔什·霍利与民主党参议员克里斯·墨菲上周提出了一项法案，旨在确保“人工智能代理运营商和开发人员对黑客事件承担责任”。

霍利在《华盛顿邮报》的一篇评论文章中写道：“这些公司需要的不是更大的权力……而是早就应该承担的责任。而赋予他们这种责任的一种方式，就是让那些受到他们伤害的人在法庭上起诉他们。”

接受 CNN 采访的法律专家表示，虽然产品责任和消费者保护问题并不新鲜，但这项技术不可避免地会有一些细微差别，最终需要由法院来裁决。

“我们已经在枪支安全、香烟或其他问题上看到过类似的情况，这真的很有意思——法院会采用什么类比呢？”康奈尔大学法学教授、人工智能法律专家弗兰克·帕斯夸莱说道。

帕斯夸莱预计法院将面临的一大难题是设计问题。例如，他举例说，想象一下，如果一家制造商制造了一辆车尾装有锋利刀剑的汽车，然后有人驾驶这辆车撞向了行人。合乎逻辑的应对方法很简单：不要设计带有刀剑的东西。

他说，人工智能产品也是如此：不要设计那种尝试密码50次甚至更多的系统。要确保它包含高级指令，能够阻止用户任何滥用密码的行为。

帕斯夸莱说：“我认为（人工智能公司）很难说，‘哦，它可以做十亿件事，但我们无法控制它。’这对我来说说不通，因为如果它能做所有这些事情，那么你也可以制定元规则，或者制定规则的规则，来阻止它做某些坏事。”

—CNN的哈达斯·戈尔德对此报道亦有贡献。