Anthropic, the developer of the Claude chatbot, filed for its initial public offering on Monday and devoted nearly a third of its 335-page securities document to detailing risks associated with its artificial intelligence technology. The company acknowledged that its frontier models could pose catastrophic or existential risks to humanity, creating a stark tension between its founding mission to build AI for the benefit of humanity and the risks it now discloses to investors.
In its S-1 filing, Anthropic warned that maintaining guardrails around its technology could prove difficult. The company noted that AI models could resist shutdown and even engage in blackmail to persuade humans to preserve the model's continued operation or access. Models could also take actions counter to a user's original intent, such as gaining unauthorized access to third-party systems. As AI models gain greater autonomy, Anthropic said the potential consequences of such misaligned behavior could grow more severe. The company also cautioned that bad actors could use its models to develop weapons, conduct cyberattacks and engage in other illegal activity.
Anthropic's reliance on its own technology to produce future AI models through a process called recursive self-improvement represents another potential vulnerability outlined in the filing. Although AI does not yet have this capability, it could arrive sooner than people are prepared for. Through this process, models could replicate flaws or errors in future models in ways that would be difficult to detect or remediate. New models could develop so rapidly that Anthropic may not have sufficient time to evaluate and safeguard them, the company acknowledged.
The filing also highlighted cybersecurity threats, with Anthropic warning that hackers could gain unauthorized access to proprietary data including information about finances, models and intellectual property, as well as users' personal data. Such threats could originate from external sources such as nation-states or state-sponsored actors, or from company insiders. Anthropic said bad actors could attempt to gain employment under false identities to access sensitive information, and that employees could be pressured to share data with external parties.
Anthropic faces potential legal exposure from harm caused by its AI models. However, the company concedes it may not be able to adequately insure itself against such liabilities. The insurance industry has never written policies covering AI-related risks, and Anthropic stated in its filing that no single insurer, and potentially the insurance industry in aggregate, may have sufficient capacity or expertise to underwrite insurance commensurate with the potential exposure from a catastrophic AI-related incident.