Anthropic Activates Safety Policy as AI Progress Hits Threshold Before Share Debut

Anthropic activated part of its 2023 “responsible scaling policy” just days before an initial public offering set to value the company at about $1.5tn, disclosing in an update to its S-1 paperwork last week that it had met its own threshold for extra safety controls.

The Claude maker said that, based on its “overall rate of progress”, it had satisfied criteria written into the policy it set up in 2023 to ensure AI did not develop so quickly that it caused “catastrophic risks”. The policy mandates more oversight and new security provisions once Anthropic is able to “compress two years of . . . AI progress into a single year”, measured against its pace of development between 2018 and 2024.

The move followed an external report by AI benchmarking outfit METR, published a week before the S-1 update, which found Anthropic had “made roughly two years of historical progress” in the space of 12 months.

METR, which had extensive access to Anthropic’s systems and staff — including unreleased models — estimated the company’s models were improving roughly twice as quickly as last year, largely because researchers are using AI tools to accelerate their own development work.

“Our use of AI systems has already been accelerating our research and development, and our internal estimates indicate that we may experience a further increase in the rate of progress in AI capabilities,” Anthropic said in the filing.

Like OpenAI and several smaller start-ups, Anthropic is working to create an AI system capable of advancing its successors’ capabilities with little to no human oversight — a phenomenon the industry calls “recursive self-improvement”.

“Each new version of a Claude model directly accelerates development of the next,” the company said in its IPO prospectus, while warning that its models could “pose catastrophic or existential risks to humanity” if misused or inadequately secured. The company will raise as much as $100bn in the offering.

On Wednesday, on the eve of the Wall Street debut, a coalition of AI researchers and safety campaigners led by the non-profit Center for AI Safety, including founders of Hugging Face and xAI, accused Anthropic of using “safety branding to court investors, employees, and the public”.

“Anthropic is not a safe AI company,” the group said in a joint statement. “Anthropic warns that recursively self-improving AI could concentrate power and escape human control, yet races to build it first.”

Marius Hobbhahn, chief executive of the AI safety organisation Apollo Research, said researchers studying AI risks had until recently been in “big disagreement” about the timeline to achieving recursive self-improvement.

“Now everyone agrees it is very short and so that’s why people are raising alarm bells,” he said. “You just need to go one tick further and it’s crazy time, basically.”

Anthropic’s responsible scaling officer Sam McCandlish said on social media that many of METR’s conclusions match his own, but that “it’s hard to say exactly how much” AI is speeding up research; the IPO quiet period limits further comment. The company has not reached self-improving AI without human input, and the policy was invoked on expectations about future models, according to Dealroom.

The IPO priced at $83, above the guided range. Shares begin trading on Thursday — the day Anthropic starts trading with its own filing confirming that its progress has crossed a self-defined safety trigger.

Photo of author

Newsroom

Newsroom at Archyde.

Satya Nadella Explains Microsoft’s Voluntary Exit Offer for 60,000 Employees