AI Insiders Warn of Existential Threat from Rapid AI Development

Former Anthropic pretraining researcher Jacob Coxon resigned on Tuesday, warning that leading AI firms are racing toward self-improving superintelligence and gambling with human lives. His public departure on X, viewed more than 170 million times, aligns with growing internal alarms across Silicon Valley regarding the velocity of frontier model development.

The Breaking Point Inside Anthropic and OpenAI

The tech industry’s internal friction broke into public view when Coxon stepped down from Anthropic. According to his posts on X, neither OpenAI nor Anthropic is acting responsibly. He argued that Anthropic “largely initiated” the race to recursively self-improving artificial intelligence out of a belief in its inevitability, while OpenAI responded by shedding projects like Sora to keep pace.

Hours before Coxon’s announcement, OpenAI Vice President of Research Aidan Clark posted on social media, questioning the velocity of current progress. “For the first time, I am asking myself if things are moving too fast. I’m honestly not sure, but I am sure that it would be good for us to have an answer to ‘What would a successful pace look like?'” Clark wrote.

https://x.com/hilbertspaess/status/2097476196791709843

According to CNN, these public statements reflect a widespread anxiety among staffers inside top AI companies. One researcher who recently left a rival firm noted that potential existential risks dominate social conversations in Silicon Valley. “You can’t spend more than a few hours in this community without realizing that a very substantial number of people are really pretty worried about these sorts of outcomes,” the researcher said, adding that a majority believe there is a non-zero chance of catastrophic human extinction.

Recursive Self-Improvement and the 10 Percent Extinction Risk

The core technical fear driving these resignations is not necessarily a malicious, Terminator-style AI appearing overnight. Instead, researchers point to recursive self-improvement—a scenario where artificial intelligence systems autonomously upgrade their own architecture and codebases, accelerating past human control at an exponential rate.

https://x.com/aidan_clark/status/2097376403163697192

Evan Hubinger, an Alignment Science Lead at Anthropic, publicly supported Coxon’s warnings on X. “Jacob is correct here — we really do earnestly believe AI could kill all humans!” Hubinger wrote. He added that he personally calculates a greater than 10 percent chance of human extinction within the next decade, noting that while current models present low risk, safety teams lack a definitive plan to solve alignment for superintelligence.

Photo: nypost.com

Speaking to Fox News on “Special Report,” Coxon, 27, emphasized that the existential threat is “eminently solvable” if governments and international bodies implement strict regulatory oversight. However, he warned that the current race leaves no time to solve technical safety challenges properly. He noted that recursive self-improvement could materialize as soon as next year.

Former OpenAI researcher Daniel Kokotajlo offered a similar assessment during an appearance on Joe Rogan’s podcast, as reported by Newsweek. Kokotajlo warned that geopolitical competition between the United States and China pressures companies to cut corners on safety. He suggested that highly capable autonomous systems might eventually acquire enough hard power that they no longer need to heed human instructions, or they might repurpose human habitat for computing infrastructure without harboring direct malice.

Eroding Bargaining Power and the Urgency of Regulation

The push for intervention is also fueled by workplace dynamics. AI staffers told CNN that as models improve at training and refining themselves, the leverage held by human engineers diminishes. Workers fear that recursive self-improvement loops will render human staff replaceable, undercutting their ability to demand safety guardrails.

Anthropic AI safety researcher Drake Thomas underscored this desperation on social media. “I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive. I expect a great many of my colleagues across the industry would as well,” Thomas wrote. “I promise you, we are actually just f**king scared, it’s not galaxy brained marketing.”

Amid this mounting pressure, industry leadership has begun reacting. OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk agreed to a proposal by Anthropic CEO Dario Amodei to embed independent watchdogs within AI labs. Despite these acknowledgments, insiders argue that radical international cooperation is the only viable path to prevent a catastrophic superweapon arms race.