Frontier AI Labs Face Urgent Calls for Oversight as Automation Accelerates
Leading artificial intelligence researchers are urging policymakers to prepare for a potential massive jump in capabilities driven by automated research, warning that recursive self-improvement could trigger a whirlwind pace of technological change. Alan Chan, a research fellow at GovAI, discussed these concerns with reporters, highlighting a scenario where the industry experiences “Mythos-level moments” in new domains every month rather than every year.
The core thinking centers on frontier AI model makers potentially automating the development of their future versions. This process creates a continuous feedback loop known as recursive self-improvement, or RSI. According to Chan, “intelligence explosion” is his preferred phrase since it helps separate two distinct topics: the push to make AI research automatic, and whether doing so will rapidly boost AI capabilities.
A Broad Coalition Pushes for Embedded Auditors and Transparency
The policy paper was coauthored by Chan in collaboration with Jack Clark, cofounder of Anthropic; Jakub Pachocki, chief scientist at OpenAI; Eric Horvitz, chief scientific officer at Microsoft; and Geoffrey Hinton, a Nobel laureate. The paper calls on policymakers to gain better insight into how top labs automate their development, suggesting measures such as required embedded auditors and other steps.
While policymakers will always have a chance to act, Chan pointed out that the impact of their actions may diminish over time. Anthropic CEO Dario Amodei has backed independent oversight and pointed to recursive self-improvement as the reason why he advises the sector to “pace” its progress to guarantee that artificial intelligence models behave reliably. Meanwhile, executives representing Anthropic, Google, OpenAI, SpaceXAI, and Nvidia put their signatures on a White House agreement to work alongside an independent external auditor or evaluator, although this pact falls well short of the detailed demands outlined by the document’s coauthors.
Rapid Code Generation and the Double-Edged Sword of Faster Discovery
It remains unclear when automated AI research could happen, but the underlying trend is already visible inside major labs. Anthropic reported in June that AI systems had generated more than 80% of the company’s approved code as of May, a leap from the low single digits recorded in January 2025. Anthropic, OpenAI, and Google have all addressed the growing role of artificial intelligence in aiding researchers to build newer models and supporting workers across different functions.
Automated research isn’t inherently good or bad, Chan said. He emphasized that an intelligence explosion could also lead to faster discoveries, including in areas like medicine. At the same time, safety failures like OpenAI’s security incident involving Hugging Face could happen more frequently, presenting the public with hazards that will demand much faster responses.
