The relentless advance of artificial intelligence is being rigorously tested in an unexpected arena: the world of advanced mathematics. For decades, math has been considered a gold standard for evaluating AI progress, offering a clear, step-by-step logic and definitively verifiable answers. But as AI systems rapidly evolve, existing benchmarks are struggling to keep pace, prompting researchers to develop increasingly challenging tests.
In November 2024, Epoch AI, a non-profit research organization, introduced FrontierMath, a benchmark designed to measure the mathematical reasoning capabilities of the latest AI tools. The benchmark’s creation reflects a growing need for more sophisticated evaluation methods as AI begins to tackle problems previously reserved for human mathematicians.
“It’s a bunch of really hard math problems,” explains Greg Burnham, Epoch AI Senior Researcher. “Originally, it was 300 problems that we now call tiers 1–3, but having seen AI capabilities really speed up, there was a feeling that we had to run to stay ahead, so now there’s a special challenge set of extra carefully constructed problems that we call tier 4.” These tiers range in difficulty from advanced undergraduate to early postdoctoral level mathematics.
When FrontierMath was first released, state-of-the-art AI models could solve less than 2% of the problems. Though, progress has been swift. Today, leading models like GPT-5.2 and Claude Opus 4.6 are solving over 40% of the 300 tier 1–3 problems and more than 30% of the 50 tier 4 problems, demonstrating a significant leap in AI’s mathematical abilities.
AI Tackles PhD-Level Mathematics
This rapid advancement isn’t limited to existing benchmarks. Google DeepMind recently announced that Aletheia, an experimental AI system derived from Gemini Deep Think, achieved publishable PhD-level research results. The achievement involved calculating certain structure constants in arithmetic geometry called eigenweights – a task considered significant, though at the lower end of what would excite most mathematicians, according to Burnham.
“They’re claiming it was essentially autonomous, meaning a human wasn’t guiding the operate, and it’s publishable,” Burnham says. Remarkably, while a human mathematician could likely achieve the same result with dedicated effort, no human had previously done so. This highlights a key trend: AI is beginning to independently generate novel mathematical insights.
These breakthroughs underscore the need for even more challenging benchmarks. “Notice easier math benchmarks that are already obsolete, several generations of them,” Burnham notes. “FrontierMath will probably saturate [meaning state-of-the-art AI models score 100%] within the next two years; could be faster.”
The First Proof Challenge and Beyond
To address this challenge, a group of 11 mathematicians proposed the First Proof challenge on February 6, presenting 10 difficult math questions that arose from their own research. The goal was to assess AI’s ability to solve research-level problems independently. The challenge quickly gained attention, attracting submissions from both professional mathematicians and AI teams, including OpenAI.
However, the results were humbling. By February 14, when the proofs were posted, no one had solved all 10 problems. The authors themselves only solved two using Gemini 3.0 Deep Think and ChatGPT 5.2 Pro. OpenAI’s most advanced internal AI system, with “limited human supervision,” solved five, as did a team from Google DeepMind using Aletheia. The outcomes sparked a range of reactions within the mathematics community, from awe to disappointment.
Epoch AI is also pushing the boundaries with FrontierMath: Open Problems, a pilot benchmark consisting of 16 unsolved research problems. Released on January 27, none have yet been solved by an AI. Burnham explains, “With Open Problems, we’ve tried to build it more challenging. The baseline on its own would be publishable, at least in a specialty journal.” A unique aspect of this benchmark is its automated grading system, designed to verify solutions to problems where the answers are currently unknown.
Burnham views both First Proof and Open Problems as complementary approaches. “I would say understanding AI capabilities is a more-the-merrier situation,” he adds. “AI has gotten to the point where it’s, in some ways, better than most PhD students, so we need to pose problems where the answer would be at least moderately interesting to some human mathematicians, not because AI was doing it, but because it’s mathematics that human mathematicians care about.”
What’s Next for AI and Mathematical Reasoning?
The rapid evolution of AI’s mathematical capabilities necessitates a continuous cycle of benchmark development and refinement. As AI systems become increasingly adept at solving existing problems, the focus is shifting towards challenges that require genuine creativity and insight – problems that even human mathematicians struggle with. The next round of the First Proof challenge is scheduled for March 14, promising an even greater test of AI’s abilities.
The ongoing pursuit of more robust benchmarks isn’t just about measuring progress; it’s about understanding the fundamental limits of AI and identifying areas where human expertise remains essential. As AI continues to push the boundaries of mathematical reasoning, the collaboration between AI researchers and mathematicians will be crucial in shaping the future of both fields.
What are your thoughts on the implications of AI’s growing mathematical prowess? Share your comments below, and let’s continue the conversation.
