As artificial intelligence models develop into increasingly sophisticated, the benchmarks used to measure their capabilities are struggling to keep pace. Traditional tests, once challenging for AI, are now often solved with ease, prompting researchers to seek new ways to assess the true limits of these systems. A new assessment, dubbed “Humanity’s Last Exam” (HLE), aims to do just that, presenting a uniquely difficult challenge for even the most advanced AI.
Developed by an international team of nearly 1,000 researchers, HLE consists of 2,500 expert-level questions spanning a vast range of disciplines – from mathematics and natural sciences to the humanities and ancient languages. The project, detailed in a recent study published in Nature, isn’t intended to signal the complete of human expertise, but rather to provide a more accurate understanding of where current AI systems fall short.
Initial results reveal a significant gap between human and artificial intelligence performance. Even as recent models like Gemini 3.1 Pro and Claude Opus 4.6 achieved accuracy rates of around 40-50%, earlier systems struggled considerably. GPT-4o scored just 2.7%, Claude 3.5 Sonnet reached 4.1%, and OpenAI’s o1 model managed only about 8% accuracy on the exam.
The Need for a New Benchmark
For years, researchers have relied on standardized tests like the Massive Multitask Language Understanding (MMLU) exam to track AI progress. Yet, as AI systems have rapidly improved, these benchmarks have become saturated, offering limited insight into their actual capabilities. “When AI systems start performing extremely well on human benchmarks, it’s tempting to think they’re approaching human-level understanding,” explains Dr. Tung Nguyen, an instructional associate professor of computer science and engineering at Texas A&M University and a contributor to HLE. “But HLE reminds us that intelligence isn’t just about pattern recognition — it’s about depth, context and specialized expertise.”
Designing an Exam Beyond AI’s Reach
The creation of HLE involved a meticulous process. Researchers from diverse academic backgrounds contributed questions requiring advanced knowledge in their respective fields. Each question was designed to have a single, verifiable answer. The questions themselves are remarkably diverse, ranging from translating ancient Palmyrene inscriptions to identifying microscopic anatomical features in birds and analyzing the phonological details of Biblical Hebrew pronunciation. Crucially, the team tested each question against leading AI models; any question an AI could answer correctly was excluded from the final set, ensuring the exam remained challenging.
The Center for AI Safety and Scale AI jointly created the benchmark, with a $500,000 prize pool awarded to contributors for the top-rated questions – $5,000 for each of the top 50 and $500 for the next 500. A “community feedback bug bounty program” was also implemented to identify and correct errors in the dataset, according to the project’s website.
What HLE Reveals About the Limits of AI
The initial trials of HLE highlighted the challenges AI faces with specialized knowledge and nuanced understanding. Even the most advanced models struggled with questions requiring expertise across multiple fields. This suggests that while AI excels at pattern recognition, it still lacks the depth of knowledge and contextual awareness that characterizes human intelligence. “Without accurate assessment tools, policymakers, developers and users risk misinterpreting what AI systems can actually do,” Nguyen said.
Researchers emphasize that HLE isn’t about creating a competition between humans and AI. “This isn’t a race against AI,” Nguyen clarifies. “It’s a method for understanding where these systems are strong and where they struggle.” The goal is to gain a clearer understanding of AI’s strengths and weaknesses, informing future development and responsible deployment.
Looking Ahead: Benchmarking AI’s Future
The development of HLE represents a significant step towards creating more robust and meaningful benchmarks for advanced AI systems. The team has kept most of the questions private to maintain the exam’s integrity as AI technology continues to evolve. “What made this project extraordinary was the scale,” Nguyen said. “Experts from nearly every discipline contributed. It wasn’t just computer scientists; it was historians, physicists, linguists, medical researchers. That diversity is exactly what exposes the gaps in today’s AI systems —perhaps ironically, it’s humans working together.”
As AI continues to advance, the need for rigorous and challenging benchmarks like HLE will only become more critical. The ongoing effort to understand the boundaries of artificial intelligence will be essential for navigating the opportunities and challenges that lie ahead.
What are your thoughts on the development of more challenging AI benchmarks? Share your perspective in the comments below.
