“We’re at the point where AI evaluations need to get more creative,” Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, told MIT Technology Review on Thursday, November 20, 2026.
A new report from the AI evaluation nonprofit METR finds that some artificial intelligence capabilities continue to improve along an exponential trajectory, while indicating that models fall short on tasks requiring human judgment compared to software engineering.
The release updates the organization’s time horizon plot. The original chart was widely shared and frequently misinterpreted due to the complexity of its underlying measures, MIT Technology Review reported.
METR tested models across a large suite of tasks previously completed by humans. Each task received a difficulty score based on the length of time human testers required to finish it, and models were subsequently rated according to the task lengths they successfully completed.
That initial plot revealed rapid capability gains. Every four months, the length of tasks that frontier models could complete successfully doubled.
Verifiable Tasks Versus Open-Ended Judgment
The updated plot incorporates additional tasks. Some are estimated to take humans considerably longer than the original suite, while others are significantly more open-ended.
A central question across the AI industry involves whether progress on verifiable tasks—where automated checks confirm success or failure—carries over to domains where outcomes resist simple measurement. Researchers train neural networks on verifiable tasks by running them repeatedly, reinforcing successful neural pathways and weakening failing ones.

This methodology explains why models excel at coding and math. Applying reinforcement learning to open-ended tasks where success lacks a clear definition, such as choosing a new research direction, remains substantially more difficult.
METR’s original time horizon plot omitted such tasks entirely due to evaluation challenges. Yet mastering these open-ended challenges remains essential if AI systems are to automate research and development.
AI research and development is currently drawing intense scrutiny. According to MIT Technology Review, a report from the AI evaluation nonprofit METR indicates that some AI capabilities are continuing to improve along an exponential trajectory.
That earlier report sparked concerns that the company could soon achieve full recursive self-improvement. In this regime, AI systems autonomously design their own successors without human involvement.
Designing Evaluations and Scoring Arguments
METR researchers integrated fuzzier, judgment-laden tasks into their latest evaluation suite to address R&D automation. In one test, models attempted to design new AI evaluations for METR’s internal research, with human employees scoring the results.
Even though the most advanced models now outperform almost every human at software engineering, they fell short here. Not one of the AI-designed evaluations met METR employees’ standards for research utility.

Another test required models to score philosophical arguments. This exercise aimed to measure a model’s capacity for good judgment, though METR noted in its report that employees disagreed on how relevant the task is for automating AI R&D.
Models fell short of human performance on the philosophical argument task as well.
Kapoor said that he was unsurprised by the models’ performance deficits on open-ended tasks like evaluation design.
“Trying to push the envelope of the space of evaluations is a welcome change.” Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, said.
Uncertain Trends and Future Evaluations
Opinions remain divided regarding performance discrepancies between verifiable and hard-to-verify tasks. Kapoor views research taste—the ability to select promising questions, hypotheses, and experiments—as a primary bottleneck in automating AI R&D.
Conversely, others argue that models trained on verifiable tasks will eventually master open-ended challenges. Meanwhile, progress on verifiable tasks shows no signs of slowing down.
Despite lagging behind human performance, newer models show incremental improvement. Frontier models such as Opus 5.5 achieved what METR classifies as borderline above-the-bar performance on evaluation design tasks.
While still not usable for actual METR research, this tier represents a step up from a year prior, when every tested model scored in the clearly below-the-bar category.
Because judgment-heavy tasks are difficult to design and score, METR included few of them in the latest report. This scarcity complicates predictions about how models will progress in these domains.
“The improvement we observe, I think that does tell us something for the current moment,” said Najoung Kim, a linguist and computer scientist at Boston University. “I’m not sure just yet how to think about how it would extrapolate.”
Predicting progress on such tasks will likely remain challenging, according to Matt Putz, a member of the technical staff at METR. “Ultimately, I don't think we'll be able to establish trends as clean and robust as for the easy-to-specify [tasks].” Matt Putz, a member of the technical staff at METR, said in an email.
Founded in December 2023 as an independent nonprofit spun off from the Alignment Research Center, METR is based in Berkeley, California, and led by CEO Beth Barnes, a former OpenAI alignment researcher. The institute focuses on evaluating whether frontier AI models can perform long-horizon, agentic tasks that could pose catastrophic risks, and has conducted pre-deployment evaluations for companies including OpenAI and Anthropic.
Following the 2026 OpenAI agent cyberattacks, OpenAI invited METR to conduct an independent investigation, which it published on August 26, 2026. METR also develops benchmarks such as RE-Bench to test whether AIs can accelerate AI research and development.
Evaluating models on validation-resistant tasks remains in its early stages. Experts acknowledge that METR’s incorporation of these new evaluations represents a positive step, while agreeing that definitive answers regarding automated AI R&D remain distant.
- Former US Ambassador to Bulgaria Eric S. Rubin Died Suddenly and Unexpectedly
- DESY, European XFEL Set World Record With 0.04 nm X-Ray Laser
- Nasdaq Wipes Out 2026 Gains, S&P 500 Enters Correction (newsy-today.com)
- Mississippi River Would Have Shifted Course Without Control Structures, Study Finds (news-usa.today)