<>
OpenAI updated several evaluation benchmarks for its GPT-6 Astra model shortly after a delayed publication on September 3. Adjustments shifted metrics for Astra and competitors like Anthropic.
The Bottom Line
- Benchmark Volatility: OpenAI altered core metrics—including hallucination rates and math scores—multiple times within hours of the GPT-6 Astra rollout.
- Competitive Impact: Adjustments briefly depressed rival scores, such as Anthropic’s Fable 5.1 model, before figures were partially reverted.
Decoding the Rollout Delays and Shifting Specs
When OpenAI pushed its long-awaited GPT-6 Astra announcement live mid-afternoon on September 3, the deployment faced immediate technical friction. Originally slated for a 2 p.m. ET release, the company delayed the post’s broad availability by nearly two hours due to what executives initially attributed to a content management system bug and subsequent internet outages. But the logistical hiccups were only the beginning of the anomaly.
Here is the math. By the time the blog post stabilized online, the evaluation metrics embedded within the text had already undergone significant revisions. According to internet archive snapshots, Astra’s reported hallucination rate sat at 4.2% across the first five snapshots up to 3:11 p.m. ET. By the sixth snapshot at 5:20 p.m., that figure had been halved to 2%. Scores for its predecessor, GPT-5.6 Sol, dropped simultaneously from 12.2% to 9.4%. As of this writing, those hallucination rates have cycled back to their original baselines.
Other metrics saw similar turbulence. An embargoed pre-publication draft reviewed by media organizations listed Astra’s score on the ARC-AGI-3 evaluation at 98.6%. On the live blog, that figure jumped to 99.99%. Meanwhile, internal scores for GPT-5.6 Sol on the ExploitBench cybersecurity evaluation swung from 5.5% to 11.5% before OpenAI announced it was investigating a reversion, admitting the higher score reflected a reasoning level not yet commercially available for that model.
The Mechanics of Benchmaxxing and Industry Skepticism
The practice of optimizing performance under specific test conditions—colloquially known in the artificial intelligence sector as “benchmaxxing”—has cast a long shadow over commercial model releases. Independent researchers have repeatedly flagged the malleability of standardized tests when executed under maximal effort harnesses rather than standard production constraints.
“This can be done in a very tight timeframe, and it’s better for their marketing,” noted Anka Reuel and Mike Hardy, researchers at the Stanford Intelligent Systems Laboratory and Stanford Trustworthy AI Lab, pointing out the scarcity of granular methodology in system cards.
Discrepancies extended beyond OpenAI’s proprietary ecosystem to affect rival architectures. Anthropic’s Fable 5.1 model saw its FrontierMath Tier 4 score drop nearly 10 percentage points from 87.8% down to 78% in afternoon snapshots, before recovering to 83%. OpenAI representatives defended the adjustments as standard operational procedure.
“We care deeply about getting evaluations right,” an OpenAI spokesperson told Fortune. “Most evaluations have noise within a few percentage points based on the exact checkpoint, scaffold, and evaluation run used in reporting. For our launch blog, we made fixes to ensure the numbers represent our best estimate of available model performance, so that users can make meaningful comparisons.”
Comparative Evaluation Snapshot
| Model | Benchmark | Initial Published Score | Revised/Current Score |
|---|---|---|---|
| OpenAI GPT-6 Astra | Hallucination Rate | 4.2% | 2% (Briefly), Reverted to 4.2% |
| OpenAI GPT-5.6 Sol | ExploitBench Cybersecurity | 5.5% | 11.5% (Pending Reversion) |
| Anthropic Fable 5.1 | FrontierMath Tier 4 | 87.8% | 78% (Trough), Recovered to 83% |
| OpenAI GPT-6 Astra | ARC-AGI-3 Evaluation | 98.6% (Draft) | 99.99% (Live) |
Financial Stakes and Enterprise Procurement Realities
For institutional investors and corporate buyers evaluating enterprise-grade artificial intelligence deployments, benchmark volatility introduces friction into capital allocation decisions. When metrics shift post-publication without clear changelogs, parsing genuine performance gains from marketing optimization becomes increasingly difficult.
Vincent Sunn Chen, an AI engineer at Snorkel AI, noted that shifting metrics in the final hours of a launch cycle are tied to non-deterministic grading configurations and changing compute allocations. However, he emphasized that establishing clearer industry standards for reporting version changes is vital for empirical analysis.
As competition intensifies and major players position themselves for potential public listings—including market speculation surrounding an OpenAI IPO expected by 2027—clarity in performance reporting remains paramount. Until standardized, immutable auditing practices take hold across the generative AI landscape, enterprise customers will need to independently verify model capabilities in production environments rather than relying solely on launch-day leaderboards.
Disclaimer: The information provided in this article is for educational and informational purposes only and does not constitute financial advice.
