Google has overhauled Android Bench, its AI coding leaderboard, by adopting the standardized Harbor framework to enable secure, sandboxed evaluations. In the resulting re-scores, Anthropic’s Claude Fable 5 claimed the top spot with 84.5% accuracy, while Google’s Gemini 3.1 Pro lagged in fifth place, signaling a gap in native Android development capabilities.
By migrating Android Bench to the GitHub-integrated Harbor framework, the company has moved away from opaque internal testing and toward a transparent, reproducible sandbox environment.
When the rules changed, the baseline shifted, and Gemini didn't just slip—it cratered in specific efficiency metrics.
The Accuracy Gap: Fable 5 and the Cost of Precision
The new hierarchy is stark. Anthropic’s Claude Fable 5 has established a dominant lead with an 84.5% accuracy score. OpenAI’s GPT-5.5 follows closely at 80.2%.
But precision isn't free. Running the standard 100-problem, 10-run benchmark costs over $130 for Fable 5 and GPT-5.5. In contrast, Gemini 3.1 Pro—despite its fifth-place ranking—only cost $87 for the same workload.
While designed for speed, Flash proved utterly incapable of handling the 100-problem evaluation dataset efficiently. It clocked a brutal 28-hour runtime and racked up $165 in operational costs.
| Model | Accuracy Score | Operational Cost (100-prob/10-run) | Performance Note |
|---|---|---|---|
| Claude Fable 5 | 84.5% | >$130 | Current Leader |
| GPT-5.5 | 80.2% | >$130 | Strong Runner-up |
| Gemini 3.1 Pro | (5th Place) | $87 | Cost-efficient but less accurate |
| Gemini 3.5 Flash | (Low) | $165 | 28-hour runtime failure |
Why the Harbor Framework Changes the Game
The adoption of the Harbor framework is the real story here. Harbor introduces secure sandbox environments.
By opening the repository on GitHub, Google is allowing the global developer community to submit custom tasks.
It effectively strips away the marketing fluff.
The Strategic Fallout for Agentic Workflows
Google is currently pivoting its core engineering toward autonomous, agentic development. However, these results show that Google's native models are struggling on their own home turf.
The 30-Second Verdict
By integrating with GitHub and Harbor, Google has turned Android Bench into the definitive, unbiased destination for AI code evaluation.
