Key facts
- Google has updated its Android Bench benchmark for evaluating large language models in Android app development.
- The benchmark now includes eight new models, such as Claude Fable 5, Claude Sonnet 5, and Gemini 3.1 Pro.
- Claude Fable 5 leads the updated leaderboard with 84.5 percent accuracy.
- Google's Gemini 3.1 Pro is now in fifth place.
- The benchmark now incorporates metrics for cost and efficiency, with Fable 5 and GPT 5.5 being the most expensive.
- Google is adopting the Harbor framework to facilitate developer contributions to Android Bench.
Google has significantly updated its Android Bench, a benchmark designed to evaluate the performance of large language models (LLMs) in Android app development tasks. The benchmark now features eight new models, including prominent ones like Claude Fable 5, Claude Sonnet 5, and Google's own Gemini 3.1 Pro.
In the updated leaderboard, Claude Fable 5 has emerged as the top performer with 84.5 percent accuracy, surpassing Google's Gemini 3.1 Pro, which has slipped to fifth place. Other models like OpenAI's GPT 5.4 and Claude Sonnet 5 also rank higher than Gemini.
Beyond accuracy, the benchmark now also considers cost and efficiency. Models like Fable 5 and GPT 5.5 incurred costs exceeding $130 for the 100-problem, 10-run benchmark due to token usage, while Gemini 3.1 Pro was more cost-effective at $87. Gemini 3.5 Flash, intended for lower cost, proved the most expensive at $165 per run with a 28-hour runtime.
This development is significant for Google as it shifts towards agentic development, aiming to encourage developers to use its tools. To foster community involvement and continuous improvement, Google is transitioning Android Bench to the Harbor framework, a testing sandbox that simplifies running, evaluating, and sharing results. Developers are invited to contribute their own benchmarks and tasks.