Databricks built an internal benchmark to evaluate coding agents on real PRs from their multi-million line codebase. Key findings: open models like GLM 5.2 are competitive with top-tier proprietary models, and harness choice drastically affects cost. The company now routes tasks to cheaper models automatically.
Databricks published results from an internal benchmark that evaluates coding agents on actual pull requests from their multi-million line codebase. The benchmark covers tasks in Python, Go, TypeScript, Scala, and more. They tested models from OpenAI, Anthropic, and open-source, plus different harnesses (Claude Code, Codex, Pi).
Three capability tiers emerged. Top-tier models (Opus 4.8, GPT 5.4, GLM 5.2) scored 85-87% task completion. Mid-tier (Sonnet 5, GPT 5.4 Mini, Haiku 3.5) scored 75-81%. Lower-tier models scored below 70%. The top tier costs $1.28-$1.94 per task; mid-tier costs $0.35-$2.09.
Open models are viable daily drivers. GLM 5.2 scored 86% (statistically tied with Opus 4.8's 87%) at $1.28/task vs Opus's $1.94. Databricks is now deploying GLM for everyday coding tasks.
Token price is misleading. Sonnet 5 is 1.7x cheaper per token than Opus 4.8, but costs $2.09/task (vs Opus's $1.94) because it consumed 1.9x more tokens. Sonnet scored 81% vs Opus's 87%.
Harness choice matters more than expected. Running the same model through different harnesses (Claude Code vs Pi) showed 2x cost differences with identical quality. Pi sent ~3x less context per turn, finishing tasks with fewer runs.
How the benchmark works:
- Select recent, human-written PRs with high-quality tests.
- Summarize the PR intent as a prompt (excluding solution details).
- Hold out test files; let the agent reproduce the code change.
- Run the held-out tests to determine pass/fail (no LLM judge).
- Seal git history to prevent agents from finding the original commit.
Example: A task might ask the agent to "update the config parser to handle YAML anchors" without revealing the exact implementation. The agent must write code that passes the pre-existing test suite.
Databricks now uses Omnigent to automatically route tasks to the most cost-effective model. They plan to expand the benchmark with harder tasks and test every new agent/harness.
Why build your own? Public benchmarks like SWE-Bench have tasks that leak into training data and don't reflect Databricks' polyglot codebase (10+ languages, Scala/Go/Rust/Java/Python, Bazel, Protobuf).
Next steps: Databricks will share how they use intelligent routing in Unity AI Gateway and Omnigent to cut costs while maintaining quality.
