Snapshot: 2026-05-23.
The useful way to track coding-agent state of the art is not “which model is best at coding?” It is:
- Can it modify a real repo and pass tests?
- Can it operate in a terminal without hand-holding?
- Can it handle unfamiliar languages and frameworks?
- Can it build product-grade apps rather than toy diffs?
- Does the score survive better tests, hidden evals, or newer tasks?
Benchmark Map
SWE-bench family
SWE-bench is still the default reference point for autonomous code repair. The official site tracks several leaderboards: Full, Verified, Lite, Multilingual, and Multimodal.
Use it for:
- real GitHub issue resolution
- patch generation against existing repos
- comparing agent scaffolds and base models
Important caveat: SWE-bench Verified is a 500-instance human-filtered subset, and the official metric is percent resolved. That makes it legible, but also easy to overfit as it gets popular.
Terminal-Bench
Terminal-Bench tests whether an agent can actually use a shell. Terminal-Bench 2.0 has 89 tasks across software engineering, machine learning, security, data science, and system work.
Use it for:
- CLI competence
- debugging and build workflows
- file-system and system-administration tasks
- “can this thing work like a real terminal coding agent?”
This is closer to how Claude Code, Codex CLI, OpenCode, and similar agents actually feel in use.
Aider Polyglot
Aider’s polyglot benchmark tests model-driven code editing on 225 Exercism tasks across C++, Go, Java, JavaScript, Python, and Rust.
Use it for:
- edit accuracy
- multi-language code changes
- whether a model follows patch/edit format reliably
It is less agentic than SWE-bench or Terminal-Bench because it focuses on code edits rather than long-horizon repo work, but it is still useful for measuring raw editing reliability.
SWE-PolyBench
SWE-PolyBench expands repository-level evaluation beyond Python. It has 2,110 instances from 21 repositories across Java, JavaScript, TypeScript, and Python, covering bug fixes, feature additions, and refactors.
Use it when a model looks good on Python-heavy benchmarks but you care about real product stacks.
SWE-Bench Mobile
SWE-Bench Mobile tests agents on an industrial iOS codebase with Swift/Objective-C, PRDs, Figma designs, and comprehensive tests. The paper reports that the best configurations reached only 12% task success.
This is important because it shows how far the frontier still is from product-grade mobile engineering.
Key lesson: agent design matters as much as model choice. The same model can vary massively depending on the coding agent harness, context system, tools, and prompts.
SWE-WebDevBench
SWE-WebDevBench evaluates “vibe coding” and app-builder platforms as virtual software agencies. It looks beyond code correctness into requirements, architecture, production readiness, security, ops, and app modification.
Use it for:
- full-stack app builders
- product-readiness claims
- frontend/backend integration
- security and deployment realism
The important signal is not just “did it create an app?” but whether the app is actually maintainable, secure, and business-ready.
What I Would Trust
Strong signal:
- SWE-bench Full/Verified for repo issue repair
- Terminal-Bench for real terminal autonomy
- SWE-PolyBench for non-Python and product-stack breadth
- SWE-Bench Mobile for hard industrial mobile work
- live internal evals on your own repos
Weak signal:
- HumanEval-style toy functions
- vendor screenshots of isolated demos
- single benchmark claims without cost, retry policy, tool scaffold, or pass/fail logs
- rankings that do not separate model capability from agent harness capability
Current Takeaway
Coding agents are dope now, but the benchmark story is fragmented:
- SWE-bench tells you whether an agent can patch real GitHub issues.
- Terminal-Bench tells you whether it can survive in a shell.
- Aider Polyglot tells you whether the model can make clean edits across languages.
- SWE-PolyBench tells you whether Python scores transfer to real product stacks.
- SWE-Bench Mobile and SWE-WebDevBench show the frontier is still weak on full product engineering.
The practical SoTA question is:
Which agent + model + context system + tool harness solves my repo’s tasks with low regression risk?
That is why production evals should track not only pass rate, but also:
- cost per accepted patch
- tests run and failure recovery
- regression rate
- number of tool calls
- human review time
- whether the agent can explain and revert its changes