Snapshot: 2026-09-29. Scores and coverage below come from the supplied snapshot; check the live maintainer leaderboards before refreshing them.
Track benchmarks that measure progress toward autonomous ML and scientific researchers, not generic coding agents. Keep a benchmark only when it is unsaturated, has a public GPT-6 Astra result and a public Claude Opus 5.5 or Claude Fable 5.1 result, tests research capability, and has scores that can be refreshed repeatedly.
Strict tracker
| Benchmark | GPT-6 Astra | Claude Opus 5.5 | Claude Fable 5.1 | Research signal |
|---|---|---|---|---|
| WeirdML v3 | 42.2% | 31.2% | 26.0% | Discover structure in unfamiliar ML/data tasks |
| Terminal-Bench-Science 0.1 | 68.1% | 63.3% | 40.0% | Execute scientific workflows with tools |
| MLS-Bench-Lite | 50.0 | — | 50.3 | Invent ML methods that generalize |
These are not directly comparable scores: they use different tasks and scales. The MLS-Bench-Lite rows are on a human-SOTA reference scale, not percentages. Keep the model, harness, effort, and benchmark version attached to each score.
WeirdML v3 — discovery
Agents explore unfamiliar datasets, discover latent structure, build analysis pipelines, and iterate with limited feedback. A frontier score of 42.2% leaves substantial room. It is a compact test of modeling intuition rather than recognition of a standard benchmark.
Leaderboard rows: GPT-6 Astra 42.2%; Claude Opus 5.5 31.2%; Claude Fable 5.1 26.0%.
Terminal-Bench-Science 0.1 — execution
70 expert-curated, executable-test tasks span life, physical, mathematical, engineering, and earth sciences. Agents must use data, software, and terminal tools to complete scientific workflows. The leading system still fails roughly a third of tasks.
Live-leaderboard configurations: GPT-6 Astra (max, Codex) 68.1%; Claude Opus 5.5 (max, Claude Code) 63.3%; Claude Fable 5.1 (max, Claude Code) 40.0%. Do not mix these with vendor launch-table runs using different harnesses or evaluation settings.
MLS-Bench-Lite — invention
The full MLS-Bench has 140 tasks in 12 ML-research domains; the official 30-task Lite subset compares frontier systems. Given research scaffolds and strong baselines, agents must design algorithmic improvements that transfer across seeds, datasets, settings, or scales. Current agents remain far from reliably exceeding human-designed methods.
Lite leaderboard: Claude Fable 5.1 + Claude Code max 50.3; GPT-6 Astra + Codex max 50.0. Reference rows: Claude Fable 5 49.9; Claude Opus 5 49.8. Fable 5.1 satisfies the Anthropic coverage rule even without an Opus 5.5 row.
Removed from the strict tracker
| Benchmark | Reason |
|---|---|
| WeirdML v2 | Superseded by v3; approaching saturation for frontier systems. |
| MLE-bench / MLE-bench High | Useful ML engineering, but no clean comparable public GPT-6 + Opus 5.5/Fable 5.1 coverage. MLE-Bench Revised has a GPT-6 result near saturation but no matching Anthropic row. |
| MLRC-Bench, RE-Bench, RExBench | Relevant research benchmarks, but no verified public pair for the required current models. |
| FML-bench, MLGym-Bench, MLR-Bench, ReX-MLE | No verified required frontier-model coverage; ReX-MLE is also domain-specific. |
A missing public result is not a zero. Re-add a benchmark when a comparable current-model pair appears and the task remains unsaturated.
Refresh protocol
- Require public GPT-6 Astra and Claude Opus 5.5 or Fable 5.1 rows on the same version.
- Prefer live maintainer leaderboards over vendor tables; record model + harness + reasoning effort together.
- Do not mix versions or evaluation configurations, or compare scores across benchmarks as one scale.
- Drop benchmarks near effective saturation. Prefer hidden-test, executable, or objective scoring and tasks requiring exploration and iteration over static QA.
- Recheck the removed list when new comparable rows appear.
Together the tracker covers discovery → scientific execution → algorithmic invention.