Research Notes

Top ML Agent Benchmarks To Track

7 min read

Snapshot: 2026-06-12.

Purpose: track benchmarks that measure whether agents are becoming useful ML researchers, not just coding assistants.

Answer First

No: not every benchmark has public scores for the requested model slots.

  • Every top-five benchmark has a current public frontier result worth tracking.
  • WeirdML is the only one here with all three requested slots: Chinese model, GPT-5.5, and Fable 5.
  • MLE-bench has a top Chinese-model row, but no official GPT-5.5 or Fable 5 row in the checked leaderboard.
  • RExBench has DeepSeek-R1 and GPT-5 rows, but no GPT-5.5 or Fable 5 row in the checked paper.
  • MLRC-Bench and RE-Bench should keep the requested slots blank until comparable public rows exist.

Do not treat a missing row as a zero score. It means no comparable public result was found in the checked source.

Model Ranking

This is an opinionated, evidence-weighted ranking. Do not average these benchmarks directly: MLRC/RE/REx are closer to actual research, MLE-bench is closer to known ML engineering work, and WeirdML is a compact agentic modeling signal.

Overall

  1. Claude Fable 5 / Claude Opus line

    Best bet if the work is agentic, messy, and research-code-heavy. Fable 5 is the current WeirdML leader, and Claude Opus variants have the strongest direct RExBench and high-end MLE-bench evidence. Caveat: Fable 5 itself still needs direct public rows on MLRC, RE-Bench, RExBench, and MLE-bench.

  2. Gemini 3 Pro / Gemini exp line

    Best evidenced for Kaggle-style ML engineering. Gemini-3-Pro-Preview powers the top MLE-bench rows, and the older gemini-exp-1206 row is still the MLRC-Bench result to beat. Caveat: the MLRC result is old relative to current frontier models.

  3. GPT-5.5 / GPT-5 / Codex line

    Strong runner-up across weird modeling and research-code implementation. GPT-5.5 is second on WeirdML, GPT-5 is competitive on RExBench, and gpt-5-codex has a meaningful MLE-bench row. Caveat: public GPT-5.5 rows are missing from the harder ML research benchmarks.

  4. DeepSeek V3.2 / V4 line

    Best China-friendly ML-engineering bet. Deepseek-V3.2-Speciale is the top Chinese-model row found on MLE-bench. Caveat: the public research-extension and RE-Bench evidence is weaker than the frontier Claude/Gemini/OpenAI evidence.

  5. GLM / Kimi / Qwen Chinese frontier

    Good watchlist tier. GLM-5.1 is the top Chinese-model row found on WeirdML, and Kimi/Qwen variants are worth tracking, but the evidence is not yet as complete across the main ML research benchmarks.

By Job

Actual ML research:

  1. Claude Opus/Fable line
  2. Gemini 3 Pro / Gemini exp line
  3. GPT-5.5 / GPT-5 line
  4. DeepSeek V3.2 / V4 line
  5. GLM / Kimi / Qwen frontier

Implementing known ML work:

  1. Gemini-3-Pro-Preview
  2. Claude Opus 4.6 / 4.5
  3. Deepseek-V3.2-Speciale
  4. gpt-5-codex / GPT-5
  5. Gemini-2.5-Pro

Agentic weird modeling:

  1. Claude Fable 5
  2. GPT-5.5
  3. Claude Opus 4.8 / 4.6
  4. GPT-5.3-codex / GPT-5.4
  5. GLM-5.1

Current Top Scores

BenchmarkCurrent public top to beatRequested score coverage
MLRC-BenchMLAB (gemini-exp-1206): 9.3% average relative improvement to top-human solution.No requested model rows found.
RE-BenchOriginal paper: frontier agents beat human experts at 2h, but humans overtake with longer budgets.DeepSeek-R1 follow-up exists; no GPT-5.5 or Fable 5 row found.
RExBenchOpenHands + Claude 4.5 Opus: 42% final success without hints; 62% with hints.DeepSeek-R1 and GPT-5 rows exist; no GPT-5.5 or Fable 5 row found.
MLE-bench / HighOverall: Famou-Agent 2.0 + Gemini-3-Pro-Preview, 64.44%. High split: AIBuildAI + Claude-Opus-4.6, 46.67%.Chinese-model row exists; no official GPT-5.5 or Fable 5 row found.
Agentic WeirdML / WeirdML v2claude-fable-5 (no thinking): 87.85% average accuracy.Chinese, GPT-5.5, and Fable 5 rows all exist.

The expanded watchlist below does not have score rows yet. It is a shortlist for future tracking, not a completed leaderboard.

Score Cards

1. MLRC-Bench

Best use-case: actual ML research progress.

Current top: MLAB (gemini-exp-1206), 9.3% average relative improvement to top-human solution.

Requested slots:

  • Chinese model: no public row found in the checked paper table.
  • GPT-5.5: no public row found.
  • Fable 5: no public row found.

Why track it: it uses seven ML research competition tasks and objective scoring against top human competition results. This is the cleanest “can it do novel ML research?” signal here.

2. RE-Bench

Best use-case: AI R&D research engineering.

Current top: the original paper reports frontier agents beating human experts at a 2-hour budget, while humans overtake with longer budgets.

Requested slots:

  • Chinese model: DeepSeek-R1 follow-up result exists, comparable to a 28th-percentile human expert at 16h on a six-task RE-Bench variant.
  • GPT-5.5: no public row found.
  • Fable 5: no public row found.

Why track it: it compares agents with human experts in realistic ML research-engineering environments.

3. RExBench

Best use-case: extending existing ML papers and codebases.

Current top: OpenHands + Claude 4.5 Opus, 42% final success without hints and 62% with hints.

Requested slots:

  • Chinese model: OpenHands + DeepSeek-R1, 0% no hints, 0% hints, 10% detailed hints.
  • GPT-5.5: no public row found. Nearby row: OpenHands + GPT-5, 27% no hints, 37% hints, 43% detailed hints.
  • Fable 5: no public row found.

Why track it: it tests whether agents can turn a research-extension idea into working code inside an existing AI codebase.

4. MLE-bench / MLE-bench High

Best use-case: Kaggle-style ML engineering.

Current top: overall, Famou-Agent 2.0 + Gemini-3-Pro-Preview at 64.44%; High split, AIBuildAI + Claude-Opus-4.6 at 46.67%.

Requested slots:

  • Chinese model: ML-Master 2.0 + Deepseek-V3.2-Speciale, 56.44% overall and 42.22% High.
  • GPT-5.5: no official row found. Nearby OpenAI-family rows checked: Thesis + gpt-5-codex, 48.44% overall / 31.11% High; R&D-Agent + gpt-5, 35.11% overall / 22.22% High.
  • Fable 5: no public row found.

Why track it: it is the best-known Kaggle-style ML-agent benchmark, but scores depend heavily on scaffold and runtime.

5. Agentic WeirdML / WeirdML v2

Best use-case: weird data/modeling intuition.

Current top: claude-fable-5 (no thinking), 87.85% average accuracy.

Requested slots:

  • Chinese model: glm-5.1, 57.10% average accuracy.
  • GPT-5.5: gpt-5.5 (xhigh), 84.91%; gpt-5.5 (high), 83.90%.
  • Fable 5: claude-fable-5 (no thinking), 87.85%.

Why track it: it is compact and practical for checking whether a model can understand strange datasets, design features, iterate, and debug.

Top 5 To Track

  1. MLRC-Bench - best for actual ML research progress; very unsaturated.
  2. RE-Bench - best for AI R&D research engineering; very unsaturated.
  3. RExBench - best for extending ML papers/codebases; very unsaturated.
  4. MLE-bench / MLE-bench High - best for Kaggle-style ML engineering; unsaturated but scaffold-heavy.
  5. Agentic WeirdML / WeirdML v2 - best quick signal for weird modeling intuition; semi-saturated but useful.

Concise Tracking List

MLRC-Bench
RE-Bench
RExBench
MLE-bench
MLE-bench High split
Agentic WeirdML
WeirdML v2
FML-bench
MLGym-Bench
MLR-Bench
ReX-MLE

Expanded Tracker

MLRC-Bench - ML research competitions; objective novelty/progress signal; very unsaturated.
RE-Bench - AI R&D research-engineering environments; human comparison; very unsaturated.
RExBench - implement extensions to existing ML papers/codebases; very unsaturated.
MLE-bench - Kaggle-style ML engineering; scaffold/model/tooling benchmark; still hard.
MLE-bench High - harder split of MLE-bench; better frontier signal than aggregate score.
Agentic WeirdML - weird ML tasks with tool-use/iteration; good quick practical signal.
WeirdML v2 - nonstandard ML tasks; useful but becoming more saturated.
FML-bench - fundamental ML research problems; good for search/exploration strategy tracking.
MLGym-Bench - open-ended AI research tasks across CV/NLP/RL/game theory; good framework benchmark.
MLR-Bench - 201 open-ended ML research tasks; broad, but judge-dependent.
ReX-MLE - medical-imaging ML-engineering benchmark; domain-specific, very hard.

Future AI Researcher Signal

If the only question is “does this predict future AI researcher capability?”, track these first:

MLRC-Bench
RE-Bench
RExBench
FML-bench
MLE-bench High

Track WeirdML too, but do not rank it above those five for frontier progress. It is more compact and closer to saturation.

Refresh Protocol

  • Update the score tracker from primary sources only.
  • Keep agent scaffold and model name together; model-only comparisons are misleading.
  • Separate no-hint, hinted, and detailed-hint scores.
  • Prefer harder splits and human-comparison metrics over aggregate leaderboard numbers.
  • Do not fill GPT-5.5 or Fable 5 slots from unrelated benchmarks.

Sources