FrontierCS 2: A Benchmark for Computer Science R&D

What happened

FrontierCS 2 (Berkeley, Princeton, Stanford, MIT and others) is a benchmark asking: can AI agents do real computer-science R&D? It has 48 public tasks across 13 domains (ML systems, HPC, databases, security, networking, cryptography, robotics...), plus 100+ private tasks held back.

Each task works like this:

1. Take a published paper and the code its authors released.
2. Delete the paper's contribution from the repo. The deleted code becomes the human reference.
3. The agent gets the working repo and must rebuild the missing piece so it beats the original.
4. It can submit patches anytime and gets a score on a development suite. The final grade uses a held-out suite it never sees.
5. A run passes only if it beats the authors' code on every metric and every workload.

Results over 12 agents: the best, Claude Opus 5.5 (in Claude Code), passes 58% of runs; others range from about 44% down to 8%. 16 tasks were never passed by any run.

Why it works

Most open-ended benchmarks hand the agent a problem statement and grade one program at the end. Real research is different: you inherit a codebase, change one component, test on the paper's workloads, iterate, and get judged against the original. FrontierCS 2 copies that loop and fixes common weaknesses:

• The bar is real: the authors' own implementation, not a rubric or a re-implementation.
• Hidden final evaluation stops agents from overfitting to the visible scores.
• "Pass on everything" catches the common trick of winning the average while losing badly on one workload. The paper notes most runs that beat the reference on the combined score still failed some metric.

The bigger pattern

Benchmarks are moving from "solve a puzzle" to "improve a real system under realistic feedback". The median task is about as hard as what the best agent can handle, so this benchmark should stay useful for a while. It also shows the gap between looking better on average and being better everywhere, which is exactly what code review should catch.

What it means in practice

• Agents can now improve real systems code about half the time on research-grade tasks. That is useful for performance work on well-tested code, with a human judging the result.
• Copy the evaluation design for your own agent use: separate dev and held-out benchmarks, and require no regression on any key workload, not just a better average.
• Being able to design the evaluation harness for AI-written changes is becoming as valuable as writing the change.