← Back to summary
FrontierCS 2: A Benchmark for Computer Science R&D
Converted from PDF with markitdown. Layout may be rough.
FrontierCS
| FrontierCS | 2: | A Benchmark | for Computer |
| ---------- | --- | ----------- | ------------ |
| Science | R&D | | |
FrontierCSTeam
6202 tcO 12 ]GL.sc[ 1v23992.0162:viXra
| (cid:128) Website | | | § Code |
| ----------------- | --- | --- | ---------------------------------- |
| frontier-cs.org | | | github.com/FrontierCS/FrontierCS-2 |
Abstract
CanLLMagentsmakecontributionstofrontiercomputerscienceR&D?WeintroduceFrontierCS2,a
benchmarkof48open-endedtasks. Wetakeapaperandthecodeitsauthorsreleasedanddeletethe
paper’scontribution. Thedeletedcodebecomesthehumanreferencethatsetsthebar. Earlieropen-ended
benchmarksgiveeachagentaproblemstatementandgradeoneprogramaftertherun. FrontierCS2
placestheagentinsidearesearchenvironment. Theagentinheritsaworkingrepositoryandmaysubmita
patchatanypoint. Foreachpatch,ajudgereturnsadevelopment-suitescorebutkeepstheevaluator,the
held-outfinalsuiteandthereferenceprivate,andgradesthefinalsubmissiononthatsuiteagainstthe
reference. Thetasksspan13domains,frommachine-learningsystemsandhigh-performancecomputing
todatabases,securityandrobotics. Beyondthese48tasks,FrontierCS2keepsaprivatesetofmorethan
100tasks,tobeannouncedinthefuture. Weevaluate12agents. ThebestoneisClaudeOpus5.5,passing
58%ofitsruns. Norunpasseson16tasks. Ahigherscoreisnotthesameasapass:mostrunsthatbeatthe
referenceonthecombinedfinalscorestillfallshortonsomemetricorworkload. Onasharedcapability
scaleforagentsandtasks,themediantaskisroughlyasdifficultaswhatthestrongestagentcanhandle.
Programming Languages4
| MLSys 6 | HPC | 4 Machine Learning | 9 |
| ------- | --- | ------------------ | --- |
1gnireenignE erawtfoS
Cryptography 3
Database 9
| Architecture 3 | Network3 | Theory | |
| -------------- | -------- | ------ | --- |
2
| × = | | | |
| --- | --- | --- | --- |
Bioinformatics scitoboR
Security 2
Figure1: FrontierCS2. Eachtaskremovesapaper’skeycontribution;agentsmustbeattheoriginalcode.
Author list
| QiuyangMang*,† | WenhaoChai*,† | RunyuanHe* | DongruixuanLi* |
| -------------- | ------------- | ---------- | -------------- |
UCBerkeley PrincetonUniversity UCBerkeley UniversityofWaterloo
| ZihanZheng* | YujunHe* | | |
| ----------------- | ---------------------------- | ----------- | ------ |
| NewYorkUniversity | GeorgiaInstituteofTechnology | | |
| XuanyiZhou | HaoWang | SiddhantRay | BoPeng |
UCBerkeley UCBerkeley UniversityofChicago PrincetonUniversity
| RuyiJi | ShuoYang | HaochengXi | BoruChen |
| -------------------- | --------------- | ------------ | -------------------- |
| UniversityofMichigan | UCBerkeley | UCBerkeley | UCBerkeley |
| XinyangDavidHan | MelissaPan | MayankMishra | BoxiYu |
| UCBerkeley | UCBerkeley | UCBerkeley | UniversityofLimerick |
| SijiaLiu | KeremAkillioglu | YiPan | ZiruiWang |
| PrincetonUniversity | UCBerkeley | UCBerkeley | UCBerkeley |
| MertCemri | DiogoAntunes | YichuanWang | AiweiIvyZhang |
UCBerkeley UCBerkeley UCBerkeley GeorgiaInstituteofTechnology
| ZhenyangChen | QizhengZhang | KaiyuanLiu | WenyuanHuang |
| ------------ | ------------ | ---------- | ------------ |
GeorgiaInstituteofTechnology StanfordUniversity UniversityofWashington UniversityofWisconsin-Madison
| SherryRuan | ShuLiu | DachengLi | ZeruiCheng |
| ------------------- | ----------- | ---------------------------- | ------------------- |
| StanfordUniversity | UCBerkeley | UCBerkeley | PrincetonUniversity |
| ZeyuShen | ShangZhou | HanfeiChen | YuzhongZhang |
| PrincetonUniversity | UCSD | GeorgiaInstituteofTechnology | UNSW |
| BohanLyu | QuAo | HanZheng | ChongheJiang |
| UCBerkeley | MIT | MIT | MIT |
| JingzhuoHu | XiaokunChen | KunjunLi | ZhiqingZhong |
UniversityofPennsylvania HarborFramework UCBerkeley AaltoUniversity
HanchenLi†
| HaoyeTian | TongZhao | YunzhongHe | |
| --------------- | -------- | ---------- | ---------- |
| AaltoUniversity | ScaleAI | ScaleAI | UCBerkeley |
ADVISORS
| OfirPress | JiantaoJiao | KunleOlukotun | KarthikR.Narasimhan |
| ----------- | ---------------- | ------------------ | ------------------- |
| | NVIDIA | StanfordUniversity | PrincetonUniversity |
| JingboShang | KoushikSen | MateiZaharia | AlexDimakis |
| UCSD | UCBerkeley | UCBerkeley | UCBerkeley |
| IonStoica | JosephE.Gonzalez | AlvinCheung | |
| UCBerkeley | UCBerkeley | UCBerkeley | |
CORRESPONDENCE
| †qmang@berkeley.edu | wenhao.chai@princeton.edu | lihanc@berkeley.edu | |
| ------------------- | ------------------------- | ------------------- | --- |
2
Pass rate
58
44
40
35
25
21 21
15 12 12 12
8
Claude GPT-6 Gemini 4 GPT-6.1 GLM 5.3 Kimi K3 Muse DeepSeek Muse Qwen 3.8 GLM 5.3 Step 5
Opus 5.5 Astra Argon Sol ZCode Kimi Code Wonder V4.1 Spark 1.3 Max Flash Preview
Claude Codex CLI Pi Codex CLI 1.0 Flash Muse Qwen ZCode Step Code
Code Muse DeepSeek Code Code
Code Harness
Figure2: Passrate. Shareofeachagent’srunswhosefinalsubmissionbeatsthehumanreferenceonevery
metricandheld-outworkload;theharnessisingray.
1 Introduction
Consideraresearcherwhopicksupapublishedsystemtomakeitbetter. Theyinheritaworking
codebase, change one component, test the change on the workloads the paper reported, and
are judged against the paper’s own implementation. Much of research and development in
computersciencelookslikethis. WeaskwhetheranLLMagentcandothesame.
Existingopen-endedbenchmarksansweranarrowerquestion. OnFrontierCS[1],ALE-Bench[2]
andAtCoderheuristiccontests[3],agentsstaybelowthetophumanexperts. Thesebenchmarks
share one design: the agent receives a problem statement and writes one program, and an
evaluatorscoresitonce,aftertherunends.
Benchmarks that do place the agent in a real environment are narrow in another way. SWE-
bench [4] and MLE-bench [5] grade against tests or a leaderboard. ResearchGym [6], MLS-
Bench [7] and PaperBench [8] stay within machine learning, and those that span other fields
gradeagainstarubricorareimplementationratherthantheauthors’owncode.
Inthispaper,weevaluateagentsagainstthepublishedcodeitself,insidetherepositoryitcame
from,acrosscomputerscience. FrontierCS2isthesecondversionofFrontierCS(Table1). We
takethecodeapaper’sauthorsreleased,deletethepaper’scontribution,andasktheagentto
rebuilditsothatitbeatstheoriginal(Figure1). Wecallthedeletedcodethehumanreferenceand
measureitonthepaper’sownworkloads(Section3). Theagentworksasaresearcherwould. It
editsthecode,submitsapatchwheneveritlikes,andgetsbackascoreonadevelopmentsuite
fromajudge. Theevaluator,theheld-outfinalsuiteandthehumanreferencestaywiththejudge
(Section4). Attheend,theagentnamesonepatchasfinal,andthejudgegradesthatpatchon
theheld-outsuite. Arunpassesonlywhenthepatchbeatsthehumanreferenceoneverymetric
3
Table1: FrontierCSandFrontierCS2. Eachtaskmovesintoapaper’sownrepository,withtheauthors’
codeasthebar.
| | FrontierCS | FrontierCS2 |
| --- | ---------- | ----------- |
Task Aproblemstatement Arepositorywithapaper’scontributiondeleted
| Output | Oneprogram | Apatch |
| -------- | -------------- | ----------------------------------- |
| Bar | Noknownoptimum | Theauthors’code,onheld-outworkloads |
| Feedback | None | Adevelopmentscoreforeverypatch |
Domains Algorithmsandsixresearchareas 13computer-sciencedomains
| Metric | Meanscore | Passrate,Meanscore,andFCI |
| ------ | --------- | ------------------------- |
andeveryheld-outworkload.
Weevaluate12models,eachpairedwithoneagentharness,withonerunpertask. Thebestagent,
Claude Opus 5.5 in Claude Code, passes 58% of its runs, and every other agent passes fewer
thanhalfofitsruns(Figure2). Noagentpassesanyof16tasks. Tocompareagentsandtasks
ononescale,wefittheFrontierCSCapabilitiesIndex(FCI)togetherwithamatchingdifficulty
index. Themediantasksitsataboutthestrongestagent’sFCI(Section5.2). Theleadingagent
alsochangeswiththecostbudgetperrun(Section5.3).
Insummary,ourcontributionsare:
• Tasksbuiltbysubtraction: wedeleteapaper’scontributionfromitsrepositoryandkeepthe
deletedcodeasthebar(Section3).
• Ajudgethatscorespatchesduringarunandkeepstheevaluator,theheld-outsuiteandthe
referenceprivate(Section4).
• 48tasksin13domains,with12agentsevaluated(Section5)andthreecasestudies(Section6),
andaprivatesetofmorethan100tasks,tobeannouncedinthefuture.
4
Contents
1 Introduction 3
2 Relatedwork 6
3 Taskbuilding 7
4 Judge 8
5 Evaluationresults 9
5.1 Experimentalsetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.2 FrontierCSCapabilitiesIndex(FCI) . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5.3 Test-timescaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
6 Casestudies 11
6.1 Twochecksumsshrinkprivatelookups . . . . . . . . . . . . . . . . . . . . . . . . 11
6.2 Twofasterclocksforapacketscheduler . . . . . . . . . . . . . . . . . . . . . . . . 12
6.3 Ameasuredbudgetforsparsevideoattention . . . . . . . . . . . . . . . . . . . . 14
7 Conclusion 15
A Tasksources 24
B Taskresources 26
C Agentsettings 27
D ComputingFCIandFDI 27
5
2 Related work
Open-endedcodingbenchmarks. Unlikecompetitionbenchmarkswithknownsolutions,such
as LiveCodeBench Pro [9], benchmarks that score a program on an objective with no known
optimumnowspanheuristiccontests[2],combinatorialoptimization[10,11],GPUkernels[12],
and algorithmic and research problems [1]. Speed-up benchmarks start from a codebase and
measure against a fixed baseline: the human nanoGPT speedrun record [13, 14] or a naive
PyTorchinferenceserver[15]. FrontierCS2insteadgradesmostofitstasksagainstthehuman
reference,codethepaper’sauthorsreleased,andscoreseachpatchduringtherun.
Agentbenchmarksforresearchandengineering. Asecondfamilyscoreswhatanagentpro-
duces inside an environment. SWE-bench [4] and Terminal-Bench [16] grade with held-out
tests,MLE-bench[5]withKaggleleaderboards,andRE-Bench[17]againsthumanexpertsgiven
the same time budget. PostTrainBench [18] measures the downstream score of a post-trained
model,andDeployBench[19]checkswhetheraresearchartifactrunsitsdesignatedexperiment.
EdgeBench [20] also gives the agent feedback during a run. PaperBench [8] grades replica-
tionfromscratchwitharubric,whileAIRS-Bench[21]andResearchGym[6]compareagainst
published machine-learning results. RSI-Forge [22] turns papers from many fields into tasks
whosebaselineisanagent’sreimplementationofeachpaper’smethod. IntotheUnknown[23]
grades follow-up papers on unpublished directions against expert criteria, and ReuF2F [24]
posesformalizedundergraduateresearchproblems. Noneofthesebenchmarkssetsitsbarby
measuring the original authors’ code. MLS-Bench [7] is the closest in design: each of its 140
machine-learningtasksacross12areasaskstheagenttoimproveonenamedcomponent. We
extendthatdesignbeyondmachinelearningto13computer-sciencedomains,fromdatabasesto
cryptography.
Buildingtasksfromexistingartifacts. Methodsthatbuildtasksatscalestartfromanartifact
that already works and break it in a controlled way. SWE-bench pairs a pull request with
the tests it fixed, and SWE-smith [25] injects bugs into a repository so that existing tests fail.
FrontierSmith[26]mutatesclosedcompetitive-programmingproblemsintoopen-endedones,
and BenchEvolver [27] evolves solved problems into harder variants. DiscoGen [28] instead
generates algorithm-discovery tasks procedurally. Like ResearchGym, we remove a paper’s
methodfromitsrepository. Unlikeit,ouroracleistheremovedcodeitself,andourtasksspan13
domains.
6
FrontierCS 2 task building
Delete the contribution
Agent workspace
Keep aside Measure
Paper and repository Human reference Pass thresholds
Add held-out workloads
Held-out final suite
Figure3: FrontierCS2taskbuilding. Wedeletethepaper’scontributionfromitsownrepositoryandgive
theagentwhatremains. Thejudgekeepsthedeletedcodeasthehumanreference,thepassthresholds
measuredfromitonthepaper’sworkloads,andafinalsuitewithheld-outworkloads.
3 Task building
Webuildthetasksbysubtractionfromthecodethepaper’sauthorsreleased(Figure3andTa-
ble2).* Welocatethecodethatimplementsthepaper’scontribution,deleteit,andleaveevery-
thingitdependsonunchanged. TakeSVG-EAR[29],whichwebuildfromtheSparseVideoGen
repository†. Wedeletetheerror-awareattentionclassandthreeofitsroutines. Theseroutines
averageeachcluster,choosetheclusterpairstocompute,andcomputethosepairswhileesti-
matingtherest. Theagentreceivesthisrepositorywiththedeletedfunctionsstilldeclaredand
called,anditmustwritethem. Itmayeditonlythefilesthatheldthecontribution. Thehuman
referenceisaprivatepatchthatrestoresthedeletedcode. Subtractiongiveseachtaskbuiltthis
wayanoraclethatalreadyrunsinitsrepositoryonthepaper’sownworkloads.
Weseteachmetric’spassthresholdfromthereference’sownmeasurementsontheworkloads
the paper reported. For SVG-EAR, latency must be at most 0.96 times the reference’s latency,
PSNRmuststaywithin0.25dBofthereferencevalue,SSIMwithin0.01andLPIPSwithin0.005.
Thejudgescoreseverysubmissiononadevelopmentsuiteoftheseworkloads. Onlythefinal
submissionisscoredonafinalsuite,whichaddsheld-outinstancestheagentneversees(different
seeds,queriesorinputgraphs).
*ExceptStructuredLWE,anoriginalcryptanalysischallenge(AppendixA).
†https://github.com/svg-project/Sparse-VideoGen
7
FrontierCS 2 judge
Model API
Evaluator
Patch to the allowed files Held-out final suite
Reference score
Agent Driver Judge
Raw stdout and stderr
Development-suite score
No network Sanitized message
except the driver
Agent GPU Judge GPU Test sandboxes
GPU tasks only GPU tasks only Some tasks only
Figure4: FrontierCS2judge. Eachcoloredboxisasandbox. Theagent’ssandboxreachesonlythedriver.
Theagentseesonlyadevelopment-suitescoreandasanitizedmessageperpatch;allthatdecidesitsresult
stayswiththejudge.
4 Judge
Theagentandthejudgeholdseparateinformation(Figure4). Theagentholdstherepository;
thejudgeholdstheevaluator,theheld-outfinalsuiteandthehumanreference. Theagenthas
fivehoursandmaysubmitapatchtotheallowedfilesatanytime. Thejudgereturnsthepatch’s
scoreonthedevelopmentsuiteandamessagethatcountsthedevelopmentworkloadsmeeting
eachmetric’sthreshold,withoutthereferencescoreortheevaluator’srawoutput.
Theyalsorunapart,eachinitsownsandbox,linkedonlybyadriverstartedfortherun. The
agent’ssandboxreachesnonetworkbutthedriver,whichforwardsitsmodelcallsandpatches,
so it cannot download the authors’ code. On GPU tasks, the agent may start a development
GPUthroughthedriver,whileonlythejudgeusestheGPUthatmeasuresthepatchagainstthe
humanreference. Sometasksalsogivethejudgetestsandboxes: onContinuum[30],avLLM
servingtask,theyrunSWE-benchepisodesagainstthepatchedvLLM.
Development-suitescoresguidetheagentanddonotcounttowarditsresult. Attheendofthe
run,theagentnamesonesubmissionasfinal,andthejudgeevaluatesthatpatchaloneonthe
held-outsuite. Wedonotpickitsbestdevelopment-suitesubmissionforit,sincechoosingthe
patchispartofaresearcher’swork. Wesayasubmissionbeatsthereferenceonametricwhen
itmeetsthatmetric’sthreshold. Arunpasseswhenitsfinalsubmissionbeatsthereferenceon
everymetricacrosseveryheld-outworkload. Thejudgealsoassignsafinalscorefrom0to100
onascaleeachtaskdefines.‡ Arunwithoutasavedfinalsubmissionscores0.
‡Twoexamples.KVCompactionscoreseachofits24articlesastheshareofquestionsansweredcorrectlytimes
0.75+0.25e, where e = min(1,max(0,0.5+log r/6)) and r is the agent’s speedup over the human reference,
10
discountedwhenitscompactedcacheislarger(thetask’sbytesmodeswapstherolesoftimeandsize).Progentscores
eachofitsfoursuitesastheutilityunderattacktimestheshareofattackedepisodesthatarenotbreached.Bothreport
100timesthemean.
8
Pass rate and cost
Pass rate (%)
80
60
Claude Opus 5.5
GPT-6 Astra
40 Gemini 4 Argon
GPT-6.1 Sol
GLM 5.3
20
Kimi K3
Muse Wonder 1.0
DeepSeek V4.1 Flash
GLM 5.3 Flash Qwen 3.8 Max Muse Spark 1.3
0 Step 5 Preview
0.1 0.2 0.5 1 2 5 10 20
Cost per run (US$, log scale)
Figure5: Passrateandcost. Eachpointisoneagent: passrateonthey-axisagainstmeancostperrunin
US$onalog-scalex-axis. GPT-6.1SolpasseslessoftenthanGPT-6Astraandsitsfartoitsleftonthecost
axis.
5 Evaluation results
5.1 Experimentalsetup
Agents. Anagentpairsamodelwithaharness,theprogramthatrunsthemodel’stoolcalls
andsubmitsitspatchestothejudge. Weevaluate12models,eachinoneharness. Wenameeach
agentbyitsmodel,buttheharnessispartofwhatwemeasure. Table4listseachagent’sharness,
version,reasoningeffortandoutputlimit.
Tasksandruns. Arunisoneagentononetask,fromthestartingrepositorytoitsfinalsubmis-
sion. Eachofthe12reportedagentsrunseachtaskonce. TheproviderrefusedtwoGemini4
Argonrunsbeforeafinalsubmission. Likeeveryrunwithoutapassingfinalsubmission,they
countasfailures,inthepassrateandinFCIalike.
Metrics. Ourmainmetricisthepassrate,theshareofanagent’srunsthatpass. Wealsoreport
anagent’smeanscore,itsmeanmarginoverthehumanreference. Thisistheaverageoverits
runsofthefinalscoreminusthereference’sscoreonthesametask(0isatie). Figure5setseach
agent’spassrateagainstitsmeancostperrun. Ahigherfinalscoredoesnotmeanapass,anda
passdoesnotneedahigherscore(Figure6). Mostrunswhosefinalscoreexceedsthehuman
reference’smissathresholdonatleastonemetricorworkload. Conversely,somepassingruns
scoreatorbelowthereference’sfinalscore,sincethefinalscoreisonenumberoneachtask’s
ownscale,whileapassneedseverymetrictoholdatonce.
9
Final score against the human reference
Final score minus the human reference
Claude Opus 5.5 +9.4
GPT-6 Astra +7.1
GPT-6.1 Sol +5.7
Gemini 4 Argon +2.1
GLM 5.3 +1.8
Muse Wonder 1.0 +4.1
DeepSeek V4.1 Flash +0.5
Kimi K3 −1.2
GLM 5.3 Flash −8.4
Muse Spark 1.3 −3.3
Qwen 3.8 Max −6.4
Step 5 Preview −10.6
−100 −80 −60 −40 −20 0 +20 +40 +60
Figure6: Finalscoreagainstthehumanreference. Eachfaintpointisonerun’sfinalscoreminusthe
humanreference’sonthesametask,onerowperagent;thelinespanseachagent’srangeandthedot
marksitsmean. Theverticallineat0marksatie.
5.2 FrontierCSCapabilitiesIndex(FCI)
TheFrontierCSCapabilitiesIndex(FCI)putstheagentsononecapabilityscale,andtheFrontierCS
DifficultyIndex(FDI)putsthetasksonthesamescale. FCIadaptsEpochAI’sEpochCapabilities
Index[31],anitemresponsemodel. Wefitittoper-metricoutcomesandtoscoremarginsover
thehumanreference,ratherthantorawscores.
Onejointfittakestwoobservationsfromeveryrun: whichofthetask’smetricstherunbeatsthe
humanreferenceon,andthemarginofitsfinalscoreoverthereference’s. Themetricoutcomes
tellthefithowhardeachpartofataskis, andthemargintellsithowcloseafailedruncame.
Bothshareonefittedruneffect, whichsayshowfarthatrunlandsaboveorbelowitsagent’s
strength.
Eachagenthasonestrength,whichwereportasitsFCI:tenpointsperlogit,withKimiK2.7at0.
KimiK2.7,runinKimiCode,isathirteenthagentthatservesonlyasthis0point: itsrunsenter
thefit,butwedonotreportit. Atask’sFDIistheFCIatwhichanagentpassesithalfthetime.
ThehumanreferencehasnoFCI,sinceitscodecannotbeatitself. Intervalsare90%bootstrap
intervalsoverthetasks,andAppendixDgiveseachstepofthefit.
ClaudeOpus5.5leadsFCI,followedbyGPT-6AstraandGPT-6.1Sol,andStep5Previewislast
(Figure7);FCIandthepassrateordertheagentsalmostalike. Thetasksspanawiderrangeof
difficultythantheagentsdo: themediantasksitsataboutthestrongestagent’sFCI.Eachtask’s
FDIisuncertain,however,sowedonotorderindividualtasksbyit. FDImeasureshowharda
taskistopass,andacrosstasksitfollowsthepassrateclosely.
10
FrontierCS Capabilities Index
FCI
Claude Opus 5.5 55.7
GPT-6 Astra 50.7
GPT-6.1 Sol 42.6
Gemini 4 Argon 39.5
GLM 5.3 36.6
Muse Wonder 1.0 34.5
DeepSeek V4.1 Flash 30.7
Kimi K3 26.4
GLM 5.3 Flash 22.9
Muse Spark 1.3 22.1
Qwen 3.8 Max 21.9
Step 5 Preview 9.8
0 10 20 30 40 50 60 70
Figure7: FrontierCSCapabilitiesIndex. Eachbarshowsanagent’sFCIfrom0withits90%bootstrap
intervalasawhisker. Theintervalsseparateonlyagentsthatsitfarapartonthecapabilityscale.
5.3 Test-timescaling
Test-time scaling asks what FCI an agent would reach under a cost cap per run. At budget
B, a run that finished within B keeps its held-out result, and a run still going at B counts as
onewithoutafinalsubmission,whichscores0. WiththetaskparametersfixedattheirFCIfit
(Section 5.2), the FCI at B is the strength that best fits these outcomes (Appendix D). A run’s
outcomecanonlyimproveas Bgrows,sonocurvefalls,andeachendsattheagent’sFCI.
Whichagentleadsdependsonthebudget(Figure8). Atthesmallestbudgets,DeepSeekV4.1
Flashleads. Itscostsarelowerbounds,sothisleaddependsonhowwepriceitstokens. Claude
Opus5.5leadsatthelargestbudgets.
6 Case studies
6.1 Twochecksumsshrinkprivatelookups
Privateinformationretrievalletsaclientreadoneentryofadatabaseheldbyaserver,without
theserverlearningwhichentryitread. QuarterPIR[32]makesthischeap. Theclientfirstbuildsa
compactsummaryofthedatabase;afterwards,eachlookupcostsoneshortrequestandoneshort
reply. Inourtasktheagentrewriteshowtheserveranswersrequests,anditmayalsochangethe
messagesandtheclient. Arunpassesifeverylookupreturnstherightentryandtherunbeats
thehumanreferenceonthebyteseachlookupsends,oneverytiming,andonthememorythe
clientkeeps.
11
Test-time scaling
FCI
60
Claude Opus 5.5
GLM 5.3
GPT-6 Astra
40 GPT-6.1 Sol
Gemini 4 Argon
DeepSeek V4.1 Flash Muse Wonder 1.0
Kimi K3
20 Muse Spark 1.3
Qwen 3.8 Max
GLM 5.3 Flash
Step 5 Preview
0
0.1 0.2 1 10
Cost budget per run (US$, log scale)
Figure8: Test-timescaling. FCIofeachagentagainstthecostbudgetperrun,inUS$onalogscale;arun
stillgoingatabudgetcountsasonewithoutafinalsubmission. Eachcurvestartsatthefirstbudgetat
whichitsFCIisatleast0andendsattheagent’sFCI.Twoslantedstrokesmarkanemptybudgetrange
skippedontheaxis. Theleaderchangeswiththebudget.
Inthehumanreference,everyreplycarriesvaluestheclientrarelyneeds. Alongwiththeanswer,
the server returns one parity value for each group of the database. The client needs only one
of them, to correct the hint it refreshed after its previous lookup. Asking for that one alone,
however,wouldtelltheserverwhereitspreviouslookupwent. MuseWonder1.0replacesthese
values with twochecksums, the construction that lets aRAID-6 disk array rebuild two failed
disks. Sincetheclientalreadyknowstheonlytwogroupsinwhichitsownvaluescandifferfrom
theserver’s,twochecksumsareenoughtorecovertheoneitneeds. Thisshrinkseveryreply,and
thereforeeverylookup. Privacyisunchanged,becausethetaskletstheserverseeonlyapublic
functionoftherequesttheauthors’clientwouldsend.
MuseWonder1.0’srunpasseswithafinalscoreabovethehumanreference’s. Mostagentspass
thistask,butnootherpassingrunshrinksthelookupbynearlyasmuch(Figure9). Wefound
thisideaonlybyreadingthepatches,sinceneitherofourmeasurespicksitout. Apasscounts
thesamewhetherarunsavesafewbytesorthemost. GPT-6.1Solevenscoresslightlyhigher
thanMuseWonder1.0withafarsmallersaving.
6.2 Twofasterclocksforapacketscheduler
Anetworkswitchornetworkcardhastodecide,foreverypacketitsends,whichwaitingpacket
goes next. Programmable schedulers express this decision as a priority queue: each packet
carriesapriority,andthequeuemusthandoutthemosturgentonemillionsoftimespersecond.
BBQ[33]buildssuchaqueueasanFPGAcircuit. Itmarkstheoccupiedprioritiesinatreeof
12
QuarterPIR bytes per lookup
Bytes per lookup at 67 million entries
Human reference 1,760
Claude Opus 5.5 1,728
GPT-6 Astra 1,752
GPT-6.1 Sol 1,664
Gemini 4 Argon 1,664
GLM 5.3 1,744
Muse Wonder 1.0 1,280
DeepSeek V4.1 Flash 1,728 not pass
Qwen 3.8 Max 1,760 not pass
Muse Spark 1.3 1,760 not pass
GLM 5.3 Flash 1,728
1,200 1,300 1,400 1,500 1,600 1,700 1,800
Figure9: QuarterPIRbytesperlookup. Bytesperprivatelookuponthe67-million-entrydatabase;the
x-axisstartsat1,200. MuseWonder1.0’schecksumsremovefarmoreofthereference’sbytesthanany
otherpassingrundoes. KimiK3,farbelowtherestandnotpassing,isleftout. Step5Preview’sruntimed
out.
bitmaps. Eachbitkeepsacountofthepacketsbelowit,sothatthebitcanbeclearedwhenits
last packet leaves. In our task the agent rewrites the logic of the tree’s levels. A run passes if
thecircuitiscorrectandbeatsthehumanreferenceonthroughputandonworst-caselatencyat
everytreesize,includingalargeronethattheagentneverseesduringdevelopment.
On the smaller trees, the slowest step of each clock cycle in the human reference begins with
a counter read from on-chip memory. The operation just ahead may have changed the same
counter a moment earlier. The circuit therefore has to check the value it read against recent
updatesbeforeitcanuseit. Gemini4Argonkeepsthememoriesbutsettlesthatcheckwhilethe
readisstillinflight,leavinglittleworkforwhenthevaluearrives. ClaudeOpus5.5takesthe
otherroute. Itmoveseverycounterfrommemoryintoregistersandupdatesawholelevelofthe
treeinasinglecycle,sothatonthedevelopmenttreesnooperationwaitsforaread.
OnlythesetwoandGPT-6.1Solpass,andthetwodesignstradespeedforsize(Figure10). Both
handlemoreoperationspersecondthanthehumanreferenceoneverytree. Onthelargesttree,
however,thecircuitofClaudeOpus5.5usesthemostlookuptablesofthepassingcircuits. Its
designalsodoesnotservetreesaslargeasthoseBBQwasdesignedfor. Gemini4Argonkeeps
roughlythehumanreference’scircuitsizeandmemorylayout. Registersalonedonotexplain
ClaudeOpus5.5’sclock. Otheragentsalsomovethecountersintoregisters,andonthelargest
treetheyendupslowerthanthehumanreference. ThefinalscoreranksClaudeOpus5.5first,
becauseitrewardsspeedandignorescircuitsize.
13
BBQ clock period and circuit size
Clock period at width 16 (ps)
2,000
Claude Opus 5.5
Gemini 4 Argon
Human reference
GPT-6.1 Sol
4,000
Kimi K3
GPT-6 Astra
Muse Wonder 1.0
6,000 Muse Spark 1.3
8,000
DeepSeek V4.1 Flash
1,000 3,000 10,000 30,000
Lookup tables at 4,096 priorities
Figure10: BBQclockperiodandcircuitsize. Eachpointisoneagent’scircuitonthelargesttree,4,096
priorities. Thex-axisgivesitssizeinlookuptablesandthey-axisitsclockperiod,withshorter,faster
clocksatthetop;thegreydotmarksthehumanreference. ClaudeOpus5.5’scircuithastheshortestclock
periodandisamongthelargest. Agentswithoutacorrectcircuitatthissizeareleftout.
6.3 Ameasuredbudgetforsparsevideoattention
Video diffusion models spend most of their time in attention. SVG-EAR [29], our running
example in Section 3, computes some blocks of attention exactly and estimates the rest from
clustercentroids. Withinafixedbudget,itcomputestheblockswhoseestimateswoulderrthe
most. Arunpassesonlyifeveryheld-outclipisfasterthanwiththehumanreferenceandstays
withinnarrowbandsofitsPSNR,SSIMandLPIPS.
Norunpasses,andGPT-6Astracomesclosest. Itkeepstheauthors’designbutmeasuresthe
errorinsteadofpredictingit. Everyfewdenoisingsteps,itcomputestheexactattentionofafew
realqueriesandrebuildstheiroutputunderseveralbudgets. Eachattentionheadthenrunswith
thesmallestbudgetwhoseerrorstaysunderabound. Astraalsocorrectsabiasofthecentroid
estimatethatthepaperleavesin. Itsvideoscomeoutfasterthanthehumanreference’sonevery
clipand,onaverage,closertofullattention(Figure11). Itstillfailsonimagequalityintwoclips.
Speedwasnottheobstacleonthistask. Mostrunsbeatthereference’stimeoneveryclip,andno
runstayswithineveryimageband.
14
SVG-EAR time and PSNR
| | | Mean time per clip (s) | | | | | Mean PSNR (dB) | | | | |
| ------------------- | --------------- | ---------------------- | ------- | --- | ----- | ----- | -------------- | ----- | ---- | ---- | ---- |
| | Human reference | | | | 891 | | | | 21.7 | | |
| | Claude Opus 5.5 | | | 804 | | | | | 21.5 | | |
| | GPT-6 Astra | | 729 | | | | | | | 23.8 | |
| | GPT-6.1 Sol | | | | 942 | | | | | | 25.4 |
| | Gemini 4 Argon | | | 833 | | | | | 21.2 | | |
| | GLM 5.3 | | | | | 1,087 | | | 20.1 | | |
| | Muse Wonder 1.0 | | | 850 | | | | | 21.2 | | |
| DeepSeek V4.1 Flash | | | | 830 | | | | | 21.4 | | |
| | Kimi K3 | | | 768 | | | | | 21.3 | | |
| | Qwen 3.8 Max | | | 836 | | | | 18.7 | | | |
| | Muse Spark 1.3 | | | 844 | | | | | 21.4 | | |
| | GLM 5.3 Flash | | | 800 | | | | | 20.8 | | |
| | | 600 | 700 800 | 900 | 1,000 | 1,100 | 16 | 18 20 | 22 | 24 | 26 |
Figure11:SVG-EARtimeandPSNR.Eachrun’smeantimeperclipandmeanPSNRagainstfullattention
ontheheld-outclips,withthehumanreference’singrey;thex-axesstartat600sand16dB.OnlyGPT-6
Astraisbothfasterandclosertofullattentionthanthehumanreferenceonaverage,anditstillfails. Step
5Preview,farslower,isoffthechart.
7 Conclusion
WebuiltFrontierCS2toaskwhetherLLMagentscanmakecontributionstofrontiercomputer
scienceR&D.Eachtaskmeasuresanagentagainstthecodethatresearchersreleasedwiththeir
paper,inthatcode’sownrepository. Beyondthese48tasks,FrontierCS2keepsaprivatesetof
morethan100tasks,tobeannouncedinthefuture.
Thecasestudiesshowwhatthepassandthefinalscorecaptureandwhattheymiss. OnSVG-
EAR,arunfasterthanthehumanreferenceoneveryclipandclosertofullattentiononaverage
stillfailsonimagequalityintwoclips(Section6.3). OnQuarterPIR,neithermeasureseparates
therunthatremovesfarmorebytesperlookupfromtheotherpassingruns(Section6.1). OnBBQ,
thefinalscoreranksthelargestpassingcircuitfirst,becauseitignorescircuitsize(Section6.2).
Wethinkbenchmarkslikethisonewillneedtoreportsuchsidecostsnexttothepass.
FCIandFDIarethemeasureswerecommendforbenchmarkslikethisone. Theyplaceagents
and tasks on one scale, use every metric outcome and score margin of a run rather than one
number,andcomewithintervalsthatshowwhichdifferencesthedatasupport(Section5.2).
15
Acknowledgments
We give special thanks to Modal for its strong support of this work, including the compute
resourcesitsponsored.
We also thank the Laude Institute’s Slingshot program and Scale AI for compute credits and
communitysupport. WethankKaggleforsponsoringFrontierCS.
16
References
[1] Qiuyang Mang, Wenhao Chai, Zhifei Li, Huanzhi Mao, Shang Zhou, Alexander Du,
HanchenLi,ShuLiu,EdwinChen,YichuanWang,etal. Frontiercs: Evolvingchallengesfor
evolvingintelligence. arXivpreprintarXiv:2512.15699,2025.
[2] YukiImajuku, KohkiHorie, YoichiIwata, KenshoAoki, NaohiroTakahashi, andTakuya
Akiba. Ale-bench: Abenchmarkforlong-horizonobjective-drivenalgorithmengineering.
AdvancesinNeuralInformationProcessingSystems,38,2026.
[3] KaiyuanLiu,QiuyangMang,BoPeng,WenhaoChai,HanchenLi,ShreyasPimpalgaonkar,
LukeZettlemoyer,AlexDimakis,andAlvinCheung. Whenagentsslowdown: Understand-
ingllmagents’test-timestrategiesviaelo-per-tokenanalysis. arXivpreprintarXiv:2609.15309,
2026.
[4] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and
KarthikNarasimhan. Swe-bench: Canlanguagemodelsresolvereal-worldgithubissues?
InInternationalConferenceonLearningRepresentations,volume2024,pages54107–54157,2024.
[5] JunShernChan,NeilChowdhury,OliverJaffe,JamesAung,DaneSherburn,EvanMays,
Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating
machine learning agents on machine learning engineering. In International Conference on
LearningRepresentations,volume2025,pages50466–50494,2025.
[6] AnikethGarikaparthi,ManasiPatwardhan,andArmanCohan. Researchgym: Evaluating
languagemodelagentsonreal-worldairesearch. arXivpreprintarXiv:2602.15112,2026.
[7] BohanLyu,YuchengYang,SiqiaoHuang,JiaruZhang,QixinXu,XinghanLi,XinyangHan,
YichengZhang,HuaqingZhang,RunhanHuang,etal. Mls-bench: Aholisticandrigorous
assessmentofaisystemsonbuildingbetterai. arXivpreprintarXiv:2605.08678,2026.
[8] Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin,
RachelDias,EvanMays,BenjaminKinsella,WyattThompson,etal. Paperbench: evaluating
ai’sabilitytoreplicateairesearch. arXivpreprintarXiv:2504.01848,2025.
[9] ZihanZheng,ZeruiCheng,ZeyuShen,ShangZhou,KaiyuanLiu,HansenHe,Dongruixuan
Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, et al. Livecodebench pro: How do olympiad
medalistsjudgellmsincompetitiveprogramming? AdvancesinNeuralInformationProcessing
Systems,38,2026.
[10] HongzhengChen, YinghengWang, YaohuiCai, HinsHu, JiajieLi, ShirleyHuang, Chen-
huiDeng,RongjianLiang,ShufengKong,HaoxingRen,SamithaSamaranayake,CarlaP.
Gomes,andZhiruZhang. HeuriGym: AnagenticbenchmarkforLLM-craftedheuristicsin
combinatorialoptimization. InICLR,2026.
[11] XiaozheLi,XinyuFang,ShengyuanDing,LinyangLi,HaodongDuan,QingwenLiu,and
KaiChen. Np-engine: Empoweringoptimizationreasoninginlargelanguagemodelswith
verifiablesyntheticnpproblems. arXivpreprintarXiv:2510.16476,2025.
17
[12] Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang, William Hu, Christopher Re´,
andAzaliaMirhoseini. Kernelbench: Canllmswriteefficientgpukernels? arXivpreprint
arXiv:2502.10517,2025.
[13] Bingchen Zhao, Despoina Magka, Minqi Jiang, Xian Li, Roberta Raileanu, Tatiana Shav-
rina,Jean-ChristopheGagnon-Audet,KelvinNiu,ShagunSodhani,MichaelShvartsman,
etal. Theautomatedllmspeedrunningbenchmark: Reproducingnanogptimprovements.
AdvancesinNeuralInformationProcessingSystems,38,2026.
[14] ElieBakouchandPrimeIntellect. Measuringautonomousairesearch. PrimeIntellectBlog,
August2026. https://www.primeintellect.ai/blog/measuring-autonomous-research.
[15] JehyeokYeon,BenRank,andMaksymAndriushchenko. Inferencebench: Abenchmarkfor
open-endedllminferenceoptimizationbyaiagents. arXivpreprintarXiv:2607.20468,2026.
[16] MikeMerrill,AlexanderShaw,NicholasCarlini,BoxuanLi,HarshRaj,IvanBercovich,Lin
Shi,JeongShin,ThomasWalshe,EKellyBuchanan,etal. Terminal-bench: Benchmarking
agents on hard, realistic tasks in command line interfaces. In International Conference on
LearningRepresentations,volume2026,pages40903–40986,2026.
[17] HjalmarWijk,TaoLin,JoelBecker,SamiJawhar,NeevParikh,ThomasBroadley,Lawrence
Chan,MichaelChen,JoshClymer,JaiDhyani,etal. Re-bench: Evaluatingfrontierair&d
capabilitiesoflanguagemodelagentsagainsthumanexperts.arXivpreprintarXiv:2411.15114,
2024.
[18] BenRank,HardikBhatnagar,AmeyaPrabhu,ShiraEisenberg,KarinaNguyen,Matthias
Bethge, and Maksym Andriushchenko. Posttrainbench: Can llm agents automate llm
post-training? arXivpreprintarXiv:2603.08640,2026.
[19] YuanliWang,YaoyaoQian,YueZhang,HanhanZhou,JindanHuang,TianfuFu,Qiuyang
Mang,HuanzhiMao,WenhaoChai,WendongFan,etal. Deploybench: Benchmarkingllm
agentsforresearchartifactdeployment. arXivpreprintarXiv:2606.05238,2026.
[20] DeyaoZhu,XinZhou,ShenglingQin,XuekaiZhu,HangliangDing,ShuZhong,ZixinWen,
Zhonglin Xie, Chenhui Gou, Linxuan Ren, et al. Edgebench: Unveiling scaling laws of
learningfromreal-worldenvironments. arXivpreprintarXiv:2607.05155,2026.
[21] Alisia Lupidi, Bhavul Gauri, Thomas Simon Foster, Bassel Al Omari, Despoina Magka,
AlbertoPepe,AlexisAudran-Reiss,MunaAghamelu,NicolasBaldwin,LuciaCipolina-Kun,
et al. Airs-bench: a suite of tasks for frontier ai research science agents. arXiv preprint
arXiv:2602.06855,2026.
[22] RenxiongWang,DarvinYi,AbrilHerrlein,AnasMahmoud,AdvaitGosai,LisimanHua,
MohammadHosseinRezaei,XingangGuo,AnishaGunjal,UtkarshTyagi,etal. Rsi-forge:
From research papers to environments for recursive self-improvement. arXiv preprint
arXiv:2610.09426,2026.
18
[23] SeungoneKim,JiayiGeng,Young-JunLee,BoLiu,JuyoungSuk,NicholasLubbers,Mitchell
Wood,MarcusNoack,Yong-XinYao,YuanShi,etal. Intotheunknown: Benchmarkingai
agentsonopenscientificresearchproblems. arXivpreprintarXiv:2610.19159,2026.
[24] ZeyuZheng,YiyangLiu,YuyangZhao,CongminZheng,JiachenZhu,JianianChen,Kangn-
ing Zhang, Kayue Daniel Wong, Lingyu Yang, Rong Shan, Shihao Chen, Thomas Zhu,
WentaoZhang,XiaopengLi,YanHu,YongxiLin,YuxiangChen,ZhuoyingOu,Jianghao
Lin,JeremyAvigad,PrasadTetali,JingbangChen,andSeanWelleck. ReuF2F:Benchmarking
codingagentsonformalizedundergraduateresearchproblems. Preprint,2026.
[25] JohnYang,KilianLieret,CarlosJimenez,AlexanderWettig,KabirKhandpur,YanzheZhang,
Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for
softwareengineeringagents. AdvancesinNeuralInformationProcessingSystems,38,2026.
[26] Runyuan He, Qiuyang Mang, Shang Zhou, Kaiyuan Liu, Hanchen Li, Huanzhi Mao,
Qizheng Zhang, Zerui Li, Bo Peng, Lufeng Cheng, et al. Frontiersmith: Synthesizing
open-endedcodingproblemsatscale. arXivpreprintarXiv:2605.14445,2026.
[27] YangzhenWu,AaronJLi,WenjieMa,LiCao,ZihengZhou,MertCemri,ShuLiu,YuranXiu,
ChenxiaoYan,HaikunZhao,etal. Benchevolver: Frontiertasksynthesisviasolution-centric
evolution. arXivpreprintarXiv:2606.01286,2026.
[28] Alexander D Goldie, Zilin Wang, Adrian Hayler, Deepak Nathani, Edan Toledo, Ken
Thampiratwong,AleksandraKalisz,MichaelBeukman,AlistairLetcher,ShashankReddy,
etal. Proceduralgenerationofalgorithmdiscoverytasksinmachinelearning. arXivpreprint
arXiv:2603.17863,2026.
[29] Xuanyi Zhou, Qiuyang Mang, Shuo Yang, Haocheng Xi, Jintao Zhang, Huanzhi Mao,
JosephE.Gonzalez,KurtKeutzer,IonStoica,andAlvinCheung. SVG-EAR:Parameter-free
linearcompensationforsparsevideogenerationviaerror-awarerouting. InECCV,pages
167–185,2026.
[30] HanchenLi,RunyuanHe,QiuyangMang,QizhengZhang,HuanzhiMao,XiaokunChen,
Hangrui Zhou, Huanchen Zhang, Alvin Cheung, Joseph Gonzalez, et al. Continuum:
Efficient and robust multi-turn llm agent scheduling with kv cache time-to-live. arXiv
preprintarXiv:2511.02230,2025.
[31] Anson Ho, Jean-Stanislas Denain, David Atanasov, Samuel Albanie, and Rohin Shah. A
rosettastoneforaibenchmarks. arXivpreprintarXiv:2512.00193,2025.
[32] Ashrujit Ghoshal, Mingxun Zhou, and Elaine Shi. Efficient pre-processing pir without
public-keycryptography. InAnnualInternationalConferenceontheTheoryandApplicationsof
CryptographicTechniques,pages210–240.Springer,2024.
[33] NiravAtre,HugoSadok,andJustineSherry. {BBQ}: Afastandscalableintegerpriority
queueforhardwarepacketscheduling. In21stUSENIXSymposiumonNetworkedSystems
DesignandImplementation(NSDI24),pages455–475,2024.
19
[34] Anna Arpaci-Dusseau, Zixiang Zhou, and Xuhao Chen. Accurate and fast approximate
graphpatternminingatscale. arXivpreprintarXiv:2405.03488,2024.
[35] LieseBekkers,FrankNeven,StijnVansummeren,andYisuRemyWang. Instance-optimal
acyclic join processing without regret: Engineering the yannakakis algorithm in column
stores. arXivpreprintarXiv:2411.04042,2024.
[36] Xi Zhao, Bolong Zheng, Xiaomeng Yi, Xiaofan Luan, Charles Xie, Xiaofang Zhou, and
ChristianSJensen. Fargo: Fastmaximuminnerproductsearchviaglobalmulti-probing.
ProceedingsoftheVLDBEndowment,16(5):1100–1112,2023.
[37] Zhihao Chen, Tianji Yang, Yixiao Zheng, Zhao Zhang, Cheqing Jin, and Aoying Zhou.
Spectrum: Speedy and strictly-deterministic smart contract transactions for blockchain
ledgers. Proc.VLDBEndow.,17(10):2541–2554,June2024.
[38] Qiyan Li and Jeffrey Xu Yu. Fast local subgraph counting. Proc. VLDB Endow.,
17(8):1967–1980,April2024.
[39] Guanhao Hou, Jinchao Huang, Fangyuan Zhang, and Sibo Wang. Efficient concurrent
updatestopersistentrandomizedbinarysearchtrees. Proc.VLDBEndow.,18(5):1481–1494,
January2025.
[40] Shijie Zhang, Ru Cheng, Xinpeng Liu, Jiang Xiao, Hai Jin, and Bo Li. Seer: Accelerating
blockchain transaction execution by fine-grained branch prediction. Proc. VLDB Endow.,
18(3):822–835,November2024.
[41] Zezhou Huang, Rathijit Sen, Jiaxiang Liu, and Eugene Wu. Joinboost: Grow trees over
normalizeddatausingonlysql. Proc.VLDBEndow.,16(11):3071–3084,July2023.
[42] DavidKlindt,YannLeCun,andRandallBalestriero. Whendoeslejepalearnaworldmodel?
arXivpreprintarXiv:2605.26379,2026.
[43] NoahAmsel,DavidPersson,ChristopherMusco,andRobertMGower. Thepolarexpress:
Optimalmatrixsignmethodsandtheirapplicationtothemuonalgorithm. InInternational
ConferenceonLearningRepresentations,volume2026,pages138323–138360,2026.
[44] JiaweiYang,TianhongLi,LijieFan,YonglongTian,andYueWang. Latentdenoisingmakes
goodtokenizers. InInternationalConferenceonLearningRepresentations,volume2026,pages
61642–61668,2026.
[45] Jiawei Yang, Zhengyang Geng, Xuan Ju, Yonglong Tian, and Yue Wang. Representation
fr\’echetlossforvisualgeneration. arXivpreprintarXiv:2604.28190,2026.
[46] ZhenFang,WenxuanHuang,YuZeng,YimingZhao,ShuangChen,KaituoFeng,Yunlong
Lin,LinChen,ZehuiChen,ShaoshengCao,etal. Flow-opd: On-policydistillationforflow
matchingmodels. arXivpreprintarXiv:2605.08063,2026.
20
[47] ZihanWang,KangruiWang,QinengWang,PingyueZhang,LinjieLi,ZhengyuanYang,Xing
Jin,KefanYu,MinhNhatNguyen,LichengLiu,etal. Ragen: Understandingself-evolution
inllmagentsviamulti-turnreinforcementlearning. arXivpreprintarXiv:2504.20073,2025.
[48] JiachenZhu,XinleiChen,KaimingHe,YannLeCun,andZhuangLiu. Transformerswithout
normalization. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR),pages14901–14911.IEEE,2025.
[49] SiqiaoHuang,ParthaKaushik,MichaelChen,HengkaiPan,KaiwenGeng,OmarChehab,
FernandoMoreno-Pino,andMaxSimchowitz. Nanoworldmodels: Aminimalistimple-
mentationoffuturevideoprediction. arXivpreprintarXiv:2605.23993,2026.
[50] AdamZweiger,XinghongFu,HanGuo,andYoonKim. Fastkvcompactionviaattention
matching. arXivpreprintarXiv:2602.16284,2026.
[51] Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Xiaoze Fan, Jintao Zhang, Han Cai,
Yujun Lin, Xiuyu Li, Kurt Keutzer, et al. Flash-kmeans: Fast and memory-efficient exact
k-means. arXivpreprintarXiv:2603.09229,2026.
[52] Shuo Yang, Haocheng Xi, Yilong Zhao, Qiuyang Mang, Zhe Wang, Shanlin Sun, Kurt
Keutzer,JosephE.Gonzalez,SongHan,ChenfengXu,andIonStoica. Flashlib: Bringing
flashmagictoclassicalmachinelearningoperators,2026.
[53] Julian Shun. Practical parallel hypergraph algorithms. In Proceedings of the 25th ACM
SIGPLANSymposiumonPrinciplesandPracticeofParallelProgramming,pages232–249,2020.
[54] JessicaShiandJulianShun. Parallelalgorithmsforbutterflycomputations. InMassiveGraph
Analytics,pages287–330.ChapmanandHall/CRC,2022.
[55] YiqiuWang,YanGu,andJulianShun. Theoretically-efficientandpracticalparalleldbscan.
In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data,
pages2555–2571,2020.
[56] OmarObeya,EndriasKahssay,EdwardFan,andJulianShun. Theoretically-efficientand
practicalparallelin-placeradixsorting.InThe31stACMsymposiumonparallelisminalgorithms
andarchitectures,pages213–224,2019.
[57] David Cao, Rose Kunkel, Chandrakana Nandi, Max Willsey, Zachary Tatlock, and Na-
diaPolikarpova. babble: Learningbetterabstractionswithe-graphsandanti-unification.
ProceedingsoftheACMonProgrammingLanguages,7(POPL):396–424,2023.
[58] Max Willsey, Chandrakana Nandi, Yisu Remy Wang, Oliver Flatt, Zachary Tatlock, and
PavelPanchekha. Egg: Fastandextensibleequalitysaturation. ProceedingsoftheACMon
ProgrammingLanguages,5(POPL):1–29,2021.
[59] Pinhan Zhao, Yuepeng Wang, and Xinyu Wang. Polygon: Symbolic reasoning for sql
usingconflict-drivenunder-approximationsearch. ProceedingsoftheACMonProgramming
Languages,9(PLDI):1315–1340,2025.
21
[60] Gus Henry Smith, Andrew Liu, Steven Lyubomirsky, Scott Davidson, Joseph McMahan,
Michael Taylor, Luis Ceze, and Zachary Tatlock. Pure tensor program rewriting via ac-
cesspatterns(representationpearl). InProceedingsofthe5thACMSIGPLANInternational
SymposiumonMachineProgramming,pages21–31,2021.
[61] AbdullahAlRaqibulIslam,HelenXu,DongDai,andAydinBuluc¸. Improvingspgemm
performance through matrix-reordering and cluster-wise computation. In Proceedings of
theInternationalConferenceforHighPerformanceComputing,Networking,StorageandAnalysis,
pages1446–1463,2025.
[62] HemengWang, YangDu, SiduLi, XiaowenTian, QingxiaoSun, andWeifengLiu. Kami:
Communication-avoidinggeneralmatrixmultiplicationwithinasinglegpu. InProceedings
oftheInternationalConferenceforHighPerformanceComputing,Networking,StorageandAnalysis,
pages1572–1589,2025.
[63] ToruKoizumi,ToshikiMaekawa,MasanariMizuno,MaruKuroki,TomoakiTsumura,and
Ryota Shioya. Runlts: Register-value-aware predictor utilizing nested large tables. 6th
ChampionshipBranchPrediction(CBP),2025.
[64] Aayush Jain, Huijia Lin, and Sagnik Saha. A systematic study of sparse lwe. In Annual
InternationalCryptologyConference,pages210–245.Springer,2024.
[65] Jean-Philippe Bossuat, Anamaria Costache, Christian Mouchet, Lea Nu¨rnberger, and
JuanRamo´nTroncoso-Pastoriza. Accurateandcomposablenoiseestimatesforckkswith
applicationtoexacthecomputation. CryptologyePrintArchive,2024.
[66] Pengfei Zheng, Rui Pan, Tarannum Khan, Shivaram Venkataraman, and Aditya Akella.
Shockwave: Fairandefficientclusterschedulingfordynamicadaptationinmachinelearn-
ing. In20thUSENIXSymposiumonNetworkedSystemsDesignandImplementation(NSDI23),
pages703–723,2023.
[67] Pooria Namyar, Behnaz Arzani, Ryan Beckett, Santiago Segarra, Himanshu Raj, Umesh
Krishnaswamy,RameshGovindan,andSrikanthKandula. Findingadversarialinputsfor
heuristicsusingmulti-leveloptimization. In21stUSENIXSymposiumonNetworkedSystems
DesignandImplementation(NSDI24),pages927–949,2024.
[68] TiannengShi,JingxuanHe,ZhunWang,HongweiLi,LinyuWu,WenboGuo,andDawn
Song. Progent: Securingaiagentswithprivilegecontrol. arXivpreprintarXiv:2504.11703,
2025.
[69] Sicheol Sung, Joonghyuk Hahn, and Yo-Sub Han. Repairing regex vulnerabilities via
localization-guidedinstructions. InProceedingsofthe19thConferenceoftheEuropeanChapter
oftheAssociationforComputationalLinguistics(Volume1: LongPapers),pages8128–8142,2026.
[70] ChengChi,ZhenjiaXu,SiyuanFeng,EricCousineau,YilunDu,BenjaminBurchfiel,Russ
Tedrake,andShuranSong. Diffusionpolicy: Visuomotorpolicylearningviaactiondiffusion,
2024.
22
[71] KevinZakka,BaruchTabanpour,QiayuanLiao,MustafaHaiderbhai,SamuelHolt,JingYuan
Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A. Kahrs, Carmelo Sferrazza,
YuvalTassa,andPieterAbbeel. Mujocoplayground,2025.
[72] MartinSˇosˇic´ andMileSˇikic´. Edlib: ac/c++libraryforfast,exactsequencealignmentusing
editdistance. Bioinformatics,33(9):1394–1395,2017.
[73] JacobBurnimandKoushikSen. Heuristicsforscalabledynamictestgeneration. In2008
23rdIEEE/ACMInternationalConferenceonAutomatedSoftwareEngineering,pages443–446.
IEEE,2008.
[74] Thomas Mueller Graf and Daniel Lemire. Binary fuse filters: Fast and smaller than xor
filters. ACMJ.Exp.Algorithmics,27,March2022.
23
Appendix
A Task sources
Table2liststhepaperbehindeachofthe48tasks. ThethreeFlashLibtasksshareonesource,and
sodothetwoFARGOtasks. StructuredLWEistheonetasknotbuiltfromapaper’scode: itis
anoriginalcryptanalysischallengeinwhichtheagentattacks200publishedinstances,andany
secretthatmeetsthepublishedconditionscounts. Sincetheseinstancesarepublic,itsfinalsuite
hasnoheld-outpart.
| | Table2: Tasksources. Thepapereachtaskisbuiltfrom. | |
| ------ | ------------------------------------------------- | ----- |
| Domain | Task | Paper |
Database ScaleGPM AccurateandFastApproximateGraphPatternMiningat
Scale[34]
Yannakakis Instance-OptimalAcyclicJoinProcessingWithoutRegret:
EngineeringtheYannakakisAlgorithminColumn
Stores[35]
FARGOQueryStopping FARGO:FastMaximumInnerProductSearchviaGlobal
Multi-Probing[36]
| | Spectrum | Spectrum:SpeedyandStrictly-DeterministicSmart |
| --- | -------- | --------------------------------------------- |
ContractTransactionsforBlockchainLedgers[37]
| | SCOPE | FastLocalSubgraphCounting[38] |
| --- | ----- | ----------------------------- |
PersistentTreap EfficientConcurrentUpdatestoPersistentRandomized
BinarySearchTrees[39]
| | FARGOQueryCore | FARGO:FastMaximumInnerProductSearchviaGlobal |
| --- | -------------- | -------------------------------------------- |
Multi-Probing[36]
| | Seer | Seer:AcceleratingBlockchainTransactionExecutionby |
| --- | ---- | ------------------------------------------------- |
Fine-GrainedBranchPrediction[40]
| | JoinBoost | JoinBoost:GrowTreesoverNormalizedDataUsingOnly |
| --- | --------- | ---------------------------------------------- |
SQL[41]
| MachineLearning | LeJEPA | WhenDoesLeJEPALearnaWorldModel?[42] |
| --------------- | ------------ | ------------------------------------------- |
| | PolarExpress | ThePolarExpress:OptimalMatrixSignMethodsand |
TheirApplicationtotheMuonAlgorithm[43]
| | DeTok | LatentDenoisingMakesGoodTokenizers[44] |
| --- | -------------------------- | ------------------------------------------------- |
| | RepresentationFre´chetLoss | RepresentationFre´chetLossforVisualGeneration[45] |
| | Flow-OPD | Flow-OPD:On-PolicyDistillationforFlowMatching |
Models[46]
| | RAGEN | RAGEN:UnderstandingSelf-EvolutioninLLMAgents |
| --- | ----- | -------------------------------------------- |
viaMulti-TurnReinforcementLearning[47]
| | DyT | TransformerswithoutNormalization[48] |
| --- | -------------- | ------------------------------------------- |
| | NanoWorldModel | NanoWorldModels:AMinimalistImplementationof |
FutureVideoPrediction[49]
| | KVCompaction | FastKVCompactionviaAttentionMatching[50] |
| ----- | ------------ | ------------------------------------------- |
| MLSys | SVG-EAR | SVG-EAR:Parameter-FreeLinearCompensationfor |
SparseVideoGenerationviaError-awareRouting[29]
| | FlashLibK-Means | Flash-KMeans:FastandMemory-EfficientExact |
| --- | --------------- | ----------------------------------------- |
K-Means[51]
| | FlashLibDBSCAN | FlashLib:BringingFlashMagictoClassicalMachine |
| --- | -------------- | --------------------------------------------- |
LearningOperators[52]
24
| Domain | Task | Paper |
| ------ | -------------- | --------------------------------------------- |
| | FlashLibIVF-PQ | FlashLib:BringingFlashMagictoClassicalMachine |
LearningOperators[52]
| | FlashLibKNN | FlashLib:BringingFlashMagictoClassicalMachine |
| --- | ----------- | --------------------------------------------- |
LearningOperators[52]
| | Continuum | Continuum:EfficientandRobustMulti-TurnLLMAgent |
| --- | --------- | ---------------------------------------------- |
SchedulingwithKVCacheTime-to-Live[30]
| HPC | Hygra | PracticalParallelHypergraphAlgorithms[53] |
| --- | -------------- | ---------------------------------------------- |
| | ParButterfly | ParallelAlgorithmsforButterflyComputations[54] |
| | ParallelDBSCAN | Theoretically-EfficientandPracticalParallel |
DBSCAN[55]
| | RegionsSort | Theoretically-EfficientandPracticalParallelIn-Place |
| --- | ----------- | --------------------------------------------------- |
RadixSorting[56]
ProgrammingLanguages BABBLE babble:LearningBetterAbstractionswithE-Graphsand
Anti-unification[57]
| | egg | egg:FastandExtensibleEqualitySaturation[58] |
| --- | ------- | ------------------------------------------- |
| | Polygon | Polygon:SymbolicReasoningforSQLusing |
Conflict-DrivenUnder-ApproximationSearch[59]
| | Glenside | PureTensorProgramRewritingviaAccessPatterns |
| --- | -------- | ------------------------------------------- |
(RepresentationPearl)[60]
Architecture ClusterwiseSpGEMM ImprovingSpGEMMPerformanceThrough
Matrix-ReorderingandCluster-wiseComputation[61]
| | KAMI | KAMI:Communication-AvoidingGeneralMatrix |
| --- | ---- | ---------------------------------------- |
MultiplicationwithinaSingleGPU[62]
| | RUNLTS | RUNLTS:Register-value-awarePredictorUtilizing |
| --- | ------ | --------------------------------------------- |
NestedLargeTables[63]
| Cryptography | StructuredLWE | ASystematicStudyofSparseLWE[64] |
| ------------ | ------------- | ------------------------------------------- |
| | QuarterPIR | EfficientPre-processingPIRWithoutPublic-Key |
Cryptography[32]
| | CKKSNoise | AccurateandComposableNoiseEstimatesforCKKS |
| --- | --------- | ------------------------------------------ |
withApplicationtoExactHEComputation[65]
| Network | BBQ | BBQ:AFastandScalableIntegerPriorityQueuefor |
| ------- | --- | ------------------------------------------- |
HardwarePacketScheduling[33]
| | Shockwave | Shockwave:FairandEfficientClusterSchedulingfor |
| --- | --------- | ---------------------------------------------- |
DynamicAdaptationinMachineLearning[66]
| | MetaOpt | FindingAdversarialInputsforHeuristicsusing |
| --- | ------- | ------------------------------------------ |
Multi-levelOptimization[67]
Security Progent Progent:SecuringAIAgentswithPrivilegeControl[68]
| | LRR | RepairingRegexVulnerabilitiesviaLocalization-Guided |
| --- | --- | --------------------------------------------------- |
Instructions[69]
Robotics DiffusionPolicy DiffusionPolicy:VisuomotorPolicyLearningviaAction
Diffusion[70]
| | MuJoCoPlayground | DemonstratingMuJoCoPlayground[71] |
| --- | ---------------- | --------------------------------- |
Bioinformatics Edlib Edlib:aC/C++libraryforfast,exactsequencealignment
usingeditdistance[72]
SoftwareEngineering CREST HeuristicsforScalableDynamicTestGeneration[73]
Theory BinaryFuseFilter BinaryFuseFilters:FastandSmallerThanXorFilters[74]
25
B Task resources
Table3listswhateachtaskgivestheagentandtheverifier. Theagenthasfivehoursonevery
task. GPU tasks give the agent short-lived GPU runners for its own experiments and score
submissionsonseparateverifierGPUs(everyA100isthe80GBpart);thelistedGPUtimeisa
budgetforthewholerun,notameasuredcost. Progentistheonlytaskthatalsocallsamodel
frominsidethetask: theagentsitsecuresrunonataskmodelwiththeirownper-runspending
limit.
Table3: Taskresources. Whateachtaskgivestheagentandtheverifier. Agentcoresandmemory: the
containertheagentworksin,memoryinGiB.GPU:theGPUtypeofthetask;adashmeansthetaskuses
noGPU.GPU-hours: thetotaltheagentmayuseforitsownexperiments,andthetotalforscoringits
submissions;n×hmeansnGPUsatatimeforhhourseach.
Trainevals: themostsubmissionstheagent
mayscoreonthedevelopmentworkloads. Finalevaltimelimit: howlongthehiddenfinalevaluation
mayrun,inminutes.
| | | Agent | | GPU-hours | | Train Finaleval | |
| --------------- | -------------------------- | ------------ | ------- | --------- | -------- | --------------- | --- |
| Domain | Task | Cores Memory | GPU | Agent | Verifier | evals timelimit | |
| Database | ScaleGPM | 8 | 16 – | | – – | 100 | 15 |
| | Yannakakis | 2 | 8 – | | – – | 100 | 15 |
| | FARGOQueryStopping | 4 | 4 – | | – – | 100 | 15 |
| | Spectrum | 4 | 8 – | | – – | 100 | 60 |
| | SCOPE | 1 | 4 – | | – – | 100 | 30 |
| | PersistentTreap | 8 | 16 – | | – – | 100 | 30 |
| | FARGOQueryCore | 4 | 4 – | | – – | 50 | 15 |
| | Seer | 8 | 4 – | | – – | 50 | 30 |
| | JoinBoost | 2 | 16 – | | – – | 100 | 120 |
| MachineLearning | LeJEPA | 2 | 12 – | | – – | 8 | 6.7 |
| | PolarExpress | 2 | 4 – | | – – | 100 | 30 |
| | DeTok | 3 | 12 L4 | 1.5 | 1.5 | 6 | 90 |
| | RepresentationFre´chetLoss | 3 | 12 A100 | | 2 9 | 3 | 180 |
| | Flow-OPD | 3 | 12 H100 | 0.83 | 4.17 | 4 | 50 |
| | RAGEN | 3 | 12 A100 | 0.5 | 2 | 3 | 30 |
| | DyT | 3 | 12 L4 | 0.5 | 1 | 3 | 15 |
| | NanoWorldModel | 3 | 12 A10G | 0.5 | 5.67 | 3 | 170 |
8×3
| | KVCompaction | 4 | 16 H100 | | 5 | 8 | 90 |
| -------------------- | --------------- | --- | ------- | --- | ------ | --- | ---- |
| MLSys | SVG-EAR | 3 | 12 H100 | 15 | 4×91 | 25 | 210 |
| | FlashLibK-Means | 4 | 8 H100 | | 3 5 | 25 | 30 |
| | FlashLibDBSCAN | 4 | 8 H100 | | 3 3.67 | 25 | 40 |
| | FlashLibIVF-PQ | 4 | 8 H100 | | 3 7 | 25 | 240 |
| | FlashLibKNN | 4 | 8 H100 | | 3 4.5 | 25 | 90 |
| | Continuum | 8 | 32 H100 | | 5 4×48 | 25 | 1800 |
| HPC | Hygra | 8 | 4 – | | – – | 100 | 30 |
| | ParButterfly | 4 | 8 – | | – – | 100 | 30 |
| | ParallelDBSCAN | 8 | 4 – | | – – | 100 | 10 |
| | RegionsSort | 4 | 2 – | | – – | 100 | 15 |
| ProgrammingLanguages | BABBLE | 2 | 8 – | | – – | 100 | 10 |
| | egg | 8 | 4 – | | – – | 100 | 30 |
| | Polygon | 2 | 8 – | | – – | 100 | 5.5 |
| | Glenside | 2 | 4 – | | – – | 100 | 5 |
26
| | | | Agent | GPU-hours | Train | Finaleval |
| ------------------- | ----------------- | --- | ------------ | ----------- | -------------- | --------- |
| Domain | Task | | Cores Memory | GPU Agent | Verifier evals | timelimit |
| Architecture | ClusterwiseSpGEMM | | 2 | 16 – | – – | 3 120 |
| | KAMI | | 3 | 4 H100 0.25 | 0.5 | 12 30 |
| | RUNLTS | | 1 | 4 – | – – 100 | 120 |
| Cryptography | StructuredLWE | | 8 | 32 – | – – 100 | 2 |
| | QuarterPIR | | 4 | 8 – | – – | 8 30 |
| | CKKSNoise | | 2 | 2 – | – – 100 | 10 |
| Network | BBQ | | 2 | 4 – | – – 100 | 30 |
| | Shockwave | | 4 | 4 – | – – | 3 30 |
| | MetaOpt | | 4 | 8 – | – – 100 | 60 |
| Security | Progent | | 4 | 8 – | – – | 5 180 |
| | LRR | | 4 | 12 L4 | – 2 100 | 60 |
| Robotics | DiffusionPolicy | | 2 | 4 – | – – 100 | 62 |
| | MuJoCoPlayground | | 3 | 12 H100 | 4 8.5 | 3 240 |
| Bioinformatics | Edlib | | 2 | 4 – | – – | 20 10 |
| SoftwareEngineering | CREST | | 8 | 16 – | – – 100 | 180 |
| Theory | BinaryFuseFilter | | 2 | 4 – | – – 100 | 3 |
| C Agent settings | | | | | | |
Table4listseachagent’sharnessversion,reasoningeffortandoutputtokenlimitintheevaluation.
Theeffortandthelimitarethevaluestheharnesssentwithitsmodelrequests;forClaudeCode,
theeffortistheonewesetwhenlaunchingit. Anemptycellmarksavaluewecouldnotconfirm.
Someharnesseschangeasettingfromcalltocall. MuseCodeaddsshortsidecallsthatwatchthe
mainagentanddecidewhethertoreminditofsomething;theserunatlowormediumeffort,
andthetablegivestheeffortofthemainagent’scalls. PisetsGemini4Argon’soutputlimitto
theroomleftinthecontextwindow. KimiCodegivesKimiK2.7thesmallerofthelistedlimit
andtheroomleft. ZCodeandStepCodealsoshrinktheirlimittotheroomleft,whichhappens
onlywhenthecontextwindowisnearlyfull.
| D Computing | FCI and | FDI | | | | |
| ----------- | ------- | --- | --- | --- | --- | --- |
FCIandFDIcomefromonestatisticalmodel,fittedtoallrunsatonce. Ourfittingcodecarriesout
thetwelvestepsbelow,andeveryvalueinthisappendixcomesfromthefitbehindSection5.2.
Table5liststheconstantsandfittedvalues.
Step1:collecttheruns. Thefituseseveryrunwithafinalscoreontheevaluatedtasks. Besides
the 12 reported agents, it includes Kimi K2.7, which anchors the zero of the scale but is not
reported. Arunthatendswithoutasavedfinalsubmissionscores0,andsodothetworunsthat
| Gemini4Argon’sproviderrefused(Section5.1). | | | Thisgives624runs. | | | |
| ------------------------------------------ | --- | --- | ----------------- | --- | --- | --- |
27
Table 4: Agent settings. Harness and version of each agent, with the reasoning effort and output
tokenlimittheharnessrequested. “Notset”leavesthedefaulttotheprovider. RowsfollowFCIorder
(Section5.2);KimiK2.7servesonlyasthe0pointofFCI.
| Model | | Harness | Version | Reasoningeffort | Outputlimit | |
| ----------------- | --- | --------------- | ---------- | --------------- | ------------- | --- |
| ClaudeOpus5.5 | | ClaudeCode | 2.1.296 | max | | |
| GPT-6Astra | | CodexCLI | 0.162.1 | xhigh | notset | |
| GPT-6.1Sol | | CodexCLI | 0.162.1 | xhigh | notset | |
| Gemini4Argon | | Pi | 1.2.0 | high | restofcontext | |
| GLM5.3 | | ZCode | 0.16.9 | max | 262,144 | |
| MuseWonder1.0 | | MuseCode | 1.4.2 | max | 128,000 | |
| DeepSeekV4.1Flash | | DeepSeekHarness | 0.2.0-rc.2 | max | 262,144 | |
| KimiK3 | | KimiCode | 0.31.0 | high | 131,072 | |
| Qwen3.8Max | | QwenCode | 0.25.0 | high | notset | |
| MuseSpark1.3 | | MuseCode | 1.4.2 | high | 128,000 | |
| GLM5.3Flash | | ZCode | 0.16.9 | max | 262,144 | |
| Step5Preview | | StepCode | 0.1.0 | high | 64,000 | |
| KimiK2.7 | | KimiCode | 0.31.0 | medium | upto131,072 | |
Step2: scoreeachmetric. Eachtask’sjudgemeasuresafixedsetofmetrics,suchaslatency,
throughput or accuracy, on every held-out workload. A run beats a metric when its final
submissionbeatsthehumanreferenceonthatmetricineveryheld-outworkload. Arunthat
failsbeforetheheld-outevaluationbeatsnometric. Arunthereforepassesexactlywhenitbeats
everymetricofitstask.
Step3:mergemetricsintogroups. Twometricsthateveryrunofataskbeatsormissestogether
cannotbetoldapartbythedata,andfittingeachwithitsowndifficultywouldmakethetask
lookharderthanitis. Suchmetricsformonegroup. Arunbeatsagroupwhenitbeatsevery
member. LetK bethenumberofgroupsoftaskt,andlet x ∈ {0,1}recordwhetheragent a
| | t | | | atg | | |
| ---------- | --------- | ----------------------- | --- | --- | --- | --- |
| beatsgroup | goftaskt. | Thetasksyield115groups. | | | | |
Step4: measurethescoremargin.
| | | | Finalscoresuseeachtask’sownscale, | | sothefitconverts | |
| --- | --- | --- | --------------------------------- | --- | ---------------- | --- |
a run’s distance from the human reference into a common unit. The reference score sref is the
t
median,overthetask’sruns,ofthehumanreference’sfinalscoremeasuredwithineachrun. If
therunsofataskrecordnopositivereferencescore,avaluesetinadvancetakesitsplace. The
spreadd t isthestandarddeviationofthetask’sfinalscoresthatliestrictlybetween0and100,or
ofallitsfinalscoreswhenfewerthanthreedo. Theunitisc = max(d ,c),whosefloorcisthe
| | | | | t t | | |
| --------- | ------------- | ---------------- | --- | ---------- | --- | --- |
| medianofd | overthetasks. | Themarginofagent | | aontasktis | | |
t
| | | | s | −sref | | |
| --- | --- | --- | ------ | ----- | --- | --- |
| | | | y = at | t , | | (1) |
at
c
t
where s is the run’s final score. A run enters the fit with its margin in one of three ways. A
at
runthatpasses,orfailsandscoresbelowthereference,contributesy asmeasured. Arunthat
at
28
≥
failsbutscoresabovethereferencecontributesonlyy at 0: itisatleastlevelwiththereference,
sinceatleastoneofitsmetricgroupsmisses. Arunthattiesthereferenceexactlycontributesno
margin.
Step 5: write the likelihood of one run. Each agent has a strength θ , each metric group a
a
difficulty β ,andeachtaskatielevelh ,allinlogits. Thetielevelisthestrengthatwhicharun’s
| | tg | | | | t | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
∼ N(0,σ2),
expected margin is zero. A run also has a run effect u at which says how far this
u
runlandsaboveorbelowitsagent’sstrength. Thegroupsandthemargincomefromthesame
measurements,sobothsharetheruneffect. Givenu = u,therunbeatsgroupgwithprobability
at
1
| | | | P(x | = 1 | u) | = σ(θ | +u−β | ), | σ(z) | = | , | (2) |
| --------------------------------------- | --- | --- | --- | -------- | ----- | ---- | --- | ---- | ----- | --- | --- |
| | | | atg | | | a | tg | | 1+e−z | | |
| anditsmarginisnormalwithslopeαandnoiseσ | | | | | | | , | | | | |
e
| | | | | | | ∼ | N (cid:0) α(θ | +u−h | ), σ2(cid:1) | | | |
| --- | --- | --- | --- | ---- | --- | ------------- | ---- | ------------ | --- | --- | --- |
| | | | | y at | u | | a | t | . | | (3) |
e
Ameasuredmarginentersthroughthisdensity,andamarginknownonlytobeatleast0through
| | | Φ(cid:0) | | | (cid:1) | | | | | | |
| --- | --- | -------- | --- | --- | ------- | --- | --- | --- | --- | --- | --- |
the probability α(θ +u−h )/σ , where Φ is the standard normal distribution function.
| | | | a | t | e | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
Write f (u) for this density, this probability, or 1 for a run that ties. The likelihood of the run
at
integratestheruneffectout,
Kt
| | | (cid:90) | ∏ | | | (cid:0) | | (cid:1)1−xatg | | | |
| --- | --- | -------- | --- | ---- | ----- | ------- | ---- | ------------- | --- | ----------- | --- |
| | L | = | σ(θ | +u−β | )xatg | 1−σ(θ | +u−β | ) | f | (u)ϕ (u)du, | (4) |
| | at | | | a | tg | | a | tg | at | σu | |
g=1
N(0,σ2).
where ϕ σu is the density of Gauss–Hermite quadrature with 64 nodes evaluates the
u
integral.
| Step6:setthepriors. | | | | | | | ∼ | N(0,1/λ)withλ | | = | |
| ------------------- | --- | --- | ------------------------ | --- | --- | --- | --- | ------------- | --- | ------------------- | --- |
| | | | Eachstrengthhasthepriorθ | | | | a | | | 0.35. Eachtask’stie | |
levelhash ∼ N(0,62),anditsgroupdifficultieshave β ∼ N(h +δ,τ2). Theoffsetδishow
| | t | | | | | | | tg | t | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
faratypicalgroupsitsabovethetielevel,andτ ishowwidelythegroupsscatteraroundthat
point. Thispriorkeepsagroupthatnorunbeatsatafinitedifficulty,pulledtowardthetask’s
otherevidence.
Step7: findthejointmode. Forfixedhyperparameters(σ ,α,δ,τ,σ ),thefitmaximizesthelog
| | | | | | | | | u | e | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
posterior
| | | | ∑ | | ∑ | | ∑ | | | | |
| --- | --- | --- | ---- | --- | ------ | --- | --------- | ---- | ------ | --- | --- |
| | | | logL | + | logp(θ | | )+ logp(h | ,β | ,...,β | ) | (5) |
| | | | | at | | a | | t t1 | | tKt | |
a
| | | | a,t | | | | t | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
jointlyoverthe13strengthsandthe163taskparameters,namely115groupdifficultiesand48tie
levels. ItusesNewton’smethod. Eachruntouchesonestrengthandonetask,sotheHessian
issparse: thestepforthestrengthsissolvedthroughtheirSchurcomplement,andastepthat
lowerstheobjectiveishalveduntilitdoesnot.
29
Step8: fitthehyperparameters. Thefivehyperparametersmaximizeanapproximatemarginal
likelihood. AtthejointmodeofStep7,aLaplaceapproximationintegrateseachtask’sparameters
out;thesumofthesetermsovertasks,plusthestrengths’logprioratthemode,istheobjective.
Nelder–Meadsearchesitfromtwostartingpoints,withσ cappedat7logits. Table5liststhe
u
valuesatitsmaximum,andStep7isthenrepeatedatthesevalues.
Step9: placetheagentsontheFCIscale.
Anagent’sFCIisitsstrengthrelativetoKimiK2.7’s,
| | | | = 10(θ −θ | ), | | |
| --- | --- | --- | --------- | ---- | --- | --- |
| | | FCI | a a | K2.7 | | (6) |
whereθ isthestrengthofKimiK2.7. KimiK2.7thussitsat0,andtenpointsequalonelogit:
K2.7
anagenttenpointshigherhase ≈ 2.72timestheoddsofbeatinganymetricgroup,forthesame
runeffect.
Step10: bootstraptheFCIintervals. Each of 500 bootstrap draws resamples the tasks with
replacement,soataskmayappearseveraltimesornotatall. Thedrawrefitsthestrengthsand
taskparametersfromthefullfit’smode,withthehyperparametersheldattheirStep8values,
andmapsitsstrengthstoFCIthroughitsownθ . Anagent’s90%intervalspansthe5thto95th
K2.7
| percentileofitsFCIacrossthedraws. | | | All500drawsconverge. | | | |
| --------------------------------- | --- | --- | -------------------- | --- | --- | --- |
Step11: computeeachtask’sFDI. Withthegroupdifficultiesatthemode,anagentofstrength
θ passestasktwithprobability
| | | (cid:90) | Kt | | | |
| --- | --- | -------- | --- | --- | --- | --- |
∏
| | | P(θ) = | σ(θ+u−β | )ϕ (u)du, | | (7) |
| --- | --- | ------ | ------- | --------- | --- | --- |
| | | t | | tg σu | | |
g=1
sinceapassneedseverygroupbeateninthesamerun. P riseswithθ,andthetask’sdifficulty
t
| | P(D ) = | | ItsFDIis10(D | −θ ),onthescaleofEquation(6). | | |
| ---------- | ------------------------- | --- | ------------ | ----------------------------- | --- | --- |
| D t solves | t t 1/2,foundbybisection. | | | t K2.7 | | |
For the 90% interval of the FDI, 2,000 draws of the task’s parameters come from the Laplace
Gaussianatthemode,withthestrengthsheld. Eachdrawisweightedbytheratiooftheexact
posteriortotheGaussiandensity,whichcorrectstheone-sidedposteriorofagroupthatnorun
D
beats. Theintervalspanstheweighted5thto95thpercentilesof t acrossthedraws.
| Step12: | refitFCIunderacostbudget. | | | | | |
| ------- | ------------------------- | --- | ----------------- | -------------------- | ---------- | --- |
| | | | Test-timescaling, | definedinSection5.3, | holdsevery | |
task’sparametersandthehyperparametersatthefullfit. Atabudgetof BUS$perrun,arun
thatfinishedwithin BcontributesitsoutcomesfromSteps2to4. Arunstillgoingat Bcountsas
arunwithnofinalsubmission: itbeatsnogroup,anditsmarginisthatofafinalscoreof0. The
agent’sFCIat Bisthestrengththatmaximizesitsruns’loglikelihoodplusitslogprior,found
byNewton’smethodandmappedbyEquation(6). Its90%intervalcomesfrom100bootstrap
drawsovertheagent’stasks. Theruleusesnodevelopmentscore,becausedevelopmentscores
swing: thejudgeassigns0wheneverasubmissionfailsacorrectnessorqualitycheck,yetmost
runs that drop to 0 after a positive score recover above 0 later. A rule based on development
30
Table5: ConstantsandfittedvaluesoftheFCImodel. Fixedconstantsaresetbeforethefit;fittedvalues
comefromStep8or,fortheunitfloorc,fromthedatainStep4. ValuesinlogitsarealsogiveninFCI
points,attenpointsperlogit.
| Symbol Meaning | | Logits FCIpoints | |
| -------------- | --- | ---------------- | --- |
Fixed
| λ Precisionofthestrengths’prior | | 0.35 | |
| ---------------------------------- | --- | ---- | --- |
| Priorstandarddeviationofatielevelh | t | 6 | 60 |
Gauss–Hermitenodes 64
BootstrapdrawsforFCI,andforFCIunderabudget 500,100
WeighteddrawsforanFDIinterval 2,000
Fitted
| σ u Standarddeviationoftheruneffect | | 2.041 | 20.41 |
| --------------------------------------- | --- | ----- | ----- |
| α Slopeofthemarginperlogit | | 0.597 | |
| δ Offsetofatypicalgroupabovethetielevel | | 1.276 | 12.76 |
| τ Spreadofgroupdifficulties | | 3.954 | 39.54 |
| σ Noiseofthemargin,inmarginunits | | 2.323 | |
e
| c Floorofthemarginunit,inscorepoints | | 5.78 | |
| ------------------------------------ | --- | ---- | --- |
scoreswouldalsohavetopickapatchateachbudget(thenewest,thebestsofarorthelastvalid
one)withoutheld-outverification.
31