← Back to summary
VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
Converted from PDF with markitdown. Layout may be rough.
VerifierBench: Benchmarking the Generation of Executable Verifiers for
LLM Agents
| | JianingYin1 | | | ZiyiChen2 | HaoWang3 | | YiyouSun3 | | | | |
| --- | ----------- | ---------- | --- | --------- | -------- | ----------- | --------- | --- | --- | --- | --- |
| | | ZijianJin4 | | DawnSong3 | | DaoanZhang5 | | | | | |
1UniversityofPennsylvania 2UniversityofIllinoisatUrbana-Champaign
3UniversityofCalifornia,Berkeley 4NewYorkUniversity 5UniversityofRochester
| | Abstract | | | | etal.,2026). | | Theseverifiersjudgecandidatesolu- | | | | |
| --- | -------- | --- | --- | --- | ------------ | --- | --------------------------------- | --- | --- | --- | --- |
tions,determiningwhichreceiverewardsorenter
Agentsneedreliablewaystovalidatetheirout-
6202 tcO 91 ]ES.sc[ 1v97682.0162:viXra the training data. An incorrect verifier can there-
| puts, both | when completing | | tasks and | when | | | | | | | |
| ---------- | --------------- | --- | --------- | ---- | --- | --- | --- | --- | --- | --- | --- |
forerepeatedlyrewardaviolationordiscardavalid
| filteringsynthetictrainingdata. | | | Westudyveri- | | | | | | | | |
| ------------------------------- | --- | --- | ------------ | --- | --- | --- | --- | --- | --- | --- | --- |
solution,makingthereliabilityofagent-generated
| fiergeneration: | fromtheinstructionandenvi- | | | | | | | | | | |
| --------------- | -------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
verifierscriticaltothislearningprocess.
ronmentavailableforsolvingatask,anagent
writesanexecutableverifierthatmustaccept Toexaminethisreliability,westudyverifiergen-
everyvalidsolutionandrejectinvalidones. We eration (Zhang et al., 2025; Dinella et al., 2022):
introduceVerifierBench,abenchmarkof222 given the instruction and environment a solver
tasksfromsixagentbenchmarksspanningtool-
wouldreceive,anagentwritesanexecutableveri-
call, state-based, and code verification, with fierthatreturnsPASSorFAILforanycandidate
4,993hiddentestcasesthatincludealternative
| | | | | | solution. | | Unlike a solver, | which | needs | one | valid |
| --------------- | --- | -------- | ----------------- | --- | --------- | --- | ---------------- | ----- | ----- | --- | ----- |
| valid solutions | and | attacks; | Verifier4Verifier | | | | | | | | |
solution,averifiermustacceptallvalidsolutions
(V4V),asix-checkaudit,screensitssourcever-
ifiers and test suites. Across 19 models in a whilerejectinginvalidones. Itmustthereforedis-
commonagentharness, normalizedbalanced tinguish task requirements from choices that can
accuracy ranges from 10.6 to 62.1, with gen- varyacrossvalidsolutions. Figure2showsaveri-
| erated verifiers | consistently | | rejecting | invalid | | | | | | | |
| ---------------- | ------------ | --- | --------- | ------- | --- | --- | --- | --- | --- | --- | --- |
fierrejectingapermittedread-onlyquerybecause
solutionsmorereliablythantheyacceptvalid
| | | | | | theagentomitsitstoolfromanallowlist. | | | | | Theself- | |
| --- | --- | --- | --- | --- | ------------------------------------ | --- | --- | --- | --- | -------- | --- |
ones. Thisapparentstrengthcanmaskoverly
testsomitthesamequeryandneverestablishthat
| restrictive verification: | | most | false rejections | | | | | | | | |
| ------------------------- | --- | ---- | ---------------- | --- | --- | --- | --- | --- | --- | --- | --- |
acompletevalidsolutionpasses.
| in the analyzed | records | come | from | verifiers | | | | | | | |
| --------------- | ------- | ---- | ---- | --------- | --- | --- | --- | --- | --- | --- | --- |
thatacceptnotestedsolutiontoatask. Case A verifier that accepts one expected solution
| | | | | | may | still | reject other | valid ways | to | complete | a |
| --- | --- | --- | --- | --- | --- | ----- | ------------ | ---------- | --- | -------- | --- |
studiesrevealunsupportedassumptionsabout
solution formats, tool use, and code changes task, while one that catches ordinary mistakes
| thatexcludevalidsolutions. | | | Theseerrorssur- | | | | | | | | |
| -------------------------- | --- | --- | --------------- | --- | --- | ----- | ---------------- | ---- | ------- | ---- | ------ |
| | | | | | may | still | accept solutions | that | exploit | gaps | in its |
viveself-testingbecauseagentstestsolutions checks. Evaluating verifier generation therefore
thatsharetheverifier’smistakenassumptions,
| | | | | | requires | | testing both | acceptance | of | valid alterna- | |
| --- | --- | --- | --- | --- | -------- | --- | ------------ | ---------- | --- | -------------- | --- |
orsubmitverifierswithoutconfirmingthatany
| | | | | | tives | and | rejection of | invalid | solutions, | including | |
| ---------------------- | --- | -------------------- | --- | --- | ----- | --- | ------------ | ------- | ---------- | --------- | --- |
| correctsolutionpasses. | | Thesefindingssuggest | | | | | | | | | |
testingbothwhetheraverifieracceptsacorrect deliberate attacks. Prior work evaluates gener-
atedtestassertions(Dinellaetal.,2022),unittests
solutionandwhetheritacceptsothersolutions
permittedbythetask. Theprojectpage1 and (Jainetal.,2025),issue-reproducingtests(Mündler
dataset2arepubliclyavailable.
etal.,2024),andexecutablespecifications(Endres
| | | | | | et | al., 2024; | Chen et | al., 2026). | We | extend | this |
| --- | --- | --- | --- | --- | --- | ---------- | ------- | ----------- | --- | ------ | ---- |
1 Introduction
lineofevaluationtoagenttaskcompletion,where
Indata-orientedrecursiveself-improvement(RSI) correctnesscandependontoolinteractions,result-
(Yinetal.,2025;Minetal.,2026),agentsconstruct ingenvironmentstates,andcodechanges. Tothis
| | | | | | end, | webuildVerifierBench: | | | 222tasksfromsix | | |
| --- | --- | --- | --- | --- | ---- | --------------------- | --- | --- | --------------- | --- | --- |
trainingdatafortheirownsubsequentlearningand
canjointlygeneratetasksandverificationfunctions open-sourceagentbenchmarkswith4,993hidden
thatprovidereinforcementlearningrewards(Zhou testcases,comprisingreferencesolutions,validal-
ternatives,ordinaryerrors,andattacksonverifica-
1https://verifierbench.github.io/
tion. Thisdesigntestswhethergeneratedverifiers
2https://huggingface.co/datasets/Catherine1213/VerifierBench
distinguishtasksuccessfromfailureacrossdiffer-
1
| ent ways | of carrying | | out the | same | task. | Reliable | neversees. | | | | |
| -------- | ----------- | --- | ------- | ---- | ----- | -------- | ---------- | --- | --- | --- | --- |
measurementalsorequirestrustworthyevaluation Atool-callverifierreceivesrawcalltexttogether
labels. Becauseboththebenchmarks’sourceveri- withparsedfunctionnamesandarguments;astate
fiersandourconstructedtestcasescancontainer- verifierreceivesthecandidate’sinteractionrecord
rors,weauditthemwithVerifier4Verifier(V4V), or, for AppWorld, its solution code; and a code
six checks that include replay, repeatability, and verifierreceivesthecandidatepatchandareposi-
mutation testing. Source verifiers that pass this tory checkout. For state and code tasks, both the
auditserveasthereferenceverifiers. generatingagentandthedeliveredverifiercanexe-
Evaluating19modelsinacommonharnessand cutecodeinthetaskenvironmentthroughapublic
17 further model–harness configurations, we re- interface,run_public,forexampletoreplaya
port three findings. First, generated verifiers ac- candidate’sactionsorrunarepository’stests. Exe-
cept valid solutions less reliably than they reject cutionsduringgenerationarebudgetedseparately
invalidones: asreportedinSection3.2, truepos- from those during evaluation, as documented in
| itive rates | range | from | 21.1% | to | 70.2%, | whereas | AppendixE. | | | | |
| ------------- | ----- | ----- | ----- | ---------- | ------ | ------- | --------------------------- | --- | --- | --- | --- |
| true negative | rates | range | | from 79.2% | to | 96.3%. | | | | | |
| | | | | | | | 2.2 TaskSourcesandTestCases | | | | |
Second,asanalyzedinSection3.4,thesefalsere-
jections concentrate in all-reject verifiers, which Wedrawtasksfromopen-sourceagentbenchmarks
return FAIL on every test case of their task and with executable evaluation procedures, each sup-
produce87.9–96.0%offalserejectionsinthean- plyinganinstruction,anenvironment,andasource
alyzedrecords. Third,inselectedcasesexamined verifier. AssummarizedinthetopofTable1,tool-
inSection4,theseverifiersimposechecksthatthe call tasks check called functions and arguments,
| | | | | | | | state tasks | check | interactions | or the | resulting en- |
| --- | --- | --- | --- | --- | --- | --- | ----------- | ----- | ------------ | ------ | ------------- |
taskdoesnotrequire,andtheagents’self-testsei-
therexerciseonlysolutionsthosechecksallowor vironment state, and code tasks check proposed
never confirm that any valid solution passes. We software changes. Each test case receives a con-
structionlabel,PASSorFAIL,assignedfromthe
thuscontributeabenchmarkandscreeningproce-
dureforverifiergenerationinSection2,together taskrequirementsandthechangemadetotheso-
with an analysis in Section 5 that motivates test- lution before any verifier is run. As listed in the
inggeneratedverifiersonpermittedvariations,not bottom of Table 1, test cases form four groups:
| | | | | | | | the designated | | reference | solution (gold); | valid |
| --- | --- | --- | --- | --- | --- | --- | -------------- | --- | --------- | ---------------- | ----- |
onlyonachosensolution.
variants,whichchangepresentation,implementa-
2 VerifierBench
tion,orexecutionpathwhilestillsatisfyingevery
| | | | | | | | | ordinary | errors, | | |
| ------------------- | --- | --- | --- | --- | --- | --- | -------------- | ---------- | ------- | ------- | ------------ |
| 2.1 TaskFormulation | | | | | | | requirement; | | | namely | initial out- |
| | | | | | | | comes, partial | solutions, | near | misses, | and regres- |
Fortaskt,anagentreceivestheinstructionanden-
| | | | | | | | sions; and | attacks, | which | exploit checking | omis- |
| --- | --- | --- | --- | --- | --- | --- | ---------- | -------- | ----- | ---------------- | ----- |
vironmentusedforordinarytasksolving,together
| | | | | | | | sions, fabricate | | success signals, | or interfere | with |
| -------- | ---------- | ----- | --- | ------ | ----------- | --- | ---------------- | --------------------------------- | ---------------- | ------------ | ---- |
| with the | verifier’s | input | and | output | conventions | | | | | | |
| | | | | | | | verification. | Validvariantstestwhetheraverifier | | | |
summarizedinAppendixA.Aharness,theagent
acceptsmorethanoneexpressionofacorrectso-
frameworkthatmanagesmodelcalls,tooluse,and
| | | | | | | | lution. Appendix | | C details | the nine | construction |
| --- | --- | --- | --- | --- | --- | --- | ---------------- | --- | --------- | -------- | ------------ |
thegenerationloop,runstheagentuntilitdelivers categoriesbehindthesegroups.
| anexecutableverifierV(cid:98)t | | | ;thereferenceverifierand | | | | | | | | |
| ------------------------------ | --- | --- | ------------------------ | --- | --- | --- | --- | --- | --- | --- | --- |
hiddentestcasesarewithheld. Duringgeneration, 2.3 ScreeningwithVerifier4Verifier
the agent may run self-tests: it writes candidate Thesetestcasesareusefulonlyiftheirlabelsare
solutionsofitsownandchecksthatitsdraftveri-
| | | | | | | | reliable. | Construction | labels | and source | verifiers |
| --- | --- | --- | --- | --- | --- | --- | --------- | ------------ | ------ | ---------- | --------- |
fierreturnsPASSforintendedpositivesandFAIL
| | | | | | | | can both | be wrong: | a constructed | case | may not |
| ------------ | ---------- | --- | ------- | --- | ---------- | ---- | -------- | --------- | ------------- | ---- | ------- |
| for intended | negatives. | | Because | | the hidden | test | | | | | |
changethesolutionasintended,andasourceveri-
| cases are | withheld, | self-tests | | are | the agent’s | only | | | | | |
| --------- | --------- | ---------- | --- | --- | ----------- | ---- | ------------------- | --- | --------------------- | --- | --- |
| | | | | | | | fiermaycontainbugs. | | V4Vappliessixchecksto | | |
evidenceabouthowitsverifierbehaves.
theconstructedtestcasesandsourceverifiers:
| After | generation, | | we fix | V(cid:98)t and | run it | on every | | | | | |
| ------------------------- | ----------- | --- | ------ | --------------------- | ------ | -------- | ---------- | ------------------------------- | --- | --- | --- |
| | | | | | | | 1. Replay. | Runthesourceverifieroneverytest | | | |
| testcaseinthetask’ssuiteC | | | | t . Eachtestcaseholds | | | | | | | |
onecandidatesolutionandcarriesareferencelabel case and compare its decision with the con-
| defined | in Section | 2.3, | PASS | for | a positive | case | structionlabel. | | | | |
| -------- | ---------- | ---------- | ---- | ----- | ---------- | -------- | --------------- | --- | --- | --- | --- |
| and FAIL | for | a negative | one, | which | the | verifier | | | | | |
2
Extract a source task Construct candidate solutions V4V: validate source verifiers and test suites
BFCL: parallel_114 Start with the initial task and reference solution Candidates Check construction labels
Source verifier
Toolcall verification Expand with task-specific variations and failures. Task-specific Run each candidate PASS FAIL
Generate six calls: for each sequence,
check all three mutation types. Positive candidates Negative candidates 1 Sourceverifier & label reliability 2 Test-suite sensitivity
| | | | C1 Replay | | C5 Trivial baselines | |
| -------------- | --- | ---- | ---------- | --- | --------------------- | --- |
| DNA sequences | | Base | | | | |
A: AGCTTAGCTA Gold Initial or unrepaired outcome. Compare decisions with construction labels. Valid cases → expose always-reject
| B: AGCTTAGGCTA | | | C2 Repeatability | | | |
| -------------- | --- | --- | ----------------- | --- | --- | --- |
Designated valid solution. Partial Invalid cases → expose always-accept
Reference: AGCTTAGCTA Satisfy only part of the needs. Same initial conditions → stable decisions. Valid alternatives → expose exact-match
| | C o s m e t i c | | C 3 P r o v e n a n | c e | | |
| --- | --------------- | --- | ---------------------- | --- | --- | --- |
a n a l y z e _ d n a _ s e q u ence N e a r m i s s C 6 M u t a t i o n t e s t i n g
R e q u i r e d : s e q u e n c e, V a r y p r e s e n t a t i o n w h i l e
M at c h c as e s , v e ri f i e r v er s i o n s a n d records. R e m o v e a c h e c k / a d d an u n n e c e s s a r y r u le.
| | pr e s e r v i n g ta s k v a l id i t y | . Violat e a sm a ll b u t critical condition. | C4 I n p u t – l a b | e l c o n s i s t e n c y | | |
| --- | ------------------------------------------- | ------------------------------------------------- | ------------------------ | -------------------------- | --- | --- |
re fe r e n c e _ s e q u e n c e
Same available evidence → consistent labels. C o m p a r e o r ig i n a l a n d m o d i f ie d v e r i f ie r s:
Optional: mutation_type Regression do the test cases detect the change?
| (default=insertion / deletion / | Valid alternative | Introduce a new failure or side effect. | | | | |
| -------------------------------- | ----------------- | --------------------------------------- | --- | --- | --- | --- |
substitution) Use another valid solution
Source defect Pipeline defect
Hack
| | or execution path. | Exploit verification ioopholes | Exclude the whole task. | | Repair pipeline → rerun V4V. | |
| -------------------- | --------------------------------- | ------------------------------ | ----------------------- | --- | ---------------------------- | --- |
| Public task Hidden | e.g., equivalent calls or code,
| Process violation | | | | |
+ schema solution and alternative action sequences. Violate execution constraints Retain the task + labeled hidden test suite
Only instruction + environment References, evaluation candidates and labels stay hidden from the generator. Retained hidden suite
| Generate one verifier | | | Evaluate the fixed verifier | | | |
| --------------------- | --- | --- | --------------------------- | ---- | ------------------- | --- |
| | | | | Case | Reference Generated | |
Generated verifier
| | | | | Valid reference | PASS PASS | |
| --- | --- | --- | --- | --------------- | --------- | --- |
I n s t r u ct i o n d e f v e r i f y ( c a n d i d a t e ) :
V a l i d a l t e r n a t i v e P A S S F A I L C a l c u l a t e
+ e n v i r o n m e n t M o d e l + p a r s e _ a n d _ c h e c k ( . . .)
Test cases G e n e r a t ed P a r t i a l s o l u t i o n F A I L F A I L m e t r i c s
ha r n e s s Write ↔ Self-test r e t u r n P A S S / F A I L v e r i f ie r T P R · T N R ·
T o o l s c h em a h e r e N e a r - m i s s s o l u t ion F A I L P A S S
N o r m a l i ze d B A
Figure1: VerifierBenchoverview. Sourcetasksandsourceverifiersarecombinedwithconstructedtestsuites;
V4Vscreensboth,andtheretainedsourceverifiersbecomereferenceverifiers. Theagentreceivestheinstruction
andenvironmentusedforordinarytasksolvingandgeneratesoneverifier,whichisfixedbeforeitisscoredon
multiplepositiveandnegativetestcases.
Task:credit-cardretention.Submittherequestedcardapplicationandrecordtheclosurereason.Permittedtoolsinclude
theread-onlyqueryget_user_dispute_history_7291.
Generatedcheck:Theagentcreatesatoolallowlistthatomitsthisqueryandrejectsunlistedoperations.
| Self-testsduringgeneration | | | Hiddenpositivetestcases | | | |
| -------------------------- | --- | --- | ----------------------- | --- | --- | --- |
Theintended-positivecandidateomitsthedispute-history Validcandidatesincludethepermitteddispute-historyquery.
query.
Thepositiveself-testreturnsFAIL:replayisunavailableafter Theverifierrejectsthequeryas“Unrelatedorunsupported
| budgetexhaustion. | | | mutation.” | | | |
| ----------------- | --- | --- | ---------- | --- | --- | --- |
Theagentacknowledgesthatpositivevalidationis All9positivecasesreceiveFAILwithoutenvironment
| incomplete. | | | execution. | | | |
| ----------- | --- | --- | ---------- | --- | --- | --- |
Anincompleteallowlistexcludesalegalread;self-testingneverconfirmsacceptanceofacompletevalidsolution.
Figure2: Overlookedtaskdetailsleadtoincorrectverification. Theagent’stoolallowlistomitsapermitted
read-onlyquery,causingvalidsolutionstoberejected. Itsself-testsomitthequeryandneverconfirmacceptance.
GPT-6-astrawithClaudeCodeonτ2-benchbankingtask044rejectsall9positivecases.
2. Repeatability. Rerunitonthesamecasefrom Weresolveanydisagreementbetweenaconstruc-
the same reset state; the decision must not tionlabelandasource-verifierdecisionagainstthe
| change. | | | taskrequirements;agreementalonedoesnotshow | | | |
| ------- | --- | --- | ------------------------------------------ | --- | --- | --- |
3. Provenance. Linkeveryrecordeddecisionto thateitherissemanticallycorrect.
thetest-casecontentsandverifierversionthat AsdetailedinAppendixB.1,confirmedsource
| producedit. | | | defectsexclude45ofthe267initialtasks. | | | All37 |
| ----------- | --- | --- | ------------------------------------- | --- | --- | ----- |
4. Input-labelconsistency. Withinatask,cases Terminal-Benchtasksareremoved,35ofthembe-
that give the verifier identical information causeitstest-outputparsercountsskippedtestsas
mustnotcarryoppositelabels. passes; the others are five BFCL tasks whose an-
5. Trivialbaselines. Verifiersthatalwaysaccept, swerkeysconflictwithdefault-argumentrulesand
always reject, or accept only the reference three SWE-bench tasks with parsing and empty-
solution must be exposed by negative cases, report errors. Defects in our own evaluation
positivecases,andvalidvariants,respectively. pipelinearecorrectedwithoutexcludingtasks. All
6. Mutation testing. Modify one check in the 222retainedtaskspassthesixchecks,asreported
source verifier (DeMillo et al., 1978); some inAppendixB.2,andtheirsourceverifiersbecome
testcasemustthenreceivedifferentvalidde- thereferenceverifiersV⋆. Becausereplayagrees
t
cisionsfromthetwoversions. oneverytestcase,eachconstructionlabelequals
3
thereferenceverifier’sdecision;wecallthisvalue mentsofall36configurations,andeveryreported
the reference label and score generated verifiers scorecomesfromit. Thearchivedrecordsarean
| againstit. | | | | | | | earlierexportwithper-taskjudgmentsforninePi- | | | | | | |
| ---------- | --- | --- | --- | --- | --- | --- | -------------------------------------------- | --- | --- | --- | --- | --- | --- |
harnessmodels,whoseaggregatecountsmatchthe
| 2.4 | Metrics | | | | | | mainrelease,plusearlierverifierversionsusedin | | | | | | |
| --- | ------- | --- | --- | --- | --- | --- | --------------------------------------------- | --- | --- | --- | --- | --- | --- |
AverifierinvocationyieldsavalidPASS,avalid some appendix cases; the concentration analysis
FAIL,ornovaliddecision(amissingverifier,time- in Section 3.4 uses them. The Astra trace batch
| out, crash, | or | malformed | output). | | A verifier | that | | | | | | | |
| ----------- | --- | --------- | -------- | --- | ---------- | ---- | ------------- | ---------- | --- | --- | ----------- | --------- | --- |
| | | | | | | | is a separate | generation | | of | GPT-6-astra | verifiers | |
catches its own internal error and returns FAIL underfourconditions,PiorClaudeCodeatLOW
still yields a valid decision, although it may not orHIGHreasoningeffort,thatretainsgeneration
havecompleteditsintendedcheck. Foreachtask, traces. It supports the effort comparison in Sec-
wecomputethefractionofpositivecasesreceiving tion3.3andthetraceanalysisinSection4,andits
PASS and the fraction of negative cases receiv- scoresneverreplacethoseofthemainrelease.
| ing FAIL; | the | true | positive | rate | (TPR) | and true | | | | | | | |
| -------------------------------------------- | --- | ---- | -------- | ---- | ----- | -------- | --------------- | --- | --- | --- | --- | --- | --- |
| negativerate(TNR)averagethesefractionsacross | | | | | | | 3.2 MainResults | | | | | | |
tasks,inpercent. Ourprimarymetricisnormalized Table2ranksthe19models. BA rangesfrom
norm
task-averagedbalancedaccuracy(Brodersenetal., 10.6to62.1: GPT-6-astraranksfirst,followedby
2010),whichrescalesBA = (TPR+TNR)/2as Claude Opus 4.8 (54.0) and Kimi K3 (53.4), and
nomodelapproachesperfectagreement.
| BA | = 2BA−100 | | = | TPR+TNR−100. | | | | | | | | | |
| ---- | --------- | --- | --- | ------------ | --- | --- | --------- | ----- | --------- | --- | ------ | ------ | ----- |
| norm | | | | | | | Accepting | valid | solutions | | is the | shared | weak- |
(1)
| | | | | | | | ness,asshowninFigure3. | | | Everymodelhaslower | | | |
| --- | --- | --- | --- | --- | --- | --- | ---------------------- | --- | --- | ------------------ | --- | --- | --- |
Itweightstasksequallyandthetwoclassesequally TPRthanTNR:TPRrangesfrom21.1%to70.2%,
within each task. Constant valid decisions score whereas TNR ranges from 79.2% to 96.3%. As
| 0 and | perfect | agreement | scores | | 100; missing | or | | | | | | | |
| ----- | ------- | --------- | ------ | --- | ------------ | --- | --- | --- | --- | --- | --- | --- | --- |
AppendixTable21reports,explicitFAIL,rather
| invalid | decisions | stay | in the | denominators | | with- | | | | | | | |
| ------- | --------- | ---- | ------ | ------------ | --- | ----- | --- | --- | --- | --- | --- | --- | --- |
thanamissingdecision,accountsfor57.1%–100%
outcredit. Wealsoreporttheattackfalse-positive ofeachmodel’sunacceptedpositivecases,sothe
| rate | FPR , the | task-averaged | | fraction | | of attack | | | | | | | |
| ----- | ------------ | ------------- | ---- | -------- | --- | ---------- | ---------- | -------- | ------ | --------- | --- | -------- | ----- |
| | h | | | | | | gap mainly | reflects | active | rejection | | of valid | solu- |
| cases | that receive | PASS. | More | results | | are in Ap- | | | | | | | |
tions. AsdetailedinAppendixF,thegapiswidest
| pendicesDandG). | | | | | | | onstateandcodetasks,whereTPRaveragedover | | | | | | |
| --------------- | --- | --- | --- | --- | --- | --- | ---------------------------------------- | --- | --- | --- | --- | --- | --- |
modelsis27.7%and40.2%,comparedwith79.2%
3 Experiments
ontool-calltasks.
3.1 ExperimentalSetup Lowattackacceptancedoesnotbyitselfindicate
| | | | | | | | a reliable | verifier. | As | Appendix | F | further | shows, |
| --------------------------------------------- | ------------- | ---------- | ------- | -------- | ---------- | -------- | ----------------------------- | --------- | ----------- | ----------- | ---- | ----------- | ------- |
| Models | and | harnesses. | | Our main | comparison | | | | | | | | |
| | | | | | | | on state | tasks | GPT-5.6-sol | has | an | FPR | of 0.0% |
| runs | all 19 models | | in one | common | harness, | Pi | | | | | | h | |
| | | | | | | | but accepts | only | 23.0% | of positive | | cases, | and on |
| (Zechner, | 2025). | To | measure | how | much | the har- | | | | | | | |
| | | | | | | | codetasks17modelshaveanFPR | | | | | of0.0%atthe | |
| nessmatters,weadd17alternativeconfigurations. | | | | | | | | | | | h | | |
| | | | | | | | reportedprecisionwhiletheirBA | | | | | rangesfrom | |
| Theypairmodelswitheachfamily’snativeharness, | | | | | | | | | | | norm | | |
−25.9to76.9.
| the vendor’s | | own agent | tool | (Claude | Code | (An- | | | | | | | |
| ------------ | --- | --------- | ---- | ------- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- |
thropic,2025),Codex(OpenAI,2025),orGemini 3.3 EffectsofHarnessandReasoningEffort
CLI(Google,2025)),withthethird-partyharnesses
| | | | | | | | Nativeharnesses. | | | AsTable3shows,nativehar- | | | |
| --- | --- | --- | --- | --- | --- | --- | ---------------- | --- | --- | ------------------------ | --- | --- | --- |
OpenClaw(OpenClawFoundation,2026)andmini-
| | | | | | | | nesses change | | scores | in both | directions. | | Claude |
| --- | --- | --- | --- | --- | --- | --- | ------------- | --- | ------ | ------- | ----------- | --- | ------ |
swe(Yangetal.,2024),andacrossfamilies,aswith
| | | | | | | | Code raises | Claude | | Opus 4.8 | by | 12.2 points | to |
| ----------- | --- | ------ | ----- | ------- | --- | --------- | ----------- | ------ | --- | -------- | --- | ----------- | --- |
| GPT-6-astra | in | Claude | Code. | Because | | harnesses | | | | | | | |
66.2,thebestofall36configurations,andGemini
| differ | in prompts, | tools, | and | reasoning | | settings, | | | | | | | |
| ------ | ----------- | ------ | --- | --------- | --- | --------- | --- | --- | --- | --- | --- | --- | --- |
CLIraisesGemini3.5Flashby15.0points,both
cross-harnesscomparisonsdescribeconfigurations
throughhigherTPRandhigherTNR.Incontrast,
| ratherthanmodelcapabilityalone. | | | | | Generationis | | | | | | | | |
| ------------------------------- | --- | --- | --- | --- | ------------ | --- | --- | --- | --- | --- | --- | --- | --- |
GeminiCLIlowersGemini2.5Proby9.2points,
limitedto5,400secondspertaskand32,768output
andCodexlowersallfourGPTmodelsby4.1–26.6
tokensperreply,andeveryconfigurationisscored
| | | | | | | | points. | The harnesses | | also run | with | different | rea- |
| --- | --- | --- | --- | --- | --- | --- | ------- | ------------- | --- | -------- | ---- | --------- | ---- |
onall222tasks,asdocumentedinAppendixE.
soningsettings,asnotedinAppendixE:retained
Evidencesources. Ourresultsdrawonthreesets Opus/Claude Code traces record high effort with
Themainreleaseholdsthefinaljudg-
| ofrecords. | | | | | | | thinkingenabled,whereashistoricalPitracesthat | | | | | | |
| ---------- | --- | --- | --- | --- | --- | --- | --------------------------------------------- | --- | --- | --- | --- | --- | --- |
4
| Tasktype | Sourcebenchmarks | | | | | | | | | Tasks | | PASS | FAIL |
| -------- | ---------------- | --- | --- | --- | --- | --- | --- | --- | --- | ----- | --- | ---- | ---- |
Tool-call BFCLv4(Patiletal.,2025),ACEBench(Chenetal.,2025) 95 734 2214
State τ2-bench(Barresetal.,2025),AppWorld(Trivedietal.,2024),AgentDojo 80 642 769
(Debenedettietal.,2024)
| Code | SWE-benchVerified(Jimenezetal.,2024) | | | | | | | | | | 47 | 190 | 444 |
| ----------------- | ------------------------------------ | ----- | --- | ----- | --------------------------- | --- | --- | --- | --- | --- | --- | ---- | ---- |
| Total | | | | | | | | | | | 222 | 1566 | 3427 |
| Test-casegroup | | Label | | Cases | Content | | | | | | | | |
| Referencesolution | | PASS | | 222 | Thedesignatedsolution(gold) | | | | | | | | |
Validvariants PASS 1344 Cosmeticchanges(840)oralternativeimplementationsorpaths(504)
Ordinaryerrors FAIL 2629 Initial,partial,near-miss,orregressingsolutions
Attacks FAIL 798 Exploitedcheckingomissions,fabricatedsuccesssignals,orinterference
Table1: BenchmarkcompositionafterV4Vscreening. Top: tasksandreferencelabelsbytasktype. Bottom: the
fourtest-casegroups. AppendixTables6and10givecountsbysourceandbyconstructioncategory.
| | | | | | | TPR | TNR | | | | | | |
| --- | --- | -------------------- | -------------- | --- | --- | --- | --- | --- | --- | ---- | ---- | --- | --- |
| | | | Model | | | | | | TPR | | TNR | | |
| | | | GPT-6-astra | | | | | | | 67.4 | 94.6 | | |
| | | Claude Opus 4.8 | | | | | | | | 67.7 | 86.3 | | |
| | | | Kimi K3 | | | | | | | 70.2 | 83.2 | | |
| | | | GPT-5.6-sol | | | | | | | 56.8 | 96.0 | | |
| | | | GPT-5.6-terra | | | | | | | 56.5 | 96.3 | | |
| | | | GLM-5.2 | | | | | | | 68.1 | 80.8 | | |
| | | Doubao Seed 2.1 Pro | | | | | | | | 57.0 | 89.9 | | |
| | | | GPT-5.5 | | | | | | | 61.6 | 85.1 | | |
| | | DeepSeek V4 Pro | | | | | | | | 65.5 | 79.2 | | |
| | | DeepSeek V4 Flash | | | | | | | | 60.7 | 79.8 | | |
| | | | Qwen3.7 Plus | | | | | | | 57.6 | 82.6 | | |
| | | Gemini 3.5 Flash | | | | | | | | 52.4 | 83.3 | | |
| | | | Kimi K2.6 | | | | | | | 52.5 | 82.7 | | |
| | | Doubao Seed 2.0 Code | | | | | | | | 45.0 | 84.8 | | |
| | | Doubao Seed 2.0 Lite | | | | | | | | 41.1 | 85.7 | | |
| | | Doubao Seed 2.0 Pro | | | | | | | | 27.4 | 95.4 | | |
| | | | GPT-5.1 | | | | | | | 35.2 | 87.1 | | |
| | | | Gemini 2.5 Pro | | | | | | | 31.7 | 82.5 | | |
| | | Doubao Seed 2.0 Mini | | | | | | | | 21.1 | 89.5 | | |
| | | | | 0 | 20 | 40 | 60 | 80 | 100 | | | | |
Rate (%)
Figure3:Acceptanceversusrejection.TPRandTNRforthe19modelswiththePiharness,orderedbydescending
BA . Everymodelacceptspositivecases(TPR)lessreliablythanitrejectsnegativeones(TNR).
norm
exposeathinkingsettingrecorditasoff. itivecases. AsAppendixFigure10shows,thePi
Paired code and trace inspection reveals con- verifierinsteadreliesonreplayinganexpectedex-
cretefailuresbehindthesedifferences. SomeGPT- changethatwasneversuccessfullyexecutedduring
5.5/Codex verifiers require particular patch con- generationandacceptsnone;bothrejectall8nega-
tentsorrepairlocationsandrejectvalidfixesbefore tivecases. SelectedGemini3.5Flash/GeminiCLI
anyfunctionalcheck. Onabooking-cancellation verifierslikewiseavoidimplementationerrorsthat
task, the GPT-5.6-sol/Codex verifier additionally preventvaliddecisions.
| requires | the raw | call | text to | parse | as JSON, | and | | | | | | | |
| ------------------------ | -------- | ----------- | ------- | ----------------- | ----------- | ------ | -------------------------- | ------------------------------- | ------ | ----------- | ------------- | ------- | ----- |
| | | | | | | | Otheralternativeharnesses. | | | | AsAppendixTa- | | |
| its self-test | supplies | | only | the parsed | calls, | so it | | | | | | | |
| | | | | | | | bles 11 | and 12 show, | | no harness | improves | | every |
| neverexercisesthischeck. | | | | AsAppendixFigure9 | | | | | | | | | |
| | | | | | | | modelitruns. | Codexlowersallfivemodelsitruns, | | | | | |
| illustrates, | the | verifier | accepts | none | of the | 7 pos- | | | | | | | |
| | | | | | | | including | Qwen3.7 | Plus | (−14.1 | points). | | Open- |
| itive cases, | whereas | | the Pi | verifier | accepts | all 7; | | | | | | | |
| | | | | | | | Claw | and mini-swe | lower | GPT-6-astra | | and | GPT- |
| both reject | all | 20 negative | | cases. | Conversely, | on | | | | | | | |
| | | | | | | | 5.6-sol | by 7.2–17.8 | points | but | change | Qwen3.7 | |
retail/49theClaudeOpus4.8/ClaudeCodeagent
Plusbyatmost0.3points,andClaudeCoderaises
executesthetargetexchangetoobtaintheexpected
Qwen3.7Plusby8.7pointsbutlowersGPT-5.6-sol
environmentstate,anditsverifieracceptsall9pos-
| | | | | | | | by 6.5. | The changes | also | arise | in different | | ways. |
| --- | --- | --- | --- | --- | --- | --- | ------- | ----------- | ---- | ----- | ------------ | --- | ----- |
5
| Model | | BA | | TPR | TNR FPR | Model | Native | | Pi | Native | ∆ |
| ----- | --- | --- | ---- | --- | ------- | ----- | ------ | --- | --- | ------ | --- |
| | | | norm | | h | | | | | | |
GPT-6-astra 62.1 67.4 94.6 7.4 ClaudeOpus4.8 ClaudeCode 54.0 66.2 +12.2
ClaudeOpus4.8 54.0 67.7 86.3 13.5 GPT-5.5 Codex 46.6 30.3 −16.4
| KimiK3 | | | 53.4 | 70.2 | 83.2 13.4 | GPT-5.6-sol | Codex | | 52.9 | 26.2 | −26.6 |
| ------------- | --- | --- | ---- | ---- | --------- | ------------- | ----- | --- | ---- | ---- | ----- |
| GPT-5.6-sol | | | 52.9 | 56.8 | 96.0 0.8 | | | | | | −21.8 |
| | | | | | | GPT-5.6-terra | Codex | | 52.9 | 31.0 | |
| GPT-5.6-terra | | | 52.9 | 56.5 | 96.3 3.6 | | | | | | |
| | | | | | | GPT-6-astra | Codex | | 62.1 | 58.0 | −4.1 |
GLM-5.2 48.8 68.1 80.8 12.1 Gemini3.5Flash GeminiCLI 35.7 50.7 +15.0
DoubaoSeed2.1Pro 46.8 57.0 89.9 11.4 Gemini2.5Pro GeminiCLI 14.2 5.0 −9.2
| GPT-5.5 | | | 46.6 | 61.6 | 85.1 9.9 | | | | | | |
| ------------- | --- | --- | ---- | ---- | --------- | -------------------------------- | --- | --- | --- | --- | ----- |
| DeepSeekV4Pro | | | 44.8 | 65.5 | 79.2 10.1 | | | | | | |
| | | | | | | Table3: Nativeharnesscomparison. | | | | BA | onthe |
norm
| DeepSeekV4Flash | | | 40.5 | 60.7 | 79.8 8.0 | | | | | | |
| --------------- | --- | --- | ---- | ---- | -------- | --- | --- | --- | --- | --- | --- |
same222taskswithPiandwitheachfamily’snative
| Qwen3.7Plus | | | 40.2 | 57.6 | 82.6 13.6 | | | | | | |
| ----------- | --- | --- | ---- | ---- | --------- | --- | --- | --- | --- | --- | --- |
harness. Boldmarksthehigherscoreineachpair;∆is
| Gemini3.5Flash | | | 35.7 | 52.4 | 83.3 13.2 | | | | | | |
| -------------- | --- | --- | ---- | ---- | --------- | --- | --- | --- | --- | --- | --- |
nativeminusPi,roundedindependently.
| KimiK2.6 | | | 35.2 | 52.5 | 82.7 4.6 | | | | | | |
| ----------------- | --- | --- | ---- | ---- | --------- | ---------------- | --- | ------------------------- | --- | --- | --- |
| DoubaoSeed2.0Code | | | 29.7 | 45.0 | 84.8 12.3 | | | | | | |
| DoubaoSeed2.0Lite | | | 26.8 | 41.1 | 85.7 10.5 | | | | | | |
| | | | | | | Executionbudget. | | Theclearesteffectofeffort | | | |
| DoubaoSeed2.0Pro | | | 22.8 | 27.4 | 95.4 6.8 | | | | | | |
GPT-5.1 22.2 35.2 87.1 9.7 is on the generation budget of 20 run_public
Gemini2.5Pro 14.2 31.7 82.5 4.0 executionspertask. AsAppendixTable15reports,
DoubaoSeed2.0Mini 10.6 21.1 89.5 3.7 among the 127 stateand code tasks, runs that ex-
Constant/random 0.0 — — — haustthisbudgetrisefrom4to77withPiandfrom
τ2-bench
| | | | | | | 21 to 98 | with Claude | Code. | On the | 32 | |
| --- | --- | --- | --- | --- | --- | -------- | ----------- | ----- | ------ | --- | --- |
Table2: MainresultswiththecommonPiharness, tasks, runs that exhaust it before first writing the
| on the same | 222 | tasks | and ordered | by | BA = | | | | | | |
| ------------ | --- | -------------------- | ----------- | --- | ------- | --------------------------------------- | --- | --- | --- | --- | --- |
| | | | | | norm | verifierrisefrom0to30withPiandfrom7to20 | | | | | |
| TPR+TNR−100. | | Ratesarepercentages, | | | rounded | | | | | | |
withClaudeCode.
| independently; | boldmarksthebestvaluepercolumn | | | | | | | | | | |
| -------------- | ------------------------------ | --------------------------------- | --- | --- | --- | ----------------------------------- | --- | --- | --- | --- | --- |
| (lowestforFPR | ). | Constantbaselinesreturnvaliddeci- | | | | | | | | | |
| | h | | | | | 3.4 WhereFalseRejectionsConcentrate | | | | | |
sionsthroughout,andrandomscoresareexpectations.
Table3andAppendixTable12coverthe17alternative These rates do not show whether false rejections
arespreadacrosstasksorrepeatedwithinthesame
configurations.
task. Wecallaverifierall-rejectifitreturnsavalid
| | | | | | | FAIL on | every test | case | of its task, | positive | and |
| --- | --- | --- | --- | --- | --- | ------- | ---------- | ---- | ------------ | -------- | --- |
ForQwen3.7Plus,CodexraisesTNRfrom82.6%
| | | | | | | negative | alike. On | that | task its TPR | is | 0 and its |
| -------- | ---------- | --- | ---- | ----- | --------- | -------------------------------- | --------- | ---- | ------------ | --- | --------- |
| to 92.1% | but lowers | TPR | from | 57.6% | to 34.0%, | | | | | | |
| | | | | | | TNRis100,soitcontributeszerotoBA | | | | | ,like |
norm
whereasClaudeCoderaisesbothratesforClaude
aconstantdecision.
Opus4.8,soharnesseffectscannotbereducedtoa
| | | | | | | All-reject | verifiers | produce | most | false | rejec- |
| --- | --- | --- | --- | --- | --- | ---------- | --------- | ------- | ---- | ----- | ------ |
fixedtrade-offbetweenacceptanceandrejection.
| | | | | | | tions. | As Appendix | | Figure 5 | shows, | in the |
| --- | --- | --- | --- | --- | --- | ------ | ----------- | --- | -------- | ------ | ------ |
Reasoningeffort. AsAppendixTable13reports, archived records they contribute 87.9–96.0% of
reasoning effort, by contrast, barely changes ag- each model’s positive FAIL decisions, whether
gregate scores in the Astra trace batch. With Pi, test cases or tasks are weighted equally. The As-
tratracebatchreproducesthepattern,asreported
| HIGH instead | of | LOW | effort | raises | TPR by 2.8 | | | | | | |
| ------------ | --- | --- | ------ | ------ | ---------- | --- | --- | --- | --- | --- | --- |
pointsbutlowersTNRby2.3andraisesFPR by in Appendix Table 22: in each condition, 68–75
h
2.2,foranetBA changeof+0.5;withClaude of the 222 verifiers are all-reject and contribute
norm
93.3–95.5%offalserejections.
| Code, all | three move | by | less than | one | point, and | | | | | | |
| --------- | ---------- | --- | --------- | --- | ---------- | --- | --- | --- | --- | --- | --- |
BA changesby−1.3. Thesesmalltotalshide The concentration of false rejections identifies
norm
oppositechangesbytasktype,brokendowninAp- wherefailuresoccur,butnotwhy. BA penal-
norm
pendixTable14: HIGHeffortimprovescodetasks izesthesefailuresbutdoesnotidentifythechecks
with both harnesses (+3.8 and +1.4 points) but responsibleforthem. Wethereforeinspectselected
worsens state tasks (−5.4 and −3.6), while tool- verifiersandtheirgenerationtracesinSection4.
| call tasks | change | by +3.9 | with | Pi and | −0.6 with | | | | | | |
| ---------- | ------ | ------- | ---- | ------ | --------- | --- | --- | --- | --- | --- | --- |
4 WhyGeneratedVerifiersRejectValid
| Claude Code. | Higher | | acceptance | does | not imply | | | | | | |
| ------------ | ------ | --- | ---------- | ---- | --------- | --- | --- | --- | --- | --- | --- |
Solutions
| betterdiscrimination: | | onPitool-calltasks,HIGH | | | | | | | | | |
| --------------------- | --- | ----------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
effortaccepts63morepositivecaseswithoutnewly
Section3.4showedthatfalserejectionsconcentrate
rejectingany,butitalsointroduces128falseaccep- inall-rejectverifiers. Wenowaskwhichcheckre-
tanceswhilecorrectingthree.
jectsthevalidsolutions,inSection4.1,andwhythe
6
| | | | | | Positive | task 045 | omits | the same | tool | and | also rejects | all |
| --- | --- | --- | --- | --- | -------- | -------- | ----- | -------- | ---- | --- | ------------ | --- |
FAIL
Unjustifiedcheck Tasks Verifiers ninepositives. ThreeotherPi/LOWverifiers(tasks
Self-testsusedonlyallowedsolutions 037,040,and041)rejectauthorizeduser-sidecalls
| Overbroadsource-file | | | 2 | | 3 12 | | | | | | | |
| -------------------- | --- | --- | --- | --- | ---- | ------------------------------------- | -------- | ------- | --------- | --------- | --------- | ------- |
| | | | | | | or lack | a branch | for | them, | rejecting | another | 27 |
| protection | | | | | | positiveswithoutenvironmentexecution. | | | | | | |
| Rejectspermitted | | | 3 | | 5 35 | | | | | | | |
| representations | | | | | | Source | files. | Both | Claude | Code | verifiers | for |
| Subtotal | | | 5 | | 8 47 | Matplotlib-14623 | | exclude | legal | | changes | under |
| | | | | | | lib/mpl_toolkits/, | | | including | | the | 3D axes |
Nopositiveself-testpassed
Rejectsauthorizeduser-tool 3 3 27 implementation,andeachrejectsallfourpositive
| calls | | | | | | cases. | | | | | | |
| ----------------------- | --- | --- | --- | --- | ---- | -------- | ----------- | --- | --- | ---------- | --- | ------ |
| Omitspermittedread-only | | | 2 | | 2 18 | | | | | | | |
| tools | | | | | | Category | vocabulary. | | In | retail/34, | the | Claude |
Code/HIGHverifieridentifiesthetask’sthreeoffice
| Incompletecategory | | | 2 | | 3 27 | | | | | | | |
| ------------------ | --- | --- | --- | --- | ---- | ----------------------------------------- | --- | --- | --- | --- | --- | --- |
| vocabulary | | | | | | itemswithakeywordlistthatomitsHeadphones, | | | | | | |
Subtotal 7 8 72 counts only two office items, and rejects all nine
| | | | | | | positive | cases, | as Appendix | | Figure | 8 illustrates; | |
| ----- | --- | --- | --- | --- | ------ | -------- | ------ | ----------- | --- | ------ | -------------- | --- |
| Total | | | 12 | | 16 119 | | | | | | | |
twootherHIGH-effortverifiers,forretail/33and
retail/34,sharetheomission.
Table4: UnjustifiedchecksinselectedAstraverifiers.
| Lowerblock: | nopositiveself-testreturnedacomplete | | | | | | | | | | | |
| --------------------- | ------------------------------------ | ----------- | --- | ------------ | --- | ---------------- | --- | --- | -------- | --- | -------- | ---- |
| | | | | | | Representations. | | As | detailed | in | Appendix | I.2, |
| PASSduringgeneration. | | Upperblock: | | theremaining | | | | | | | | |
theremainingverifiersacceptonlysomepermitted
verifiers,whoseself-testsusedonlysolutionsthecheck
allows. Eachverifierisatask–conditionpair;the119 representations of a value: an ISO-only currency
positive FAIL decisions cover 90 distinct test cases. format,numeric-onlydeviceidentifiers,orincom-
Thereviewistargetedanddoesnotestimateprevalence. pletedecodingofTeXspacing.
4.2 Self-TestsThatConfirmWithout
| agent’sself-testsdidnotexposeit,inSections4.2 | | | | | | Challenging | | | | | | |
| --------------------------------------------- | ----------------- | --- | ---- | --------------- | ---------- | ----------- | ------------ | ----------- | --------- | ------------- | --- | ------- |
| and 4.3. | We read generated | | code | and | generation | | | | | | | |
| | | | | | | If these | checks | are visible | in | the generated | | code, |
| tracesfromtheAstratracebatch. | | | | Foreachrejected | | | | | | | | |
| | | | | | | why did | self-testing | | not catch | them? | | Table 4 |
positivecase,wecomparetherejectedpropertyof
| | | | | | | splits the | 16 | verifiers | by self-test | | outcome. | For |
| --- | --- | --- | --- | --- | --- | ---------- | --- | --------- | ------------ | --- | -------- | --- |
thesolutionwiththepublictaskrequirementsand
| | | | | | | the eight | in the | upper | block, | the | self-tests | used |
| --- | --- | --- | --- | --- | --- | --------- | ------ | ----- | ------ | --- | ---------- | ---- |
withthegeneratedcheck,andattributetherejection
| | | | | | | only solutions | | that the | unjustified | | check | already |
| ---------------- | -------------------------------- | --- | ---------------- | --- | ---------- | -------------- | ---------------- | -------- | ----------- | ------------ | ------- | --------- |
| to that check | only when | | the requirements | | permit | | | | | | | |
| | | | | | | allows. | In Django-11734, | | | the positive | | self-test |
| theproperty. | Becausetheverifiersareselectedto | | | | | | | | | | | |
| | | | | | | patches | query.py; | | the agent | | repairs | subpro- |
| explain observed | failures, | | the analysis | | identifies | | | | | | | |
cesscompatibility,patchformatting,andenviron-
mechanismsratherthantheirprevalence.
| | | | | | | ment resets | until | its | positive | and | negative | self- |
| --- | --- | --- | --- | --- | --- | ----------- | ----- | --- | -------- | --- | -------- | ----- |
4.1 UnjustifiedChecks tests return PASS and FAIL as intended, yet
| | | | | | | no self-test | edits | an | __init__.py | | file. | The |
| ---------------------------------------- | ------------ | --- | ------- | ----- | --------- | ------------ | ---------- | ---------- | ----------- | ------ | ----- | ----- |
| Table 4 lists | 16 verifiers | | from 12 | tasks | whose re- | | | | | | | |
| | | | | | | passing | Matplotlib | self-tests | | modify | other | loca- |
| jectionsweattributetoanunjustifiedcheck: | | | | | acon- | | | | | | | |
tionsunderlib/matplotlib/,suchasaxes/
ditionthattheinstruction,tooldefinitions,anden-
_base.pyandticker.py,andtherepresenta-
| vironment | rules do | not require, | | and that | rejects | | | | | | | |
| --------- | -------- | ------------ | --- | -------- | ------- | --- | --- | --- | --- | --- | --- | --- |
tionself-testsuseonlyrepresentationsthechecks
| solutions | they permit. | One | suchcheck | | can reject | | | | | | | |
| --------- | ------------ | --- | --------- | --- | ---------- | --- | --- | --- | --- | --- | --- | --- |
accept.
severalpositivecasesofthesametask.
| | | | | | | Wecallthisconfirmatoryself-testing: | | | | | | thetests |
| --- | --- | --- | --- | --- | --- | ----------------------------------- | --- | --- | --- | --- | --- | -------- |
Inτ2-benchbankingtask044,shown
| Toolcalls. | | | | | | confirmthattheverifieracceptsachosensolution | | | | | | |
| ---------- | --- | --- | --- | --- | --- | -------------------------------------------- | --- | --- | --- | --- | --- | --- |
inFigure2,theClaudeCode/LOWverifierusesa but never ask whether its other conditions follow
tool allowlist to exclude unrelated mutations but from the task. More tests at the same permitted
omitsget_user_dispute_history_7291, locationswouldnotexposetherestriction,andnei-
apermittedread-onlyqueryincludedinthepublic therwouldnegativeself-tests: aninvalidpatchmay
toollist. Ittreatsthislegalqueryasanunsupported violateagenuinerequirementwhilealsotripping
mutationandrejectsallninepositivecasesbefore theunjustifiedcheck,sorejectingitcorrectlydoes
environment execution. The Pi/LOW verifier for notjustifythecheck. AsAppendixFigure7shows,
7
aQwen3.7Plusverifierexhibitsthesamepattern fromafailedrequirement: ablockedtestleavesits
outside the Astra batch: it treats an omitted op- checkunresolved,andrepairingtheexecutiondoes
tional argument as empty instead of applying its notvalidatechecksthetestneverexercises. Wedo
documented default, and every positive self-test notevaluatetheeffectivenessofthesedirections.
suppliesthatargumentexplicitly.
6 RelatedWork
4.3 Self-TestsThatNeverConfirmAcceptance
| | | | | | | | Executable | verifier | | generation. | | nl2postcond | |
| --------- | --------- | --- | --------- | ----- | --- | ------- | ---------- | -------- | ----- | ----------------- | --- | ----------- | ----- |
| The eight | verifiers | in | the lower | block | of | Table 4 | | | | | | | |
| | | | | | | | (Endres | et al., | 2024) | and CodeSpecBench | | | (Chen |
arethebankingandretailverifiersofSection4.1.
| | | | | | | | et al., 2026) | | generate | executable | | specifications | |
| ------------ | --- | --------------------- | --- | --- | ---------- | --- | ------------- | ---- | -------- | ---------- | --- | -------------- | --------- |
| Their agents | | ran intended-positive | | | self-tests | but | | | | | | | |
| | | | | | | | and evaluate | them | on | valid | and | faulty | behavior; |
neverobtainedacompletepositivePASS,aseach
| | | | | | | | CheckerBench | | (He | et al., | 2026) | studies | agent- |
| --- | --- | --- | --- | --- | --- | --- | ------------ | --- | --- | ------- | ----- | ------- | ------ |
agent’sfinalresponseacknowledges,sotheseruns
| | | | | | | | generated | static-analysis | | checkers. | | We | evaluate |
| ------------ | --------- | ---------- | --- | ----------- | ------ | ------- | --------------- | --------------- | --------- | --------- | ---------- | --- | ---------- |
| do not | establish | acceptance | | even | of the | agent’s | | | | | | | |
| | | | | | | | task-completion | | verifiers | for | tool-call, | | state, and |
| own intended | | solution. | In | the running | | example | | | | | | | |
codetasks,generatedfromtheinstructionanden-
| of Figure | 2, | the intended-positive | | | self-test | omits | | | | | | | |
| --------- | --- | --------------------- | --- | --- | --------- | ----- | --------------------------------- | --- | --- | --- | --- | ----------- | --- |
| | | | | | | | vironmentavailablefortasksolving. | | | | | Test-oracle | |
thepermitteddispute-historyqueryandfailswhen
| | | | | | | | andtestgeneration. | | | TOGA(Dinellaetal.,2022) | | | |
| --- | --- | --- | --- | --- | --- | --- | ------------------ | --- | --- | ----------------------- | --- | --- | --- |
replayisunavailableafterbudgetexhaustion;the
generatestestassertions,TestGenEval(Jainetal.,
| tool restriction | | remains | untested. | | At evaluation, | | | | | | | | |
| ---------------- | --------- | ------- | --------- | ----- | -------------- | -------- | --------------- | --- | --------- | ----------- | --- | --------- | -------- |
| | | | | | | | 2025) evaluates | | unit-test | generation, | | | and SWT- |
| the eight | verifiers | reject | all | 72 of | their | positive | | | | | | | |
| | | | | | | | Bench (Mündler | | et | al., 2024) | | generates | issue- |
judgmentswithoutenvironmentexecution.
reproducingteststhatalsoassesscandidatepatches.
| Generation-time | | testing | | is constrained | | more | | | | | | | |
| --------------- | ----------- | ------- | ---------------- | -------------- | --- | --------- | ---------- | --------- | -------- | --- | ----- | -------- | ------- |
| | | | | | | | Our suites | contain | multiple | | valid | variants | and vi- |
| broadly. | Exploration | | and self-testing | | | share one | | | | | | | |
| | | | | | | | olations | per task, | and | our | trace | analysis | exam- |
| execution | budget, | which | most | HIGH-effort | | state | | | | | | | |
ineswhetherself-testschallengeunjustifiedchecks.
andcoderunsexhaustasdescribedinSection3.3,
| | | | | | | | Automated | training-data | | | construction. | | Abso- |
| --- | --- | --- | --- | --- | --- | --- | --------- | ------------- | --- | --- | ------------- | --- | ----- |
and in two appendix traces shown in Appendix lute Zero (Zhao et al., 2026) learns from self-
| Figures | 9 and | 10 local | execution | | fails | because | | | | | | | |
| ------- | ----- | ------------ | --------- | ------ | ----- | ---------- | -------- | ----- | -------------------- | --- | --- | ------------- | --- |
| | | | | | | | proposed | tasks | with execution-based | | | verification. | |
| unshare | is | unavailable. | These | limits | | do not di- | | | | | | | |
Self-ChallengingLanguageModelAgents(Zhou
| rectly cause | | hidden rejections, | | since | evaluation | | | | | | | | |
| ------------ | --- | ------------------ | --- | ----- | ---------- | --- | ------------- | ------- | --- | --------- | ------ | ------------ | --- |
| | | | | | | | et al., 2026) | jointly | | generates | tasks, | verification | |
hasitsownbudget,buttheycanleaveaverifier’s
| | | | | | | | functions, | and | test examples | | for | reinforcement | |
| --- | --- | --- | --- | --- | --- | --- | ---------- | --- | ------------- | --- | --- | ------------- | --- |
checksunresolvedwhengenerationends.
| | | | | | | | learning. | We | hold tasks | | fixed | to evaluate | veri- |
| --- | --- | --- | --- | --- | --- | --- | --------- | --- | ---------- | --- | ----- | ----------- | ----- |
fiergenerationinisolation,andidentifyerrorsthat
5 Implications
couldaffectwhichsolutionsentertrainingdata.
Self-testingageneratedverifierservestwogoals:
7 Conclusion
| confirming | that | it accepts | a | chosen | solution, | and | | | | | | | |
| ---------- | ---- | ---------- | --- | ------ | --------- | --- | --- | --- | --- | --- | --- | --- | --- |
checkingthatitdoesnotrejectothersolutionsthe
VerifierBenchevaluateswhetheragentscanwrite
| task allows. | | The second | requires | | tracing | every | | | | | | | |
| ------------ | --- | ---------- | -------- | --- | ------- | ----- | --- | --- | --- | --- | --- | --- | --- |
verifiersfromtheinformationavailableforsolving
acceptanceconditiontotheinstruction,tooldefini-
| | | | | | | | a task. | Across | all 36 | configurations, | | | generated |
| --- | --- | --- | --- | --- | --- | --- | ------- | ------ | ------ | --------------- | --- | --- | --------- |
tions,orenvironmentrules,andtestingitwithcases
| | | | | | | | verifiers | accept | valid | solutions | less | reliably | than |
| ----------------------------- | --- | --- | --- | -------------- | --- | --- | --------- | ------ | ----- | --------- | ---- | -------- | ---- |
| derivedfromthoserequirements. | | | | Onetestvariesa | | | | | | | | | |
theyrejectinvalidones,andintheanalyzedrecords
permittedpropertywhilekeepingthetasksatisfied,
mostfalserejectionscomefromverifiersthatreject
forexampleeditingfields/__init__.pyin-
| | | | | | | | everycaseoftheirtask. | | | Codeandtraceinspection | | | |
| -------- | --------- | --- | -------- | --- | --------- | ----- | --------------------- | --- | --- | ---------------------- | --- | --- | --- |
| stead of | query.py, | | omitting | a | defaulted | argu- | | | | | | | |
linkstheseverifierstounjustifiedchecksandtoself-
ment,orusinganauthorizeduser-sidecall;another
teststhateitherconfirmonlysolutionsthosechecks
| violatesonerequiredcondition. | | | | Examplesdrawn | | | | | | | | | |
| ----------------------------- | --- | --- | --- | ------------- | --- | --- | ----------------------------------- | --- | --- | --- | --- | --- | -------- |
| | | | | | | | alloworneverconfirmacceptanceatall. | | | | | | Verifier |
fromsolutionsacheckalreadyallowscannotchal- generationthereforeneedsteststhatchallengeeach
| lenge that | check, | and | the Django | | configurations | | | | | | | | |
| ---------- | ------ | --- | ---------- | --- | -------------- | --- | --- | --- | --- | --- | --- | --- | --- |
acceptanceconditionagainstthetaskrequirements;
showthatpermittinglegalsourcechangesneednot
otherwise,wheregeneratedverifiersselecttraining
weakenrejectionoftestedviolations.
| | | | | | | | data, as | in data-oriented | | RSI, | such | checks | could |
| ------- | ---- | ------- | ---------- | --- | ---------- | --- | -------- | ---------------- | --- | ---- | ---- | ------ | ----- |
| Testing | must | also be | completed. | | Generation | | | | | | | | |
repeatedlyexcludelegitimatesolutionstrategies.
shouldreserveexecutionbudgetfortestingthefinal
verifierandshoulddistinguishanexecutionfailure
8
Limitations
| | | | | | | Edoardo Debenedetti, | | Jie | Zhang, | Mislav | Balunovic, | |
| --- | --- | --- | --- | --- | --- | -------------------- | --- | --- | ------ | -------- | ---------- | ------- |
| | | | | | | Luca Beurer-Kellner, | | | Marc | Fischer, | and | Florian |
Scoresmeasureagreementwithretainedreference Tramèr.2024. Agentdojo: Adynamicenvironment
toevaluatepromptinjectionattacksanddefensesfor
| verifiers | on finite, | constructed | | test suites. | As de- | | | | | | | |
| --------- | ---------- | ----------- | --- | ------------ | ------ | --- | --- | --- | --- | --- | --- | --- |
tailedinAppendixB,V4Vexcludes45taskswith llmagents. Advancesinneuralinformationprocess-
ingsystems,37:82895–82920.
confirmedsourcedefects,butagreementbetween
construction labels and reference decisions does RichardADeMillo,RichardJLipton,andFrederickG
notcertifythateverylabelissemanticallycorrect. Sayward.1978. Hintsontestdataselection:Helpfor
| | | | | | | thepracticingprogrammer. | | | | Computer,11(4):34–41. | | |
| ------ | -------------- | --- | --------- | ------- | -------- | ------------------------ | --- | --- | --- | --------------------- | --- | --- |
| The 36 | configurations | | differ in | harness | prompts, | | | | | | | |
tools,andreasoningsettings,socross-harnessdif- ElizabethDinella,GabrielRyan,ToddMytkowicz,and
| | | | | | | ShuvenduKLahiri.2022. | | | | Toga: | Aneuralmethod | |
| --- | --- | --- | --- | --- | --- | --------------------- | --- | --- | --- | ----- | ------------- | --- |
ferencesdescribeconfigurationsanddonotisolate
| | | | | | | fortestoraclegeneration. | | | InProceedingsofthe44th | | | |
| ----------- | ---------- | --- | -------- | --- | ----------- | ------------------------ | --- | --- | ---------------------- | --- | --- | --- |
| any harness | component; | | rankings | are | point esti- | | | | | | | |
InternationalConferenceonSoftwareEngineering,
mateswithoutrepeatedgenerationsorsignificance
pages2130–2141.
| tests. The | concentration | | analysis | uses | archived | | | | | | | |
| ---------- | ------------- | --- | -------- | ---- | -------- | --- | --- | --- | --- | --- | --- | --- |
per-task records for nine models and the Astra MadelineEndres,SarahFakhoury,SaikatChakraborty,
| | | | | | | andShuvenduKLahiri.2024. | | | | Canlargelanguage | | |
| --- | --- | --- | --- | --- | --- | ------------------------ | --- | --- | --- | ---------------- | --- | --- |
tracebatchforonemodel,ratherthanfinalper-task
modelstransformnaturallanguageintentintoformal
recordsforall19models. Thecodeandtraceanal- methodpostconditions? ProceedingsoftheACMon
| ysis covers | verifiers | selected | to | explain | observed | | | | | | | |
| ----------- | --------- | -------- | --- | ------- | -------- | --- | --- | --- | --- | --- | --- | --- |
SoftwareEngineering,1(FSE):1889–1912.
| failures,mostlyfromGPT-6-astra. | | | | Itattributesre- | | | | | | | | |
| ------------------------------- | --- | --- | --- | --------------- | --- | ------------ | ---------- | --- | ------------------- | --- | --- | --- |
| | | | | | | Google.2025. | GeminiCLI. | | Softwarerepository. | | | Ac- |
jectionsbyinspectingsourcecoderatherthanby
| | | | | | | cessed: | 2026-10-11. | | | | | |
| --- | --- | --- | --- | --- | --- | ------- | ----------- | --- | --- | --- | --- | --- |
interventionexperiments,anditreadsself-testout-
comesfromtraceswithoutreconstructingtheveri- Hang He, Li Wang, Hao Chen, Yuchen Shao, Yuling
Shi,LishengWang,PeiyangLiu,GooseLin,Zaiyuan
fierversionateachself-test;itthereforeidentifies
| | | | | | | Wang,HaiyingSun,etal.2026. | | | | Checkerbench: | | Can |
| --- | --- | --- | --- | --- | --- | -------------------------- | --- | --- | --- | ------------- | --- | --- |
mechanismsbutnottheirprevalence. long-horizonagentssynthesizestatic-analysischeck-
ers? arXivpreprintarXiv:2610.07557.
AIAssistanceDisclosure
| | | | | | | Kush Jain, | Gabriel | Synnaeve, | | and Baptiste | | Rozière. |
| --- | --- | --- | --- | --- | --- | ---------- | ------- | --------- | --- | ------------ | --- | -------- |
We used AI assistants for manuscript refinement, 2025. Testgeneval: A real world unit test genera-
codestructuring,anddebugging. tionandtestcompletionbenchmark. InInternational
| | | | | | | Conference | on | Learning | Representations, | | | volume |
| --- | --- | --- | --- | --- | --- | ---------- | --- | -------- | ---------------- | --- | --- | ------ |
2025,pages14947–14999.
| References | | | | | | Carlos E | Jimenez, | John | Yang, | Alexander | | Wettig, |
| -------------------------- | ----------- | ------- | ----------- | --------- | -------------- | --------------------------------- | -------- | ---------- | ---------------- | --------------- | --------------- | ------- |
| | | | | | | Shunyu | Yao, | Kexin | Pei, Ofir | Press, | and | Karthik |
| Anthropic. | 2025. | Claude | Code. | Software | repository. | | | | | | | |
| | | | | | | Narasimhan.2024. | | Swe-bench: | | Canlanguagemod- | | |
| Accessed: | 2026-10-11. | | | | | | | | | | | |
| | | | | | | elsresolvereal-worldgithubissues? | | | | | InInternational | |
| | | | | | | Conference | on | Learning | Representations, | | | volume |
| Victor Barres, | | Honghua | Dong, Soham | | Ray, Xujie Si, | | | | | | | |
| | | | | τ2-bench: | | 2024,pages54107–54157. | | | | | | |
| andKarthikNarasimhan.2025. | | | | | Evaluat- | | | | | | | |
ingconversationalagentsinadual-controlenviron-
DehaiMin,DaoanZhang,YimingZeng,HuayiZhang,
ment. arXivpreprintarXiv:2506.07982.
ZiyiChen,YanZhang,QinboBai,MengyuanChao,
KayHenningBrodersen,ChengSoonOng,KlaasEnno Jing Ning, Qiyue Hua, et al. 2026. Tracedance:
Stephan,andJoachimMBuhmann.2010. Thebal- An automated system for building agent behavior
ancedaccuracyanditsposteriordistribution. In2010 benchmarksfromreal-worldagentdeploymenttraces.
arXivpreprintarXiv:2609.33295.
20thinternationalconferenceonpatternrecognition,
pages3121–3124.IEEE.
NielsMündler,MarkNMüller,JingxuanHe,andMar-
| | | | | | | tinVechev.2024. | | Swt-bench: | | Testingandvalidating | | |
| ---------- | ------- | ---- | ------ | ---- | --------- | --------------- | --- | ---------- | --- | -------------------- | --- | --- |
| Chen Chen, | Xinlong | Hao, | Weiwen | Liu, | Xu Huang, | | | | | | | |
Xingshan Zeng, Shuai Yu, Dexun Li, Yuefeng real-worldbug-fixeswithcodeagents. Advancesin
Huang,XiangchengLiu,XinzhiWang,etal.2025. NeuralInformationProcessingSystems,37:81857–
| Acebench: | Acomprehensiveevaluationofllmtool | | | | | 81887. | | | | | | |
| ----------- | ----------------------------------- | ---- | ----------- | -------------- | -------- | ------------ | ----------- | --- | ------------------- | --------- | -------- | --- |
| usage. | InEMNLP(Findings),pages12970–12998. | | | | | | | | | | | |
| | | | | | | OpenAI.2025. | CodexCLI. | | Softwarerepository. | | | Ac- |
| | | | | | | cessed: | 2026-10-11. | | | | | |
| Zaoyu Chen, | Jianbo | Dai, | Boyu Zhu, | Jingdong | Wang, | | | | | | | |
| Huiming | Wang, | Xin | Xu, Haoyang | Yuan, | Zhijiang | | | | | | | |
| | | | | | | OpenClaw | Foundation. | | 2026. | OpenClaw. | Software | |
| Guo, | and Xiao-Ming | | Wu. 2026. | Codespecbench: | | | | | | | | |
| | | | | | | repository. | Accessed: | | 2026-10-11. | | | |
Benchmarkingllmsforexecutablebehavioralspecifi-
| cationgeneration. | | arXivpreprintarXiv:2604.12268. | | | | | | | | | | |
| ----------------- | --- | ------------------------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
9
| Shishir G Patil, | Huanzhi | Mao, | Fanjia | Yan, | Charlie |
| ---------------- | ------- | ---- | ------ | ---- | ------- |
Cheng-JieJi,VishnuSuresh,IonStoica,andJosephE
| Gonzalez.2025. | Theberkeleyfunctioncallingleader- | | | | |
| -------------------- | --------------------------------- | --------------------------- | --- | --- | --- |
| board(bfcl): | Fromtoolusetoagenticevaluationof | | | | |
| largelanguagemodels. | | InForty-secondInternational | | | |
ConferenceonMachineLearning.
HarshTrivedi,TusharKhot,MareikeHartmann,Ruskin
| Manku, Vinty | Dong, | Edward | Li, | Shashank | Gupta, |
| ----------------- | --------------------------- | ------------ | --- | ---------------- | ------ |
| Ashish Sabharwal, | | and Niranjan | | Balasubramanian. | |
| 2024. Appworld: | Acontrollableworldofappsand | | | | |
peopleforbenchmarkinginteractivecodingagents.
| In Proceedings | of | the 62nd | Annual | Meeting | of the |
| -------------- | --- | -------- | ------ | ------- | ------ |
AssociationforComputationalLinguistics(Volume
1: LongPapers),pages16022–16076.
| JohnYang, | CarlosEJimenez, | | AlexanderWettig, | | Kil- |
| --------- | --------------- | --- | ---------------- | --- | ---- |
ianLieret,ShunyuYao,KarthikRNarasimhan,and
| OfirPress.2024. | SWE-agent: | | Agent-computerinter- | | |
| ---------------------------------------- | ---------- | --- | -------------------- | --- | ----- |
| facesenableautomatedsoftwareengineering. | | | | | InThe |
Thirty-eighthAnnualConferenceonNeuralInforma-
tionProcessingSystems.
XunjianYin,XinyiWang,LiangmingPan,LiLin,Xi-
| aojunWan,andWilliamYangWang.2025. | | | | | Gödel |
| --------------------------------- | --- | --- | --- | --- | ----- |
agent: Aself-referentialagentframeworkforrecur-
| sivelyself-improvement. | | | InProceedingsofthe63rd | | |
| ----------------------- | --- | --- | ---------------------- | --- | --- |
AnnualMeetingoftheAssociationforComputational
| Linguistics(Volume1: | | LongPapers),pages27890– | | | |
| -------------------- | --- | ----------------------- | --- | --- | --- |
27913.
| Mario Zechner. | 2025. | pi: | Coding | agent. | Software |
| -------------- | --------- | ----------- | ------ | ------ | -------- |
| repository. | Accessed: | 2026-10-11. | | | |
| Lunjun Zhang, | Arian | Hosseini, | | Hritik | Bansal, |
SeyedMehranKazemi,AviralKumar,andRishabh
| Agarwal.2025. | Generativeverifiers: | | | Rewardmodel- | |
| -------------------------- | -------------------- | --- | ------------------- | ------------ | --- |
| ingasnext-tokenprediction. | | | InInternationalCon- | | |
ferenceonLearningRepresentations,volume2025,
pages12476–12505.
AndrewZhao,YiranWu,TongWu,QuentinXu,Yang
Yue,MatthieuLin,ShenzhiWang,QingyunWu,Zi-
| longZheng,andGaoHuang.2026. | | | | Absolutezero: | |
| ----------------------------------------- | ------------------ | --- | ---------- | ------------- | -------- |
| Reinforcedself-playreasoningwithzerodata. | | | | | Ad- |
| vances in | Neural Information | | Processing | | Systems, |
38:105816–105879.
YifeiZhou,SergeyLevine,JasonWeston,XianLi,and
| Sainbayar | Sukhbaatar. | 2026. | Self-challenging | | lan- |
| ----------------- | ----------- | --------------------------- | ---------------- | --- | ---- |
| guagemodelagents. | | AdvancesinNeuralInformation | | | |
ProcessingSystems,38:113959–113991.
10
A TaskContract
| | | | | | | Licensingandredistribution. | | | | OriginalVerifier- | | |
| ------- | ----------- | --- | -------- | ----- | -------- | --------------------------- | ------- | --------------- | --- | ----------------- | ------ | ---- |
| | | | | | | Bench | scripts | and annotations | | use | Apache | 2.0; |
| Table 5 | summarizes, | for | a single | task, | what the | | | | | | | |
upstream-derivedcontentretainsitssourceterms.
modelreceives,whatitmustproduce,andhowthe ACEBench,τ2-bench,AgentDojo,andtheSWE-
resultingverifierisscored.
| | | | | | | bench framework | | use | MIT | licenses; | BFCL | uses |
| ------- | --- | ------- | --- | --- | --- | --------------- | -------------------------------- | --- | --- | --------- | ---- | ---- |
| | | | | | | Apache2.0. | SWEtaskcontentretainstheapplica- | | | | | |
| Element | | Content | | | | | | | | | | |
bleAstropy,Django,orMatplotliblicensenotices.
| Modelinput | | Taskinstructionandenvironment, | | | | | | | | | | |
| ---------- | --- | ------------------------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
AppWorldprotectedcontentusesApache2.0with
identicaltothesolvingsetting;the
| | | | | | | an additional | | encryption | requirement | | | for public |
| --- | --- | --- | --- | --- | --- | ------------- | --- | ---------- | ----------- | --- | --- | ---------- |
referenceimplementationandhidden
| | | | | | | redistribution,includingderivatives. | | | | | Weomitpro- | |
| --- | --- | --- | --- | --- | --- | ------------------------------------ | --- | --- | --- | --- | ---------- | --- |
testcasesarewithheld.
tectedAppWorldtaskandsolutiontextandprovide
| Modeloutput | | OneexecutableverifierV(cid:98)t | | | fortaskt. | | | | | | | |
| ----------- | --- | ------------------------------- | --- | --- | --------- | --- | --- | --- | --- | --- | --- | --- |
metadataandscriptsforlocalreconstructionfrom
| Testsuite | | Positivecases(thereferencesolution | | | | | | | | | | |
| --------- | --- | ---------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
officialdata.
andvalidvariants)andnegativecases
(ordinaryerrorsandattacks),each
B.1 ExcludedTasks
carryingareferencelabel.
AfixedreferenceverifierV⋆retained The45exclusionscomprisefiveBFCLtaskswith
| Reference | | | | | t | | | | | | | |
| --------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
afterV4Vscreening;itsdecisionon conflicts between default-argument rules and an-
everytestcaseequalsthereference
| | | | | | | swer keys, | 35 | Terminal-Bench | | tasks | | with false |
| ------- | --- | ----------------------------------- | --- | --- | --- | ---------- | ---- | --------------- | --- | ------- | --- | ---------- |
| | | label. | | | | passes | from | the test-output | | parser, | one | Terminal- |
| Scoring | | Foreachhiddencase,compareV(cid:98)t | | | ’s | | | | | | | |
Benchtaskwithbrowserfailurestreatedassuccess-
PASS/FAILdecisionagainstthe
| | | | | | | ful checks, | one | Terminal-Bench | | task | with | incon- |
| --- | --- | --- | --- | --- | --- | ----------- | --- | -------------- | --- | ---- | ---- | ------ |
referencelabel.
| | | | | | | sistent | repeated | decisions, | and | three | SWE-bench | |
| ------- | --- | ----------------------- | --- | --- | --- | -------------------------------------- | -------- | ---------- | --- | ----- | --------- | ------ |
| Metrics | | Task-weightedTPRandTNR, | | | | | | | | | | |
| | | | | | | taskswithparsingandempty-reporterrors. | | | | | | Table7 |
| | | summarizedbyBA | | . | | | | | | | | |
norm
identifieseveryexcludedtask.
Table5: Taskcontractforverifiergenerationandevalu-
| | | | | | | Exclusionreasonsandsupportingevidence. | | | | | | D: |
| --- | --- | --- | --- | --- | --- | -------------------------------------- | --- | --- | --- | --- | --- | --- |
ation.
| | | | | | | Default-argument | | conflict. | | Public | task | require- |
| --- | --- | --- | --- | --- | --- | ---------------- | --- | --------- | --- | ------ | ---- | -------- |
mentsandthereferenceanswerkeydisagreeabout
B TaskSourcesandReferenceValidation
| | | | | | | allowedomissionofdefaultarguments. | | | | | | Evidence |
| --- | --- | --- | --- | --- | --- | ---------------------------------- | --- | --------- | ------- | ------ | --- | ---------- |
| | | | | | | comprises | 16 | test-case | records | across | | four tasks |
Table6liststhe222tasksretainedfromtheinitial
collection of 267. The initial collection also in- and a comparison of the instruction, schema, an-
swerkey,andcheckerforthefifth.
cludesTerminal-Bench,buteveryTerminal-Bench
| | | | | | | P:Falsepassesfromskippedtests. | | | | | TheTerminal- | |
| --- | --- | --- | --- | --- | --- | ------------------------------ | --- | --- | --- | --- | ------------ | --- |
taskisremovedduringscreening,sonoTerminal-
Benchtaskreachesthefrozenbenchmark;there- Bench pytest parser treats SKIPPED/XFAIL as
| | | | | | | passingoutcomes. | | Reanalysisofrecordedtestout- | | | | |
| ------------ | --------- | ---- | ----- | ------ | ------- | ---------------- | --- | ---------------------------- | --- | --- | --- | --- |
| tained tasks | therefore | form | seven | source | subsets | | | | | | | |
putprovides248recordscovering190distincttest
drawnfromsixbenchmarks,withAgentDojocon-
casesacrossall35tasks.
| tributing | separate | user-task | and | injection | subsets. | | | | | | | |
| --------- | -------- | --------- | --- | --------- | -------- | --- | --- | --- | --- | --- | --- | --- |
All main-table configurations use this complete B: Browser failures accepted. Browser
| | | | | | | startup | failure | or timeout | | is | treated | as ab- |
| --------- | ---------- | ------------- | ------- | -------- | ----------- | ------- | ------- | --------------- | --- | --- | ------- | ------ |
| task set. | The | 45 exclusions | concern | reported | de- | | | | | | | |
| | | | | | | sence | of an | alert, allowing | | the | HTML | check |
| fects in | the source | materials, | | whereas | defects in- | | | | | | | |
troducedbytheevaluationpipeline are corrected to pass; both failure types were examined for
filter-js-from-html.
withoutremovingthetask.
| | | | | | | R: Inconsistent | | decisions. | | Repeated | valid | refer- |
| ------------- | --- | ------- | -------- | --- | ----------- | --------------- | --- | ---------- | --- | -------- | ----- | ------ |
| The following | | results | describe | the | frozen task | | | | | | | |
setusedinthemaincomparison.3 ence executions return opposite decisions for the
Testcasesand
sametestcase;theevidencecontains27validruns
theirconstructionlabelsarecreatedbeforescreen-
ofcross-entropy-method.
ing. Replaycomparesthesourceverifier’soutput
| | | | | | | S:Parsingandempty-reporterrors. | | | | | | TheSWE- |
| --- | --- | --- | --- | --- | --- | ------------------------------- | --- | --- | --- | --- | --- | ------- |
withthoselabels;repeatabilitycomparesrepeated
<-
executionswithoneanother. bench parser misreads in pytest output as a
| | | | | | | test status, | and | an empty | test-count | | denominator | |
| --- | --- | --- | --- | --- | --- | ------------ | --- | -------- | ---------- | --- | ----------- | --- |
3Registry:full-upstream-
| | | | | | | receives | full | credit, allowing | | a no-op | test | case to |
| --- | --- | --- | --- | --- | --- | -------- | ---- | ---------------- | --- | ------- | ---- | ------- |
exclusion-20260919-v3.Datarevision:
swe-goldholes-native-20260919. pass. Eachofthethreetaskswascheckedinthree
nativeexecutionroundsfromthecorrectrepository
11
| Source | | | | | | | Type | | Tasks PASS | FAIL | Variants | | Attack |
| ------------------ | --- | --- | --- | --- | --- | --- | --------- | --- | ---------- | ------ | -------- | ---- | ------ |
| BFCLv4 | | | | | | | Tool-call | | 45 390 | 1247 | | 345 | 270 |
| ACEBench | | | | | | | Tool-call | | 50 344 | 967 | | 294 | 200 |
| Tool-callsubtotal | | | | | | | | | 95 734 | 2214 | | 639 | 470 |
| τ2-bench | | | | | | | State | | 32 287 | 303 | | 255 | 96 |
| AppWorld | | | | | | | State | | 28 216 | 278 | | 188 | 56 |
| AgentDojoinjection | | | | | | | State | | 10 | 60 | 86 | 50 | 42 |
| AgentDojouser | | | | | | | State | | 10 | 79 102 | | 69 | 40 |
| Statesubtotal | | | | | | | | | 80 642 | 769 | | 562 | 234 |
| SWE-benchVerified | | | | | | | Code | | 47 190 | 444 | | 143 | 94 |
| Total | | | | | | | | | 222 1566 | 3427 | | 1344 | 798 |
Allconfigurationsarescoredonthefulltaskinventory. Missingsubmissionsandinvaliddecisionsremaininthe
scoringdenominator.
Table6: Tasksources. Inventoryofthe222tasks. PASSandFAILarereferencelabels. Variantscountsvalid
variants(cosmeticchangesandalternativesolutions);Attackcombineshackandprocess_violation. These
aresubsetsofPASSandFAIL,respectively.
| base. | | | | | | | differ. Acrashorinvalidoutputdoesnotcountas | | | | | | |
| ----- | --- | --- | --- | --- | --- | --- | ------------------------------------------- | --- | --- | --- | --- | --- | --- |
adetection.
B.2 ChecksonRetainedTasks
AtaskpassesC6whenitstestsuitedetectsevery
Thefrozenscreeningrecordscontainonerowfor mutationinthetask’soperatorset,wheredetection
each of the 222 retained tasks and mark all six is defined above; all 222 retained tasks pass, so
checks as pass, giving 1,332 task–check out- each mutation across the 1,410 configurations is
comes. Table8summarizestherecordedoutcomes detectedbythecorrespondingsuite.
andtheirsupportingevidence.
| | | | | | | | Representative | | detection | evidence. | | We | addi- |
| --------------------------- | --- | --- | --- | ------------ | --- | --- | -------------- | --- | --------- | --------- | --- | --- | ----- |
| Mappingtotherecordedfields. | | | | Thescreening | | | | | | | | | |
tionallydocumentdetailedevidenceforasubsetof
recordsusethefollowingfieldnames:
configurations,drawnfromthefollow-uprecords.
Check Recordedfield ForSWE-bench,94configurationsincludeadetec-
tionexamplesupportedbyrawtestlogsmatchedto
| C1 | C1_replay_recorded | | | | | | | | | | | | |
| --- | ------------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
C2 C5a_repeatability_recorded thecandidatepatchcontents. Fortool-calltasks,87
| | | | | | | | configurations | | include | a detection | example | | from |
| --- | --------------------- | --- | --- | --- | --- | --- | ---------------- | --- | --------------------------- | ----------- | ------- | --- | ---- |
| C3 | C4_integrity_recorded | | | | | | | | | | | | |
| C4 | Det_recorded | | | | | | pairedmutations. | | Thepairedchecksaddresscases | | | | |
C5 C2_nontriviality_recorded whereanothercheckpreventsthetargetmutation
C3_mutation_recorded from changing the final decision: an additional
C6
modificationremovesthatinterferencesothatthe
| This mapping | | separates | the | paper’s | numbering | | | | | | | | |
| ------------ | --- | --------- | --- | ------- | --------- | --- | ----------------------- | --- | --- | ---------------------- | --- | --- | --- |
| | | | | | | | targetcheckcanbetested. | | | Thesetwosetsillustrate | | | |
fromthehistoricalnumberinginthesourcerecords.
thedetectionprocedurewithdifferentevidencefor-
mats;theyareexamplesratherthanthefullsetof
B.3 MutationConstructionandDetection
detectedconfigurations.
Mutationoperatorsaredefinedmanuallybyremov-
ForC6,single-targetdetectionrequirestheorig-
ingrequiredchecks,replacingconditionswithun-
| | | | | | | | inalverifierV | | anditstarget-mutatedversionV | | | | to |
| --- | --- | --- | --- | --- | --- | --- | ------------- | --- | ---------------------------- | --- | --- | --- | --- |
a
conditionalacceptance,orchanginghowstatesand
returndifferentvalidPASS/FAILdecisionsonthe
| test results | are | handled. | A | task–operator | pair | de- | | | | | | | |
| ------------ | -------------- | -------- | ---------- | ------------- | -------- | ---- | ----------------------- | ----- | -------- | --------- | ------- | ------------- | -------- |
| | | | | | | | same input | under | the same | execution | | conditions. | |
| fines one | configuration, | | rather | than | one test | case | | | | | | | |
| | | | | | | | PairedanalysiscomparesV | | | b withV | | a,b ,wherethe | |
| or one | execution. | Table | 9 lists | the 1,410 | configu- | | | | | | | | |
| | | | | | | | same auxiliary | | mutation | b is | applied | to | both; it |
| rations | by source | and | the checks | they | modify. | A | | | | | | | |
providesonlyconditionaldetectionevidenceand
| mutation | is detected | | when | the original | and | mod- | | | | | | | |
| -------- | ----------- | --- | ---- | ------------ | --- | ---- | --------------- | --- | -------------------- | --- | --------- | --- | ----- |
| | | | | | | | is not included | | in the single-target | | detection | | rate. |
ifiedverifiersbothreturnvalidPASS/FAILdeci-
| | | | | | | | The 87 | paired | configurations | | reported | above | are |
| -------- | -------- | ---- | ---- | --------- | --------- | --- | ------ | ------ | -------------- | --- | -------- | ----- | --- |
| sions on | the same | test | case | and those | decisions | | | | | | | | |
12
Source TaskID Reason
BFCL parallel_65 D
SWE astropy__astropy-7671 S
TB feal-differential-cryptanalysis P
TB catch-me-if-you-can P
TB amuse-install P
TB feal-linear-cryptanalysis P
TB code-from-image P
BFCL parallel_135 D
TB fix-git P
TB chess-best-move P
TB count-call-stack P
SWE astropy__astropy-7336 S
TB constraints-scheduling P
TB decommissioning-service-with-sensitive-data P
TB chem-property-targeting P
TB dna-insert P
TB extract-moves-from-video P
TB countdown-game P
TB dna-assembly P
TB db-wal-recovery P
BFCL parallel_180 D
TB bn-fit-modify P
TB cross-entropy-method R
TB assign-seats P
BFCL parallel_multiple_137 D
BFCL parallel_70 D
TB break-filter-js-from-html P
TB blind-maze-explorer-5x5 P
TB enemy-grid-escape P
TB deterministic-tarball P
TB 3d-model-format-legacy P
SWE astropy__astropy-7166 S
TB cartpole-rl-training P
TB acl-permissions-inheritance P
TB configure-git-webserver P
TB blind-maze-explorer-algorithm P
TB cobol-modernization P
TB cancel-async-tasks P
TB extract-safely P
TB filter-js-from-html B
TB distribution-search P
TB analyze-access-logs P
TB count-dataset-tokens P
TB crack-7z-hash P
TB aimo-airline-departures P
Table7: All45excludedtasks. BFCLdenotesBFCLv4,SWEdenotesSWE-benchVerified,andTBdenotes
Terminal-Bench. Reasoncodesrefertotheexclusioncategoriesdescribedinthissection.
13
| Check | | Recordedpasses | | Evidenceandcoverage | | | | | | | |
| ----- | --- | -------------- | --- | ------------------- | --- | --- | --- | --- | --- | --- | --- |
C1: Replay 222/222 Referencedecisionsonall4,993casesagreewiththelabelsassigned
duringconstruction.
C2: Repeatability 222/222 Repeatedexecutionsunderthesameinitialconditionsreturnstable
decisionsthatagreewiththelabels.
C3: Provenance 222/222 All4,993casefilesand222taskspecificationsmatchtheirfrozen
snapshots.
C4: Input-label 222/222 Ascanofall4,993casesfindsnoopposite-labelgroupswithidentical
| consistency | | | | publicinputswithinatask. | | | | | | | |
| ----------- | --- | --- | --- | ------------------------ | --- | --- | --- | --- | --- | --- | --- |
C5: Trivialbaselines 222/222 Everytaskhaspassingrecordsforallthreecomparisons:
always-accept,always-reject,andexact-match.
C6: Mutationtesting 222/222 Eachtask’ssuitedetectseverymutationinitsoperatorset(1,410
configurationstotal);SectionB.3reportsthecriterionand
representativedetectionevidence.
Table8: V4Voutcomesandevidence. Thesecondcolumnreportsfrozenscreeningstatuses. Checksfollowthe
C1–C6orderinSection2.3.
| Source | Tasks | Operators/task | Configurations | | Checksmodified | | | | | | |
| ------ | ----- | -------------- | -------------- | --- | -------------- | --- | --- | --- | --- | --- | --- |
BFCL 45 9 405 Requiredandextraarguments,types,values,functioncalls,
andcontainers.
ACEBench 50 11 550 Totalcalls,functionsets,per-functioncallcounts,and
arguments.
τ2-bench
| | 32 | | 3 | | 96 Communication,database,andactionrewardchecks. | | | | | | |
| --------- | --- | --- | --- | --- | ------------------------------------------------ | --- | --- | --- | --- | --- | --- |
| AgentDojo | 20 | | 2 | | 40 Textandfinal-statechecks. | | | | | | |
AppWorld 28 3 84 Answercomparison,stateassertions,andfailurehandling.
SWE-bench 47 5 235 Newlypassingandpreviouslypassingtests,missingtests,
expectedfailures,andemptyreports.
| Total | 222 | | | 1,410 | | | | | | | |
| ----- | --- | --- | --- | ----- | --- | --- | --- | --- | --- | --- | --- |
Table9: Mutationconfigurations. Operatorsalterreferencechecks;thecountsaretask–operatorpairsandarenot
numbersofindependentexecutions.
notcountedassingle-targetdetections. gories. The category records how a test case
| | | | | | | was constructed | | and its intended | positive | | or neg- |
| --- | --- | --- | --- | --- | --- | --------------- | --- | ---------------- | -------- | --- | ------- |
B.4 TargetedRechecks
| | | | | | | ative outcome; | V4V | checks | the | assigned | label |
| --- | --- | --- | --- | --- | --- | -------------- | --- | ------ | --- | -------- | ----- |
Targetedexecutionchecksprovideadditionalevi- against the source verifier before the test suite
16τ2
dence: replayon bankingtasks covers320 is fixed. Neither category names nor labels
casesandagreeswiththeirlabels;nativerechecks are supplied during generation. In the main
ofthreeretainedSWEtaskscover38cases,eachex- text, as grouped in Table 1, cosmetic and
ecutedthreetimesfromthecorrectrepositorybase, overstrict_trap cases together form the
with decisions agreeing with their labels. These valid variants; base, partial, near_miss,
| checkssupportreproducibilityforthetestedcases, | | | | | | regression | | | | | |
| ---------------------------------------------- | --- | --- | --- | --- | --- | ---------- | --- | -------- | -------- | ------- | --- |
| | | | | | | and | | form the | ordinary | errors; | and |
whilethe45exclusionsdocumentspecificsource hack and process_violation form the at-
| defectsfoundduringscreening. | | | | | | tacks. | | | | | |
| ---------------------------- | --- | --- | --- | --- | --- | ----------------- | --- | ------------ | --- | -------- | ------ |
| | | | | | | For the six-group | | diagnostics, | | ordinary | errors |
C TestCasesandConstruction
| | | | | | | combine base, | | partial, | near_miss, | | and |
| --- | --- | --- | --- | --- | --- | ------------- | --- | -------- | ---------- | --- | --- |
Categories
| | | | | | | regression | (2,629 | test | cases). | The remaining | |
| --- | --- | --- | --- | --- | --- | ---------- | ------ | ---- | ------- | ------------- | --- |
groupsarereferencesolutions(222),cosmeticvari-
| The test | suites contain | 1,566 | positive | test cases | | | | | | | |
| -------- | -------------- | ----- | -------- | ---------- | --- | --- | --- | --- | --- | --- | --- |
(31.4%) and 3,427 negative test cases (68.6%). ants (840), alternative implementations or paths
| | | | | | | (504),hacks(796),andprocessviolations(2). | | | | | The |
| -------- | --------- | -------- | ------------ | ----- | --- | ----------------------------------------- | --- | --- | --- | --- | --- |
| Table 10 | describes | the nine | construction | cate- | | | | | | | |
14
| Label | Class | | | Count | Constructionandintendeddistinction | | | | | | | | |
| ----- | ----- | --- | --- | ----- | ---------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- |
PASS gold 222 Referencesolution;testsacceptanceofthedesignatedreference
solution.
PASS cosmetic 840 Changesformattingorotherirrelevantrepresentationdetails;tests
acceptanceofequivalentpresentations.
PASS overstrict_trap
| | | | | 504 | Usesanalternativeimplementationorexecutionpath;testsacceptance | | | | | | | | |
| --- | --- | --- | --- | --- | -------------------------------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- |
ofvalidalternatives.
FAIL base 222 Usesaninitial,unrepaired,orotherwiseunfinishedoutcome;tests
recognitionofbasictaskfailure.
FAIL partial 999 Removesrequiredarguments,actions,orchanges;testswhetherall
requiredpartsarechecked.
FAIL near_miss 907 Altersaconsequentialvalueorlocalbehavior;testsdetectionofsubtle
errors.
FAIL regression 501 Introducesanadditionalfailureorsideeffect;testspreservationof
otherrequirements.
FAIL hack 796 Exploitscheckingomissions,fabricatessuccesssignals,orinterferes
withverification;testsresistancetotheseattacks.
FAIL process_violation
2 Violatesasafetyconstraint;bothcurrentexamplescomefrom
AgentDojoinjectiontasks.
| Total | | | | 4993 | 1,566PASSand3,427FAIL. | | | | | | | | |
| ----- | --- | --- | --- | ---- | ---------------------- | --- | --- | --- | --- | --- | --- | --- | --- |
Table10: Test-casecategories. Constructionandreferencelabelsforthefinal4,993testcases. Allfinalleaderboard
configurationsusethesametaskset. Testcasesarewithheldduringverifiergeneration.
lasttwogroupsformthe798attacktestcases. theretainedreferenceortheintegrityofeverycode-
taskenvironment.
| Tool-call | test | cases. | For these | test | cases, | the | | | | | | | |
| --------- | ---- | ------ | --------- | ---- | ------ | --- | --- | --- | --- | --- | --- | --- | --- |
verifier receives raw call text and a parsed repre- D MetricsandScoringProtocol
| sentation | of function | | names | and arguments. | | The | | | | | | | |
| --------- | ----------- | --- | ----- | -------------- | --- | --- | ------- | ----------- | --- | --- | ---- | ---------- | ----- |
| | | | | | | | Outcome | categories. | | For | each | test case, | P de- |
taskinstructionspecifiestherequestedoperations,
| | | | | | | | notes a | valid | PASS, | F a | valid FAIL, | and | U no |
| --- | --- | --- | --- | --- | --- | --- | ------- | ----- | ----- | --- | ----------- | --- | ---- |
andtheenvironmentprovidesthetooldefinitions.
| | | | | | | | validdecision. | | Thelastcategoryincludesmissing | | | | |
| ------- | ------------ | --- | ------- | -------- | --- | -------- | -------------- | --- | ------------------------------ | --- | --- | --- | --- |
| For the | DNA-analysis | | task in | Appendix | | I, those | | | | | | | |
mutation_type verifiers and invocations that fail to return a con-
| definitions | state | that | | | | defaults | | | | | | | |
| ---------------------------------------- | ------------- | ---------------------------- | ------- | --------------- | --- | -------- | ------------ | --------- | ----------------------------- | --- | ---------- | --- | --------- |
| | | | | | | | forming | decision. | Positive | | F outcomes | | are false |
| toinsertion; | | omittingthatoptionalargument | | | | | | | | | | | |
| | | | | | | | rejections, | while | negative | | P outcomes | | are false |
| thereforepreservestherequestedoperation. | | | | | | This | | | | | | | |
| | | | | | | | acceptances. | | BothdifferfromU,eventhoughall | | | | |
| explicit | rule provides | | a basis | for identifying | | the | | | | | | | |
receivezerocorrectnesscreditontheaffectedtest
| generated | verifier’s | incorrect | | treatment | of | the in- | | | | | | | |
| --------- | ---------- | --------- | --- | --------- | --- | ------- | -------------- | --- | ---------- | --- | ------ | ------ | ------ |
| | | | | | | | case. Optional | | confidence | | values | do not | change |
put.
| | | | | | | | thescoringlabel: | | awell-formedFAILwithlowor | | | | |
| --------- | ------ | --- | ------------------ | --- | --- | --- | ---------------- | --- | ------------------------- | --- | --- | --- | --- |
| Code test | cases. | In | task django-11740, | | | the | | | | | | | |
missingconfidenceremainsF.
instructiondescribesmissingmigrationdependen-
| | | | | | | | | | | | C+ | C− | |
| --------- | ------------------------------------- | --- | -------- | --- | ------------ | --- | ------------------------------------- | --------------- | --- | --- | --- | --- | ------ |
| | | | | | | | Equal | task weighting. | | | Let | and | be the |
| cies, and | the environment | | provides | | a repository | | | | | | t | | t |
| | | | | | | | positiveandnegativetestcasesfortaskt. | | | | | | IfA |
| checkout. | Atevaluationtime,thegeneratedverifier | | | | | | | | | | | | m,t |
countspositivetestcasesreceivingPASSandR
| receives | a candidate | | patch through | | a diff | field. | | | | | | | m,t |
| -------- | ----------- | --- | ------------- | --- | ------ | ------ | --- | --- | --- | --- | --- | --- | --- |
countsnegativetestcasesreceivingFAIL,define
| The recorded | generation | | process | inspects | | the mi- | | | | | | | |
| ------------ | ---------- | --- | ------- | -------- | --- | ------- | --- | --- | --- | --- | --- | --- | --- |
A
grationautodetectorandrunstherepository’stests r+ = m,t , (2)
m,t
| throughthepublicexecutiontool. | | | | Itthenconstructs | | | | | | |C+| | | | |
| ------------------------------ | --- | --- | --- | ---------------- | --- | --- | --- | --- | --- | ---- | --- | --- | --- |
t
| behaviorchecksfortheproposedpatchandchecks | | | | | | | | | | R | | | |
| ------------------------------------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ----- | --- | --- |
| | | | | | | | | | r− | = | m,t . | | (3) |
m,t
| existingmigrationbehavior. | | | | Thisinstancedemon- | | | | | | |C−| | | | |
| -------------------------- | --- | --- | --- | ------------------ | --- | --- | --- | --- | --- | ---- | --- | --- | --- |
t
| strates | access to | source | code | and | executed | tests, | | | | | | | |
| ------- | --------- | ------ | ---- | --- | -------- | ------ | --- | --- | --- | --- | --- | --- | --- |
Bothdenominatorsincludetestcaseswithmissing
ratherthanrelianceonlyonacandidate-authored
| | | | | | | | orinvalidoutputs. | | Alltaskshavepositiveandneg- | | | | |
| ------- | ------- | ------- | ----------------- | --- | --- | ------- | ----------------- | --- | --------------------------- | --- | --- | --- | --- |
| success | report. | It does | not independently | | | certify | | | | | | | |
ativetestcases,sobothdenominatorsarenonzero.
15
ForthefulltasksetD,scoresinpercentageunits E GenerationConfigurations
are
The Pi harness comparison contains 19 models.
100 (cid:88)
TPR = r+ , (4) GPT-5.6-sol, Qwen3.7 Plus, and GPT-6-astra ad-
m |D| m,t
t∈D ditionally use OpenClaw, mini-swe, Codex, and
100 (cid:88) ClaudeCode. GPT-5.6-terraandGPT-5.5addition-
TNR = r− , (5)
m |D| m,t ally use Codex; two Gemini models use Gemini
t∈D
CLI;andClaudeOpus4.8usesClaudeCode. This
TPR +TNR
m m
BA = , (6) gives17alternativeconfigurationsand36distinct
m
2
configurationsoverall. Table11reportsthewithin-
BA = 2BA −100. (7)
norm,m m modelcomparisons,andTable12givesfullmetrics
A missing verifier contributes zero to both class- forthe17alternativeconfigurations.
specific rates for its task. Always accepting or
Changes in acceptance and rejection. For
alwaysrejectingyieldsBA = 0whenavalid
norm
Qwen3.7Plus,CodexraisesTNRfrom82.6%with
decisionisreturnedthroughout;independentran-
the Pi harness to 92.1%, but lowers TPR from
dombinarydecisionshavethesameexpectedscore.
57.6% to 34.0%; BA falls from 40.2 to 26.1.
Perfect agreement yields 100, and no valid deci- norm
ClaudeOpus4.8withClaudeCodeimprovesboth
sionsonanytestcaseyields−100. Normalization
components: TPRrisesfrom67.7%to75.8%and
preserves the ordering induced by task-averaged
TNRfrom86.3%to90.4%,increasingBA by
BA. norm
12.2 points. Thus, changes across configurations
Attack acceptance. Let H contain the hack cannotbereducedtoanecessarytrade-offbetween
t
andprocess_violationtestcasesfortaskt. acceptance and rejection. These comparisons do
Eachofthe222retainedtaskshasatleastonesuch notisolatetheeffectofasingleharnesscomponent.
testcase. IfB countsattacktestcasesreceiving
m,t
Budgetsandtools. Thegloballimitsare5,400
avalidPASS,theattackfalse-positiverateis
secondspertaskand32,768outputtokensperre-
FPR = 100 (cid:88) B m,t . (8) ply. Thereisnoadditionalbenchmark-widecapon
h,m
|D| |H |
t total output tokens per task or context length, al-
t∈D
thoughmodelandharnesslimitsapply. Generation
Aninvalidormissingoutputdoesnotcountasan
tools permit task inspection, public environment
acceptedattack. Consequently,lowFPR mustbe
h
execution,andresetwheresupported,togetherwith
interpretedalongsidevalid-outputcoverageandthe
theharness’scode-authoringtools. Testcasesand
primaryaccuracymetric.
referenceimplementationsarewithheldduringgen-
Pooled diagnostics. Submission is the number eration.
of tasks with a submitted verifier divided by 222.
Harnessandreasoningsettings. Recordedver-
Valid-output rate is 100(P + F)/(P + F + U)
sions include Pi harness 0.84.2, Codex 0.149.0,
overtherelevanttestcases. Onpositivetestcases,
Claude Code 2.1.263, Gemini CLI 0.60.0, Open-
rejection share is 100F/(F +U), the fraction of
Claw 2026.6.34, and mini-swe-agent 2.4.6. Rea-
unaccepted test cases receiving an explicit rejec-
soningsettingsalsodiffer. Forexample,retained
tion. Thesediagnosticproportionsweighttestcases
structured Opus/Claude Code traces record high
equally, whereas the main metrics weight tasks
effort with thinking enabled, while historical Pi
equally.
harnesstracesthatexposeathinkingsettingrecord
Harnesscomparisonsandrounding. Everyfi- itasoff. Laterconfigurationsincludeothersettings.
nalharnesspairusesthesame222tasks,withmiss-
Generation instructions. The verifier-
ingresultsretainedinthedenominator. Forconfig-
generationpromptsrequireabinaryPASS/FAIL
urations a and b, ∆S = S (D)−S (D). Scores
a b
decisionandprovideoptionalscoreorconfidence
andsource-reporteddifferencesareroundedinde-
fields that do not affect the binary outcome used
pendentlytoonedecimalplace,whichcanproduce
forscoring.
a0.1-pointdiscrepancywhensubtractingdisplayed
values. Thereportedrankingusespointestimates
Retainedattempts. Thereleaseselectsthelatest
ratherthansignificancetestsorrepeated-generation
authorized attempt for a task and configuration,
estimates.
16
| | | BA | BA | | Submitted/N | Submitted/N | |
| --- | --- | ---- | ---- | --- | ----------- | ----------- | --- |
| | | norm | norm | | | | |
Model Alternative Piharness Alternative ∆BA Piharness Alternative
norm
| ClaudeOpus4.8 | ClaudeCode | 54.0 | 66.2 | +12.2 | 218/222 | 220/222 | |
| -------------- | ---------- | ---- | ---- | ----- | ------- | ------- | --- |
| GPT-6-astra | ClaudeCode | 62.1 | 61.1 | −0.9 | 222/222 | 222/222 | |
| GPT-6-astra | Codex | 62.1 | 58.0 | −4.1 | 222/222 | 219/222 | |
| GPT-6-astra | mini-swe | 62.1 | 54.9 | −7.2 | 222/222 | 222/222 | |
| GPT-6-astra | OpenClaw | 62.1 | 51.1 | −10.9 | 222/222 | 216/222 | |
| Gemini3.5Flash | GeminiCLI | 35.7 | 50.7 | +15.0 | 222/222 | 221/222 | |
| Qwen3.7Plus | ClaudeCode | 40.2 | 48.9 | +8.7 | 222/222 | 222/222 | |
| GPT-5.6-sol | ClaudeCode | 52.9 | 46.3 | −6.5 | 219/222 | 220/222 | |
| Qwen3.7Plus | mini-swe | 40.2 | 40.5 | +0.3 | 222/222 | 221/222 | |
| Qwen3.7Plus | OpenClaw | 40.2 | 39.9 | −0.3 | 222/222 | 219/222 | |
| GPT-5.6-sol | mini-swe | 52.9 | 35.8 | −17.1 | 219/222 | 218/222 | |
| GPT-5.6-sol | OpenClaw | 52.9 | 35.1 | −17.8 | 219/222 | 216/222 | |
| GPT-5.6-terra | Codex | 52.9 | 31.0 | −21.8 | 221/222 | 214/222 | |
| GPT-5.5 | Codex | 46.6 | 30.3 | −16.4 | 218/222 | 219/222 | |
| GPT-5.6-sol | Codex | 52.9 | 26.2 | −26.6 | 219/222 | 218/222 | |
| Qwen3.7Plus | Codex | 40.2 | 26.1 | −14.1 | 222/222 | 222/222 | |
| Gemini2.5Pro | GeminiCLI | 14.2 | 5.0 | −9.2 | 220/222 | 216/222 | |
ThePiharnesscolumnsreproduceTable2.Allrowsandpaireddifferencesusethesame222tasks.Source-reporteddifferences
areroundedindependentlyfromabsolutescores.
Table11:Harnesscomparison.Configurationsorderedbythealternativeconfiguration’snormalizedBA.∆BA
norm
isthealternativeharnessscoreminusthePiharnessscore,innormalized-BApoints. FullmetricsappearinTable2
forthePiharnessandinTable12forthealternatives.
| Model | Harness | | BA | TPR | TNR | | FPR |
| -------------- | ---------- | --- | ---- | ---- | ---- | --- | ---- |
| | | | norm | | | | h |
| ClaudeOpus4.8 | ClaudeCode | | 66.2 | 75.8 | 90.4 | | 7.8 |
| GPT-6-astra | ClaudeCode | | 61.1 | 68.0 | 93.2 | | 9.6 |
| GPT-6-astra | Codex | | 58.0 | 66.2 | 91.8 | | 9.2 |
| GPT-6-astra | mini-swe | | 54.9 | 61.9 | 93.0 | | 9.5 |
| GPT-6-astra | OpenClaw | | 51.1 | 63.2 | 87.9 | | 9.1 |
| Gemini3.5Flash | GeminiCLI | | 50.7 | 59.2 | 91.5 | | 7.7 |
| Qwen3.7Plus | ClaudeCode | | 48.9 | 59.7 | 89.1 | | 11.5 |
| GPT-5.6-sol | ClaudeCode | | 46.3 | 49.0 | 97.4 | | 2.3 |
| Qwen3.7Plus | mini-swe | | 40.5 | 53.3 | 87.1 | | 12.7 |
| Qwen3.7Plus | OpenClaw | | 39.9 | 51.2 | 88.6 | | 10.7 |
| GPT-5.6-sol | mini-swe | | 35.8 | 38.4 | 97.4 | | 1.1 |
| GPT-5.6-sol | OpenClaw | | 35.1 | 38.7 | 96.4 | | 0.6 |
| GPT-5.6-terra | Codex | | 31.0 | 39.2 | 91.8 | | 4.7 |
| GPT-5.5 | Codex | | 30.3 | 41.2 | 89.1 | | 6.2 |
| GPT-5.6-sol | Codex | | 26.2 | 32.3 | 93.9 | | 1.0 |
| Qwen3.7Plus | Codex | | 26.1 | 34.0 | 92.1 | | 6.5 |
| Gemini2.5Pro | GeminiCLI | | 5.0 | 23.7 | 81.3 | | 3.5 |
BA norm =TPR+TNR−100;TPR,TNR,andFPR h arepercentages.Scoresareroundedindependentlytoonedecimalplace.
BoldmarksthebestBA norm ,TPR,andTNRandthelowestFPR h amongthese17configurations.
Table 12: Full results for alternative harness configurations. The 17 configurations outside the common Pi
harness,orderedbyBA andscoredonthesame222tasksasTable2. Differencesacrossharnessesdescribe
norm
configurationsratherthanisolatedmodelcapability.
17
includingattemptsthatdonotdeliveraverifier;it betweenpositiveandnegativetestcases.
| doesnotselectthehighest-scoringattempt. | | | | | Earlier | | | | | | | |
| --------------------------------------- | --- | --- | --- | --- | ------- | -------------------------------- | --- | --- | --- | --- | ------- | --- |
| | | | | | | Executionbudgetduringgeneration. | | | | | Table15 | |
attemptsremainintheattemptmanifestandarenot
countsfinalselectedgenerationattemptsinwhich
| combined | with a later | verifier’s | judgments | | in the | | | | | | | |
| -------- | ------------ | ---------- | --------- | --- | ------ | -------- | ------------------------- | --- | --- | --- | ------- | --- |
| | | | | | | at least | one environment-execution | | | | request | ex- |
caseanalysis.
| | | | | | | ceeded | the exploration | | budget. | We | use the | gen- |
| --- | --- | --- | --- | --- | --- | ------ | --------------- | --- | ------- | --- | ------- | ---- |
Judgment rounds. Tool-call tasks use the first erationrecord’scap_budget_blockscounter;
judgment round, while state and code tasks are the judgment-stage counters are separate. The
judgedthreetimes. Foreachaggregateround,met- denominator contains the 127 state and code
ricsarecomputedoverthefull222-taskpoolwith tasks,whichsupportenvironmentexecution. The
cap_run_publiccounterincludesblockedre-
| thefirst-roundtool-calljudgmentsheldfixed. | | | | | The | | | | | | | |
| ------------------------------------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
round with median aggregate BA determines the quests and is therefore not a count of successful
| entirereportedrow;columnsarenotselectedinde- | | | | | | executions. | | | | | | |
| -------------------------------------------- | --- | --- | --- | --- | --- | ----------- | --- | --- | --- | --- | --- | --- |
pendently. Thegroupeddiagnosticsusethesame Tolocatebudgetexhaustionwithingeneration,
selectedverifiersandjudgmentroundsasthemain we compare the first native tool result contain-
table. Theseroundsexecutefixedgeneratedveri- ingBudgetExceeded:run_publicwiththe
fiersanddonotconstituteindependentgenerations. firstexplicitwritetosrc/main.py. All127se-
Nativetracesretaintask-specificpromptsandavail- lectedstateandcodetracesperconditioncontain
ableconfigurationrecords,althoughcompletesys- suchawrite. Thefinalcolumnreportsthisordering
tempromptsandstructuredtracesarenotuniformly on the 32 τ2-bench tasks. It identifies budget ex-
| available. | | | | | | haustionbeforetheverifierisfirstwritten,without | | | | | | |
| ------------------------- | --- | --- | ----------------- | --- | --- | ----------------------------------------------- | ----------- | --- | ---------- | ---------- | --- | --- |
| | | | | | | treating | tool counts | as | a complete | annotation | | of |
| Reasoning-effortablation. | | | Table13reportsthe | | | | | | | | | |
self-testing.
AstratracebatchintroducedinSection3.1,asepa-
rategenerationofGPT-6-astraverifiersunderLOW
| | | | | | | F ResultsbyTaskType | | | | | | |
| --- | --- | --- | --- | --- | --- | ------------------- | --- | --- | --- | --- | --- | --- |
andHIGHreasoningeffortwiththePiharnessand
| | | | | | | Tables | 16–18 | disaggregate | | the 19 | Pi harness | |
| ------ | -------------------- | --- | ---- | ------- | ------ | ------ | ----- | ------------ | --- | ------ | ---------- | --- |
| Claude | Code. Each condition | | uses | all 222 | tasks, | | | | | | | |
a5,400-secondlimitpertask,andatmost32,768 configurationsinTable2into95tool-call,80state,
| | | | | | | and 47 | code tasks. | They | use | the same | generated | |
| ------ | -------------------- | --- | --------- | ---------- | --- | --------- | ----------- | -------- | ------ | -------- | ----------- | --- |
| output | tokens per response. | | For tasks | supporting | | | | | | | | |
| | | | | | | verifiers | and | judgment | rounds | as | the overall | |
environmentexecution,theexplorationbudgetal-
lows20run_publicexecutionspertask,shared ranking. Eachtaskreceivesequalweightwithinits
by direct tool calls and calls through vgb_env. type,andmissingorinvalidoutputsremaininthe
| | | | | | | denominator. | | Ranks | follow unrounded | | BA | ; |
| -------------------------------------------- | --- | --- | --- | --- | --- | ------------ | --- | ----- | ---------------- | --- | --- | ---- |
| Thislimitdoesnotcountalltoolcallsorconversa- | | | | | | | | | | | | norm |
displayedvaluesareroundedtoonedecimalplace.
| tionturns. | Allfourconditionsdeliververifiersfor | | | | | | | | | | | |
| ---------- | ------------------------------------ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
all 222 tasks. Scoring follows the same full-task The additional harness configurations are not
includedinthisbreakdown.
andmedian-roundprotocolasthemainevaluation.
Theablationconditionsarenotsubstitutedforthe
| | | | | | | Model | rankings | depend | on | task | type. | GPT- |
| ---------- | ---------- | --------- | ---- | ---------- | --- | ----- | -------- | ------ | --- | ---- | ----- | ---- |
| main-table | Pi harness | or Claude | Code | configura- | | | | | | | | |
5.6-solleadstool-callverificationat85.2,Claude
tions.
| | | | | | | Opus 4.8 | leads | state-based | verification | | at | 34.6, |
| --- | --- | --- | --- | --- | --- | -------- | ----- | ----------- | ------------ | --- | --- | ----- |
Variation across task types. Table 14 decom- and Kimi K3 leads code-based verification at
| | | | | | | 76.9. Ranks | also | reverse: | Kimi | K2.6 | ranks | third |
| --- | --- | --- | --- | --- | --- | ----------- | ---- | -------- | ---- | ---- | ----- | ----- |
posesthereasoning-effortcomparisonbytasktype,
| | | | | | | on tool-call | tasks | but | seventeenth | on | state | tasks, |
| --------- | ------------- | --------- | --- | ------------ | --- | ------------ | ----- | --- | ----------- | --- | ----- | ------ |
| using the | same selected | verifiers | | and judgment | | | | | | | | |
rounds as Table 13. Within each type, BA whereas Qwen3.7 Plus ranks fourteenth and
norm
givestasksequalweight. Thechangeshavediffer- second, respectively. DeepSeek V4 Pro ranks
| | | | | | | third on | code | tasks and | fifteenth | on | state | tasks. |
| -------------------- | --- | --------------------- | --- | --- | --- | -------- | ---- | --------- | --------- | --- | ----- | ------ |
| entsignsacrosstypes: | | HIGHimprovescode-task | | | | | | | | | | |
Thesecomparisonsconcernthebenchmark’stask
| scores with | both harnesses | but | reduces | state-task | | | | | | | | |
| ----------- | -------------- | --- | ------- | ---------- | --- | --- | --- | --- | --- | --- | --- | --- |
scores. In the Pi harness tool-call subset, HIGH groups, which also differ in source benchmarks,
environments,andtest-casecomposition;theydo
additionallyaccepts63positivetestcases,withno
notisolatetheeffectoftasktypealone.
| positive | test case changing | | from accepted | | to re- | | | | | | | |
| -------- | ------------------ | --- | ------------- | --- | ------ | --- | --- | --- | --- | --- | --- | --- |
jected;italsointroduces128falseacceptancesand
Theacceptancedeficitisstrongestonstateand
| correctsthree. | Thus,theincreaseinpositiveaccep- | | | | | | | | | | | |
| -------------- | -------------------------------- | --- | --- | --- | --- | ---------- | --------------------------- | --- | --- | --- | --- | --- |
| | | | | | | codetasks. | Acrossthe19models,meanTPRis | | | | | |
tancedoesnotuniformlyimprovediscrimination
79.2%ontool-calltasks,27.7%onstatetasks,and
18
| Harness | | Effort | | BA | TPR | TNR | FPR | | Submitted/N | |
| ---------- | --- | --------- | --- | ---- | ---- | ---- | ---- | --- | ----------- | ------- |
| | | | | norm | | | | h | | |
| Piharness | | LOW | | 59.0 | 64.0 | 95.0 | | 7.3 | | 222/222 |
| Piharness | | HIGH | | 59.5 | 66.8 | 92.7 | | 9.5 | | 222/222 |
| | | ∆HIGH–LOW | | +0.5 | +2.8 | −2.3 | +2.2 | | | 0 |
| ClaudeCode | | LOW | | 60.5 | 67.2 | 93.3 | | 9.4 | | 222/222 |
| ClaudeCode | | HIGH | | 59.2 | 66.6 | 92.6 | | 9.7 | | 222/222 |
| | | ∆HIGH–LOW | | −1.3 | −0.6 | −0.7 | +0.4 | | | 0 |
Table13:Reasoning-effortablationforGPT-6-astra.Eachconditionusesthesame222tasksandstatedgeneration
budgets. Differencesarecomputedfromunroundedscoresandmaydifferfromsubtractionofdisplayedvalues.
ThesefourconditionsformtheAstratracebatch,whichisseparatefromthemainreleaseinTable2.
τ2before
| Tasktype | N | Piharness | ClaudeCode | | | | | State/code | | |
| ----------------------------------------------- | --- | ---------------------- | ---------- | ---- | ---------- | ------ | --- | ---------- | --- | ---------- |
| | | | | | Harness | Effort | | blocked | | firstwrite |
| Tool-call | 95 | +3.9 | | −0.6 | | | | | | |
| | | | | | Piharness | LOW | | 4/127 | | 0/32 |
| State | 80 | −5.4 | | −3.6 | | | | | | |
| Code | 47 | +3.8 | | +1.4 | Piharness | HIGH | | 77/127 | | 30/32 |
| | | | | | ClaudeCode | LOW | | 21/127 | | 7/32 |
| | | | | | ClaudeCode | HIGH | | 98/127 | | 20/32 |
| Table14: Reasoning-effortdifferencesbytasktype. | | | | | | | | | | |
| HIGHminusLOW,inBA | | points,intheAstratrace | | | | | | | | |
norm
| | | | | | Table15: | Execution-budgetexhaustionduringgener- | | | | |
| --- | --- | --- | --- | --- | -------- | -------------------------------------- | --- | --- | --- | --- |
batch. N isthenumberoftasksineachtype.
ation. Ataskiscountedonceifanyrequestexceedsthe
| | | | | | generationbudget. | Thefinalcolumncountsτ2-bench | | | | |
| --- | --- | --- | --- | --- | ----------------- | ---------------------------- | --- | --- | --- | --- |
taskswhosefirstblockedrequestprecedesthefirstwrite
| | | | | | oftheverifier. | Thesearegeneration-stageobservations, | | | | |
| --- | --- | --- | --- | --- | -------------- | ------------------------------------- | --- | --- | --- | --- |
notruntimefailuresduringhiddenevaluation.
| Rank | Model | | | BA | ↑ | TPR↑ | TNR↑ | | | FPR ↓ |
| ---- | ----------------- | --- | --- | --- | ---- | ---- | ---- | ---- | --- | ----- |
| | | | | | norm | | | | | h |
| 1 | GPT-5.6-sol | | | | 85.2 | 86.5 | | 98.7 | | 1.8 |
| 2 | GPT-6-astra | | | | 83.0 | 92.8 | | 90.2 | | 17.1 |
| 3 | KimiK2.6 | | | | 79.6 | 87.1 | | 92.5 | | 10.4 |
| 4 | GLM-5.2 | | | | 75.9 | 93.3 | | 82.6 | | 25.7 |
| 5 | DeepSeekV4Flash | | | | 74.7 | 87.8 | | 86.9 | | 18.2 |
| 6 | GPT-5.6-terra | | | | 74.6 | 77.6 | | 96.9 | | 6.1 |
| 7 | KimiK3 | | | | 72.8 | 92.2 | | 80.6 | | 31.0 |
| 8 | DeepSeekV4Pro | | | | 71.0 | 88.3 | | 82.7 | | 22.7 |
| 9 | ClaudeOpus4.8 | | | | 68.0 | 92.1 | | 75.9 | | 31.1 |
| 10 | DoubaoSeed2.1Pro | | | | 67.3 | 83.2 | | 84.1 | | 22.5 |
| 11 | GPT-5.5 | | | | 65.6 | 87.4 | | 78.2 | | 23.2 |
| 12 | Gemini3.5Flash | | | | 64.6 | 87.9 | | 76.7 | | 30.5 |
| 13 | GPT-5.1 | | | | 59.1 | 73.9 | | 85.2 | | 20.4 |
| 14 | Qwen3.7Plus | | | | 57.5 | 73.7 | | 83.9 | | 21.9 |
| 15 | DoubaoSeed2.0Code | | | | 56.2 | 73.7 | | 82.5 | | 23.3 |
| 16 | Gemini2.5Pro | | | | 54.8 | 62.8 | | 92.0 | | 8.1 |
| 17 | DoubaoSeed2.0Lite | | | | 53.7 | 70.9 | | 82.9 | | 20.4 |
| 18 | DoubaoSeed2.0Pro | | | | 44.3 | 52.2 | | 92.1 | | 13.8 |
| 19 | DoubaoSeed2.0Mini | | | | 26.9 | 42.2 | | 84.7 | | 7.8 |
Table16: Tool-callverificationranking. All19modelsusethePiharness,evaluatedonthesame95tasks. Rates
| arepercentages;BA | | isreportedinpoints. | | Ranksuseunroundedscores. | | | | | | |
| ----------------- | --- | ------------------- | --- | ------------------------ | --- | --- | --- | --- | --- | --- |
norm
19
| Rank | Model | BA ↑ | TPR↑ | TNR↑ | FPR ↓ |
| ---- | ----------------- | ----- | ---- | ---- | ----- |
| | | norm | | | h |
| 1 | ClaudeOpus4.8 | 34.6 | 41.2 | 93.3 | 0.5 |
| 2 | Qwen3.7Plus | 30.7 | 49.0 | 81.7 | 10.5 |
| 3 | DoubaoSeed2.1Pro | 30.0 | 36.1 | 93.9 | 3.6 |
| 4 | GPT-5.6-terra | 30.0 | 35.0 | 95.0 | 2.8 |
| 5 | GPT-6-astra | 29.1 | 31.0 | 98.1 | 0.2 |
| 6 | Gemini3.5Flash | 27.6 | 33.6 | 94.0 | 0.5 |
| 7 | GPT-5.5 | 19.0 | 32.3 | 86.7 | 0.0 |
| 8 | KimiK3 | 16.5 | 38.6 | 77.9 | 0.5 |
| 9 | GPT-5.6-sol | 16.1 | 23.0 | 93.1 | 0.0 |
| 10 | DoubaoSeed2.0Code | 13.9 | 27.5 | 86.4 | 6.5 |
| 11 | GLM-5.2 | 11.3 | 39.9 | 71.4 | 3.0 |
| 12 | DoubaoSeed2.0Lite | 9.2 | 21.1 | 88.1 | 5.0 |
| 13 | DoubaoSeed2.0Pro | 5.0 | 7.5 | 97.5 | 2.5 |
| 14 | DoubaoSeed2.0Mini | 0.8 | 4.7 | 96.1 | 0.9 |
| 15 | DeepSeekV4Pro | 0.6 | 33.8 | 66.9 | 1.1 |
| 16 | GPT-5.1 | -4.8 | 9.8 | 85.4 | 2.8 |
| 17 | KimiK2.6 | -5.6 | 23.6 | 70.8 | 0.5 |
| 18 | DeepSeekV4Flash | -7.8 | 28.3 | 63.8 | 0.5 |
| 19 | Gemini2.5Pro | -10.6 | 10.9 | 78.5 | 1.5 |
Table17: State-basedverificationranking. All19modelsusethePiharness,evaluatedonthesame80tasks.
Ratesarepercentages;BA isreportedinpoints. Ranksuseunroundedscores.
norm
| Rank | Model | BA ↑ | TPR↑ | TNR↑ | FPR ↓ |
| ---- | ----------------- | ----- | ---- | ---- | ----- |
| | | norm | | | h |
| 1 | KimiK3 | 76.9 | 79.5 | 97.5 | 0.0 |
| 2 | GPT-6-astra | 75.8 | 78.3 | 97.5 | 0.0 |
| 3 | DeepSeekV4Pro | 66.8 | 73.6 | 93.2 | 0.0 |
| 4 | ClaudeOpus4.8 | 59.0 | 63.4 | 95.6 | 0.0 |
| 5 | GLM-5.2 | 58.1 | 65.1 | 93.0 | 0.0 |
| 6 | GPT-5.5 | 55.4 | 59.1 | 96.2 | 0.0 |
| 7 | DeepSeekV4Flash | 53.9 | 61.3 | 92.7 | 0.0 |
| 8 | GPT-5.6-sol | 50.1 | 54.5 | 95.6 | 0.0 |
| 9 | GPT-5.6-terra | 47.9 | 50.6 | 97.3 | 0.0 |
| 10 | DoubaoSeed2.1Pro | 34.2 | 39.5 | 94.8 | 2.1 |
| 11 | Qwen3.7Plus | 21.4 | 40.0 | 81.4 | 2.1 |
| 12 | KimiK2.6 | 15.0 | 31.9 | 83.1 | 0.0 |
| 13 | DoubaoSeed2.0Pro | 9.9 | 11.2 | 98.7 | 0.0 |
| 14 | DoubaoSeed2.0Code | 3.1 | 16.6 | 86.5 | 0.0 |
| 15 | DoubaoSeed2.0Lite | 2.3 | 14.9 | 87.4 | 0.0 |
| 16 | DoubaoSeed2.0Mini | -5.6 | 6.4 | 88.0 | 0.0 |
| 17 | GPT-5.1 | -6.4 | 0.0 | 93.6 | 0.0 |
| 18 | Gemini3.5Flash | -8.8 | 12.8 | 78.5 | 0.0 |
| 19 | Gemini2.5Pro | -25.9 | 4.3 | 69.9 | 0.0 |
Table18: Code-basedverificationranking. All19modelsusethePiharness,evaluatedonthesame47tasks.
Ratesarepercentages;BA isreportedinpoints. Ranksuseunroundedscores.
norm
20
H Task-LevelErrorAnalysis
| 40.2% | oncode | tasks, | whilemean | | TNR | is85.8%, | | | | | | | |
| ----------- | ---------- | ------------- | --------- | ------------- | ----- | ---------- | -------- | ----------- | --- | --- | ---------- | --- | -------- |
| 85.2%, | and 90.6%, | respectively. | | | These | are arith- | | | | | | | |
| | | | | | | | Analysis | population. | | The | task-level | | analysis |
| metic means | | of the | models’ | task-averaged | | rates. | | | | | | | |
usesthearchivedrecordsdescribedinSection3.1,
All19modelshavelowerTPRthanTNRonstate
anOctober3exportofselectedjudgmentsforGPT-
| and code | tasks, | but | eight | have | higher | TPR than | | | | | | | |
| -------- | ------ | --- | ----- | ---- | ------ | -------- | --- | --- | --- | --- | --- | --- | --- |
6-astra,KimiK3,GPT-5.6-sol,Qwen3.7Plus,GPT-
| TNRontool-calltasks. | | | Theaggregateasymmetry | | | | | | | | | | |
| -------------------- | --- | --- | --------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
5.5,GPT-5.6-terra,GPT-5.1,ClaudeOpus4.8,and
thereforecoexistswithadifferentpatternonsome
| | | | | | | | Gemini2.5ProwiththePiharness. | | | | | Eachconfig- | |
| ----------- | ---------- | --- | --- | ------- | ----- | --- | ----------------------------- | ------ | ------- | ----- | --- | ----------- | ------ |
| task types. | State-task | | TPR | remains | below | 50% | | | | | | | |
| | | | | | | | uration | covers | all 222 | tasks | and | has 4,993 | evalu- |
foreverymodel,evenforthehighest-rankedcon-
| | | | | | | | ation records, | | one per | test | case. | For these | nine |
| ----------- | --------------------------------- | --- | --- | --- | --- | --- | -------------- | ------------ | ------- | --------- | ----- | --------- | --------- |
| figuration. | Lowpositiveacceptanceincludesboth | | | | | | | | | | | | |
| | | | | | | | models, | the archived | | aggregate | | scores | and every |
explicitrejectionandmissingorinvaliddecisions;
| | | | | | | | N/P/F/U | countinthesixtest-casegroupsagree | | | | | |
| --- | --- | --- | --- | --- | --- | --- | ------- | --------------------------------- | --- | --- | --- | --- | --- |
theratesalonecannotdistinguishtheircauses.
| | | | | | | | with the | final | release. | The | archive | predates | the |
| --- | --- | --- | --- | --- | --- | --- | -------- | ----- | -------- | --- | ------- | -------- | --- |
Low attack acceptance does not establish finalrejudging,sothisagreementestablishescon-
reliableverification. Onstatetasks,GPT-5.6-sol sistencyattheaggregatelevelratherthanidentity
has FPR = 0.0% but accepts only 23.0% of of every task-level decision. Gemini 3.5 Flash is
h
positives,yieldingBA = 16.1. Oncodetasks, excluded from this analysis because its archived
norm
17modelshaveFPR = 0.0%atthereportedpre- judgmentsdifferfromthefinalaggregates;all19
h
cision,yettheirBA scoresrangefrom−25.9 modelsremainintheleaderboardandgroupeddiag-
norm
to76.9. Alowattack-acceptanceratemustthere- nostics. Thenine-modelconcentrationstatisticsare
fore be interpreted alongside acceptance of valid computedfromthesearchivedtask-levelrecords;
solutions and successful rejection of negatives. the four-condition analysis below uses the Astra
| Appendix | I.1 | compares | input | handling, | | checking | tracebatch. | | | | | | |
| -------- | --- | -------- | ----- | --------- | --- | -------- | ----------- | --- | --- | --- | --- | --- | --- |
rules,andexecutiononidenticaltestcases.
| | | | | | | | Concentrationdefinition. | | | | Formodelm,letA | | |
| --- | --- | --- | --- | --- | --- | --- | ------------------------ | --- | --- | --- | -------------- | --- | --- |
m
G GroupedTest-CaseResults contain the tasks whose submitted verifier is all-
| | | | | | | | reject, | that is, | returns | a valid | FAIL | on | every test |
| --- | --- | --- | --- | --- | --- | --- | ------- | -------- | ------- | ------- | ---- | --- | ---------- |
Tables19and20givethesuppliedfinaldiagnostic caseforthattask. Taskswithanymissingorinvalid
| counts | for all | 19 Pi harness | | models. | Every | entry | | | | | | F+ | |
| ------- | ------- | ------------- | --------------------- | ------- | ----- | ----- | -------- | ------------ | --- | ---- | --- | ---- | ------ |
| | | | | | | | decision | are excluded | | from | A | . If | is the |
| | | | | | | | | | | | m | | m,t |
| hasP +F | +U | = N. | Everymodelhasall4,993 | | | | | | | | | | |
numberofpositivetestcasesrejectedontaskt,the
testcases,including1,566positivesand3,427neg- test-case-weightedcontributionis
atives. Routingfailuresandmissingsubmissions
| | | | | | | | | | | (cid:80) | F+ | | |
| ------------ | ------------ | ------------- | --- | --------------- | ----------- | ---------- | ------ | ----------- | --- | --------- | ---- | --- | -------- |
| remain | in these | denominators. | | These | | counts use | | | | | | | |
| | | | | | | | | Q | = | t∈Am | m,t | . | (9) |
| | | | | | | | | | m | (cid:80) | F+ | | |
| the same | verifier | selection | | and | judgment | rounds | | | | | | | |
| | | | | | | | | | | t∈D | m,t | | |
| as the final | leaderboard; | | the | grouped | proportions | | | | | | | | |
| | | | | | | | Figure | 5 and Table | | 22 report | 100Q | . | This ob- |
| weight | test cases, | whereas | | the leaderboard | | gives | | | | | | m | |
servedresponsepatternonthetestsuitedoesnot
tasksequalweight.
establishthattheverifierreturnsaconstantdecision
| The | heatmap | in Figure | | 4 separates | | reference | | | | | | | |
| --- | ------- | --------- | --- | ----------- | --- | --------- | --- | --- | --- | --- | --- | --- | --- |
onallpossibleinputs.
agreementfrommissingorinvalidoutputsineach
Anequal-task-weightversionreplaceseachF+
| category. | This | distinction | | prevents | a | low rate of | | | | | | | m,t |
| --------------------------------------------- | ---- | ----------- | --- | -------- | --- | ----------- | ------ | ---------------------------------- | --- | --- | --- | --- | --- |
| | | | | | | | withF+ | /|C+|inthenumeratoranddenominator. | | | | | |
| acceptednegativetestcasesfrombeinginterpreted | | | | | | | m,t | t | | | | | |
Itsrangeacrosstheninemodelsisalso87.9–96.0%;
| automaticallyassuccessfulrejection. | | | | | Forpositive | | | | | | | | |
| ----------------------------------- | --- | --- | --- | --- | ----------- | --- | --------------- | --- | ----- | ------ | -------- | --- | -------- |
| | | | | | | | for GPT-6-astra | | it is | 91.1%, | compared | | with the |
testcases,Table21insteadseparatesexplicitfalse
| | | | | | | | test-case-weighted92.7%. | | | | Thus,concentrationis | | |
| --- | --- | --- | --- | --- | --- | --- | ------------------------ | --- | --- | --- | -------------------- | --- | --- |
rejectionfromtheabsenceofavaliddecision.
| | | | | | | | not explained | | solely | by some | tasks | having | more |
| --------------- | ------ | -------- | -------- | ----------- | ------------ | ----------- | ------------------ | --- | ------ | ------- | ----- | ------ | ---- |
| For every | model, | | explicit | FAIL | accounts | for | | | | | | | |
| most unaccepted | | positive | | test cases, | | with rejec- | positivetestcases. | | | | | | |
| tion shares | from | 57.1% | to | 100%. | Valid-output | | | | | | | | |
| | | | | | | | Concentration | | in the | Astra | trace | batch. | We |
ratesacrossallgroupsrangefrom88.2%to99.9%. applythesamedefinitiontothefourconditionsof
These output categories locate the observed dis- theAstratracebatch. Thesecomprise888selected
| agreement | but | do not | identify | its | cause; | the anal- | | | | | | | |
| --------- | --- | ------ | -------- | --- | ------ | --------- | --- | --- | --- | --- | --- | --- | --- |
verifiersand19,972judgments,with222tasksand
yses of task-level outcomes and verifier code in 4,993 judgments per condition. Every judgment
Section4providethatadditionalevidence. returnsavalidbinarydecision. Table22showsthat
| | | | | | | | verifiers | returning | FAIL | on | every | test | case for a |
| --- | --- | --- | --- | --- | --- | --- | --------- | --------- | ---- | --- | ----- | ---- | ---------- |
21
(a)Positivetestcases
| Model | Gold | Cosmetic | Validalternative |
| ----------------- | ---------- | ----------- | ---------------- |
| GPT-6-astra | 154/67/1 | 573/263/4 | 285/215/4 |
| ClaudeOpus4.8 | 152/65/5 | 578/243/19 | 323/172/9 |
| KimiK3 | 156/49/17 | 590/186/64 | 323/119/62 |
| GPT-5.6-sol | 131/85/6 | 498/319/23 | 257/231/16 |
| GPT-5.6-terra | 131/89/2 | 500/332/8 | 240/259/5 |
| GLM-5.2 | 154/46/22 | 585/169/86 | 316/121/67 |
| DoubaoSeed2.1Pro | 129/89/4 | 495/332/13 | 283/215/6 |
| GPT-5.5 | 139/65/18 | 528/241/71 | 279/168/57 |
| DeepSeekV4Pro | 148/44/30 | 556/166/118 | 293/115/96 |
| DeepSeekV4Flash | 137/54/31 | 519/200/121 | 275/132/97 |
| Qwen3.7Plus | 131/83/8 | 506/307/27 | 284/211/9 |
| Gemini3.5Flash | 118/86/18 | 466/313/61 | 292/190/22 |
| KimiK2.6 | 120/70/32 | 465/254/121 | 247/170/87 |
| DoubaoSeed2.0Code | 104/107/11 | 408/394/38 | 239/249/16 |
| DoubaoSeed2.0Lite | 95/116/11 | 373/426/41 | 216/271/17 |
| DoubaoSeed2.0Pro | 63/159/0 | 248/592/0 | 142/362/0 |
| GPT-5.1 | 84/125/13 | 336/455/49 | 185/291/28 |
| Gemini2.5Pro | 73/114/35 | 290/424/126 | 163/274/67 |
| DoubaoSeed2.0Mini | 51/154/17 | 200/576/64 | 102/371/31 |
Table19: Decisionsonpositivetestcases. Groupedcountsonpositivetestcasesforall19configurationsusingthe
validPASS,validFAIL,andnovaliddecision.
Piharness. EachcellisP/F/U: Thecelldenominatoristheirsum,
N =P +F +U.
(b)Negativetestcases
| Model | Ordinaryerrors | Hack | Processviolation |
| ----------------- | -------------- | ---------- | ---------------- |
| GPT-6-astra | 144/2477/8 | 83/711/2 | 1/1/0 |
| ClaudeOpus4.8 | 390/2197/42 | 156/628/12 | 2/0/0 |
| KimiK3 | 311/2220/98 | 156/593/47 | 2/0/0 |
| GPT-5.6-sol | 35/2548/46 | 9/775/12 | 0/2/0 |
| GPT-5.6-terra | 64/2541/24 | 38/754/4 | 1/1/0 |
| GLM-5.2 | 294/2176/159 | 135/599/62 | 2/0/0 |
| DoubaoSeed2.1Pro | 265/2324/40 | 125/660/11 | 2/0/0 |
| GPT-5.5 | 226/2175/228 | 116/615/65 | 0/2/0 |
| DeepSeekV4Pro | 232/2191/206 | 117/585/94 | 2/0/0 |
| DeepSeekV4Flash | 191/2217/221 | 90/616/90 | 2/0/0 |
| Qwen3.7Plus | 378/2170/81 | 130/644/22 | 2/0/0 |
| Gemini3.5Flash | 329/2153/147 | 157/601/38 | 2/0/0 |
| KimiK2.6 | 119/2263/247 | 53/675/68 | 2/0/0 |
| DoubaoSeed2.0Code | 365/2207/57 | 132/651/13 | 1/1/0 |
| DoubaoSeed2.0Lite | 294/2234/101 | 111/660/25 | 0/1/1 |
| DoubaoSeed2.0Pro | 130/2497/2 | 79/715/2 | 0/2/0 |
| GPT-5.1 | 229/2296/104 | 112/655/29 | 1/1/0 |
| Gemini2.5Pro | 81/2270/278 | 46/668/82 | 1/1/0 |
| DoubaoSeed2.0Mini | 105/2296/228 | 44/695/57 | 0/2/0 |
Table20: Decisionsonnegativetestcases. Groupedcountsonnegativetestcasesforall19configurationsusing
thePiharness. EachcellisP/F/U: validPASS,validFAIL,andnovaliddecision. Thecelldenominatoristheir
sum,N =P +F +U. Ordinaryerrorscombinebase,partial,near-miss,andregressiontestcases.
22
(a) Reference agreement (%) (b) No valid output (%)
Model Gold Cosm. Alt. Error Hack Gold Cosm. Alt. Error Hack
GPT-6-astra 69.4 68.2 56.5 94.2 89.3 0.5 0.5 0.8 0.3 0.3
Claude Opus 4.8 68.5 68.8 64.1 83.6 78.9 2.3 2.3 1.8 1.6 1.5
Kimi K3 70.3 70.2 64.1 84.4 74.5 7.7 7.6 12.3 3.7 5.9
GPT-5.6-sol 59.0 59.3 51.0 96.9 97.4 2.7 2.7 3.2 1.7 1.5
GPT-5.6-terra 59.0 59.5 47.6 96.7 94.7 0.9 1.0 1.0 0.9 0.5
GLM-5.2 69.4 69.6 62.7 82.8 75.3 9.9 10.2 13.3 6.0 7.8
Doubao Seed 2.1 Pro 58.1 58.9 56.2 88.4 82.9 1.8 1.5 1.2 1.5 1.4
GPT-5.5 62.6 62.9 55.4 82.7 77.3 8.1 8.5 11.3 8.7 8.2
DeepSeek V4 Pro 66.7 66.2 58.1 83.3 73.5 13.5 14.0 19.0 7.8 11.8
DeepSeek V4 Flash 61.7 61.8 54.6 84.3 77.4 14.0 14.4 19.2 8.4 11.3
Qwen3.7 Plus 59.0 60.2 56.3 82.5 80.9 3.6 3.2 1.8 3.1 2.8
Gemini 3.5 Flash 53.2 55.5 57.9 81.9 75.5 8.1 7.3 4.4 5.6 4.8
Kimi K2.6 54.1 55.4 49.0 86.1 84.8 14.4 14.4 17.3 9.4 8.5
Doubao Seed 2.0 Code 46.8 48.6 47.4 83.9 81.8 5.0 4.5 3.2 2.2 1.6
Doubao Seed 2.0 Lite 42.8 44.4 42.9 85.0 82.9 5.0 4.9 3.4 3.8 3.1
Doubao Seed 2.0 Pro 28.4 29.5 28.2 95.0 89.8 0.0 0.0 0.0 0.1 0.3
GPT-5.1 37.8 40.0 36.7 87.3 82.3 5.9 5.8 5.6 4.0 3.6
Gemini 2.5 Pro 32.9 34.5 32.3 86.3 83.9 15.8 15.0 13.3 10.6 10.3
Doubao Seed 2.0 Mini 23.0 23.8 20.2 87.3 87.3 7.7 7.6 6.2 8.7 7.2
0 25 50 75 100 0 25 50 75 100
Figure4: Test-case-levelagreementandmissingorinvalidoutputs. Rowsfollowtheorderoftheconfigurations
usingthePiharnessinTable2. AgreementisP/N forthereferencesolution(Gold),cosmeticvariants(Cosm.),
andalternativeimplementations(Alt.),andF/N forordinaryerrors(Error)andhacks(Hack);missingorinvalid
outputisU/N. All19modelsareincluded. EachmodelusesthesamegroupdenominatorsfromTables19and20.
Thetwoprocess-violationtestcasesareomittedhereandretainedinTable20. Bothcolorscalesspan0–100%.
Thesetest-case-weighteddiagnosticsdifferfromthetask-averagedleaderboardmetrics.
taskaccountfor93.3–95.5%ofallfalserejections. whilefourmodelshavetheoppositerelationwithin
These results come directly from the Astra trace tool-call tasks. The global asymmetry and the
batch,independentlyofthearchivedrecordsabove. source-dependentdifferencescanthereforecoexist.
Thefourconditionsconcernonemodelanddonot
I CodeandGeneration-TraceCases
estimate variation across independent generation
seeds.
We first compare three models on the same test
The equal-task-weight concentration is 93.9%,
casestodistinguisherrorsininputhandling,check-
95.1%, 93.4%, and 94.4% for Pi harness/LOW,
ing rules, and execution. We then use the Astra
Piharness/HIGH,ClaudeCode/LOW,andClaude
tracebatchtoexaminesource-filerestrictions,tool-
Code/HIGH,respectively. Table23separatelyre-
callhandling,andincompletetesting. Supplemen-
portswhethereachverifieracceptsnone,some,or
tary DNA and booking cases use archived veri-
allpositivecases.
fier versions, while the AppWorld case concerns
compatibilitywiththecodeformsofpositivetest
Task-sourcevariation. AppendixFreportsthe
cases. The cases are selected to explain particu-
task-type rankings for all 19 Pi harness models.
lar observed failures, not to estimate mechanism
Within state tasks, GPT-6-astra’s positive accep-
prevalence. Hidden decisions are used only for
tancevarieswidelyacrosssourcebenchmarks. The
post-generationanalysis.
largewithin-typevariationcautionsagainstattribut-
ingallstate-taskfailurestoonemechanism. Within
thearchivednine-modelrecords,removinganysin-
gle source still leaves overall TPR below TNR,
23
| | | | | | | | | | | Rejection | | Valid |
| ----------------- | --- | --- | ---- | --- | ---- | ---- | ---- | --- | --- | --------- | ---------- | ----- |
| Model | | | | N | N+ | | P F | | U | share(%) | outputs(%) | |
| GPT-6-astra | | | 4993 | | 1566 | 1012 | 545 | | 9 | 98.4 | | 99.6 |
| ClaudeOpus4.8 | | | 4993 | | 1566 | 1053 | 480 | | 33 | 93.6 | | 98.3 |
| KimiK3 | | | 4993 | | 1566 | 1069 | 354 | 143 | | 71.2 | | 94.2 |
| GPT-5.6-sol | | | 4993 | | 1566 | 886 | 635 | | 45 | 93.4 | | 97.9 |
| GPT-5.6-terra | | | 4993 | | 1566 | 871 | 680 | | 15 | 97.8 | | 99.1 |
| GLM-5.2 | | | 4993 | | 1566 | 1055 | 336 | 175 | | 65.8 | | 92.1 |
| DoubaoSeed2.1Pro | | | 4993 | | 1566 | 907 | 636 | | 23 | 96.5 | | 98.5 |
| GPT-5.5 | | | 4993 | | 1566 | 946 | 474 | 146 | | 76.5 | | 91.2 |
| DeepSeekV4Pro | | | 4993 | | 1566 | 997 | 325 | 244 | | 57.1 | | 89.1 |
| DeepSeekV4Flash | | | 4993 | | 1566 | 931 | 386 | 249 | | 60.8 | | 88.8 |
| Qwen3.7Plus | | | 4993 | | 1566 | 921 | 601 | | 44 | 93.2 | | 97.1 |
| Gemini3.5Flash | | | 4993 | | 1566 | 876 | 589 | 101 | | 85.4 | | 94.3 |
| KimiK2.6 | | | 4993 | | 1566 | 832 | 494 | 240 | | 67.3 | | 88.9 |
| DoubaoSeed2.0Code | | | 4993 | | 1566 | 751 | 750 | | 65 | 92.0 | | 97.3 |
| DoubaoSeed2.0Lite | | | 4993 | | 1566 | 684 | 813 | | 69 | 92.2 | | 96.1 |
| DoubaoSeed2.0Pro | | | 4993 | | 1566 | 453 | 1113 | | 0 | 100.0 | | 99.9 |
| GPT-5.1 | | | 4993 | | 1566 | 605 | 871 | | 90 | 90.6 | | 95.5 |
| Gemini2.5Pro | | | 4993 | | 1566 | 526 | 812 | 228 | | 78.1 | | 88.2 |
| DoubaoSeed2.0Mini | | | 4993 | | 1566 | 353 | 1101 | 112 | | 90.8 | | 92.0 |
Table21: Falserejectionandvalidoutputs. Outcomesonpositivetestcasesandvalid-outputratesforall19
N+
configurationsusingthePiharness. N = 4,993isthetest-casetotalacrossallsixgroups; = 1,566isthe
positive-testdenominatorforeverymodel. Rejectionshareis100F/(F +U),thefractionofunacceptedpositives
receivinganexplicitFAIL.Thevalid-outputrateincludesallsixgroupsandpoolstheirtestcases.
| FAIL on every test case | | | | | | | Harness | Effort | | Tasks Pos.FAIL | | Share |
| ----------------------- | --- | --- | --- | --- | --- | --- | ------- | ------ | --- | -------------- | --- | ----- |
Other tasks
| | | | | | | | Piharness | LOW | | 75 | 551/587 | 93.9 |
| --------------- | --- | --- | --- | --- | ---- | --- | ---------- | ---- | --- | --- | ------- | ---- |
| Claude Opus 4.8 | | | | | 96.0 | | Piharness | HIGH | | 70 | 529/554 | 95.5 |
| | | | | | 95.5 | | ClaudeCode | LOW | | 68 | 515/552 | 93.3 |
Kimi K3
| | | | | | | | ClaudeCode | HIGH | | 70 | 535/564 | 94.9 |
| ------------ | ------------- | --- | --- | --- | ---- | --- | ------------ | -------------------------------------- | ------ | ---------- | ---------- | ----- |
| Qwen3.7 Plus | | | | | 93.8 | | | | | | | |
| | GPT-5.1 | | | | 93.5 | | Table22: | False-rejectionconcentrationintheAstra | | | | |
| | | | | | | | trace batch. | Tasks | counts | all-reject | verifiers, | which |
| | GPT-5.6-terra | | | | 92.9 | | | | | | | |
returnFAILoneverytestcaseoftheirtask,outof222.
| | GPT-6-astra | | | | 92.7 | | | | | | | |
| --- | ----------- | --- | --- | --- | ---- | --- | --------- | ------- | --- | ------------- | --------- | ---- |
| | | | | | | | Pos. FAIL | reports | the | positive FAIL | decisions | from |
Gemini 2.5 Pro 92.5 theseverifiersdividedbyallpositiverejectionsinthe
| | | | | | 91.6 | | condition;Shareexpressesthisratioasapercentage. | | | | | |
| --- | --- | --- | --- | --- | ---- | --- | ----------------------------------------------- | --- | --- | --- | --- | --- |
GPT-5.5
| | GPT-5.6-sol | | | | 87.9 | | | | | | | |
| --- | ----------- | --- | --- | --- | ---- | --- | --------- | --- | ------ | ---- | ---- | --- |
| | | | | | | | Harness | | Effort | None | Some | All |
| | | 0 | 25 | 50 | 75 | 100 | | | | | | |
| | | | | | | | Piharness | | LOW | 76 | 23 | 123 |
Share of false rejections (%)
| | | | | | | | Piharness | | HIGH | 71 | 16 | 135 |
| -------- | ------------------------------ | --- | --- | --- | ------- | --- | ---------- | --- | ---- | --- | --- | --- |
| | | | | | | | ClaudeCode | | LOW | 69 | 22 | 131 |
| Figure5: | Concentrationoffalserejection. | | | | Foreach | | | | | | | |
| | | | | | | | ClaudeCode | | HIGH | 71 | 19 | 132 |
oftheninePi-harnessconfigurations,theredsegment
givesthepercentageofallpositiveFAILdecisionscon-
| | | | | | | | Table23: | Positiveacceptancebytask. | | | Countsoutof | |
| -------- | ------------ | ---------- | --- | ----------- | ---- | --- | -------- | ------------------------- | --- | --- | ----------- | --- |
| tributed | byall-reject | verifiers, | | whichreturn | FAIL | on | | | | | | |
everytestcaseoftheirtask. Acrossmodelsthisshare 222 tasks per Astra condition. None, Some, and All
| | | | | | | | refer only | to positive | test | cases. Each | condition | has |
| -------------------------------- | --- | --- | --- | ------------------ | --- | --- | ------------------- | ----------- | ---- | -------------- | ------------ | --- |
| staysbetween87.9%and96.0%. | | | | Graysegmentsrepre- | | | | | | | | |
| | | | | | | | one zero-acceptance | | task | whose verifier | nevertheless | |
| senttheremainingfalserejections. | | | | AppendixHspeci- | | | | | | | | |
fiesthearchivedanalysisrecords. acceptsanegativecase,soNonediffersfromthetask
countinTable22.
24
I.1 ComparingModelsontheSameTest and generation traces together. Table 4 summa-
| Cases | | | | | | rizescasesfromtheAstratracebatchforwhichthe | | | | | | |
| ----- | --- | --- | --- | --- | --- | ------------------------------------------- | --- | --- | --- | --- | --- | --- |
publicrequirementssupportattributingrejectionto
Table24comparesGPT-5.6-sol,ClaudeOpus4.8,
| | | | | | | anunjustifiedcheck. | | Itslowerblockcontainsthe | | | | |
| -------- | --- | --------- | --- | ----------- | ------ | ------------------- | --- | ------------------------ | --- | --- | --- | --- |
| and Kimi | K3, | all using | the | Pi harness, | on one | | | | | | | |
verifierswhosepositiveself-testsneverreturneda
| task of | each type. | Within | each | task, | the models | | | | | | | |
| ------- | ---------- | ------ | ---- | ----- | ---------- | --- | --- | --- | --- | --- | --- | --- |
completePASSduringgeneration;itsupperblock
| areevaluatedonidenticaltest-casecontents. | | | | | The | | | | | | | |
| ----------------------------------------- | --- | --- | --- | --- | --- | -------- | --------- | ---- | ---- | --------- | ------ | --- |
| | | | | | | contains | the rest. | This | is a | purposive | review | of |
comparisonsusethearchivedrecordsdescribedin
Section3.1;theyillustratedifferentfailuremecha- failures, not a random sample or an estimate of
| | | | | | | mechanism | prevalence. | | The | counts | refer | to task– |
| --- | --- | --- | --- | --- | --- | --------- | ----------- | --- | --- | ------ | ----- | -------- |
nismsratherthanexplainthetask-typerankingsas
configurationverifierinstancesandtheirrecorded
awhole.
| | | | | | | decisions; | repeated | judgments | | across | configura- | |
| --- | --- | --- | --- | --- | --- | ---------- | -------- | --------- | --- | ------ | ---------- | --- |
Inputcompatibilityandomittedchecksproduce tionsandsurfacevariantsarenotindependenterror
different errors. On ACE preference 17, GPT- mechanisms. Sourceinspectionidentifiessufficient
5.6-solattemptstoparseasuppliedprofilestring
| | | | | | | rejection | conditions, | without | | intervention | | experi- |
| --- | --- | --- | --- | --- | --- | --------- | ----------- | ------- | --- | ------------ | --- | ------- |
asstructureddataandrejectsallsixpositiveswhen
mentsestablishingthateachconditionistheunique
| parsingfails. | ClaudeOpus4.8acceptsallsixposi- | | | | | causeoffailure. | | | | | | |
| ------------- | ------------------------------- | --- | --- | --- | --- | --------------- | --- | --- | --- | --- | --- | --- |
tivesbutcheckstherequestedlocationchangewith-
outadequatelycheckingpreservationofthename I.3 Source-FileRestrictionsandSelf-Test
Coverage
| and email, | accepting | | 12 of | 18 negatives. | Kimi | | | | | | | |
| ---------- | --------- | --- | ----- | ------------- | ---- | --- | --- | --- | --- | --- | --- | --- |
K3checksthesepreservedfieldsandacceptsonly Django-11734. ThePiharness/LOWverifierfor
twonegatives;bothcontainthesamerepeated-call
| | | | | | | vgb-t-d8bd9183d1119aa2 | | | | rejects | | patches |
| --- | --- | --- | --- | --- | --- | ---------------------- | --- | --- | --- | ------- | --- | ------- |
payload, recorded in two construction categories. modifying any file named __init__.py. The
The comparison separates failure to interpret the task concerns incorrect model references in
suppliedinputfromfailuretoenforceallrequired OuterRef; its patch specification does not
conditions.
| | | | | | | prohibit | modifying | the | normal | | source | mod- |
| --- | --- | --- | --- | --- | --- | -------- | --------- | --- | ------ | --- | ------ | ---- |
uledjango/db/models/fields/__init_
| Early rejection | | can | prevent | state verification. | | | | | | | | |
| --------------- | --- | --- | ------- | ------------------- | --- | --- | --- | --- | --- | --- | --- | --- |
_.py. Allfourpositivetestcasesmodifythismod-
Onretail/34,GPT-5.6-solrequiresatoolresultto
| | | | | | | ule and | receive FAIL | | before | environment | | execu- |
| -------- | ---------- | ---- | ------ | -------- | -------- | ------- | ------------ | --- | ------ | ----------- | --- | ------ |
| parse as | structured | data | before | reaching | its han- | | | | | | | |
tion. Thesecasescompriseareferencerepairand
| dlingofabareuser-IDstring. | | | | Thepositivecases | | | | | | | | |
| -------------------------- | --- | --- | --- | ---------------- | --- | --- | --- | --- | --- | --- | --- | --- |
surfacevariants,notfourindependentrepairstrate-
providesuchastring,soallninereceiveFAILbe-
gies.
| foreanyenvironmentreplay. | | | | ClaudeOpus4.8and | | | | | | | | |
| ------------------------- | --- | --- | --- | ---------------- | --- | --- | ---------------- | --- | --------- | ------- | -------- | --- |
| | | | | | | The | agent’s positive | | self-test | instead | modifies | |
KimiK3handlethisinputandcorrectlyjudgeall
| | | | | | | django/db/models/sql/query.py. | | | | | | After |
| -------------- | --- | ----- | ---------- | ---- | -------- | ------------------------------ | --- | --- | --- | --- | --- | ----- |
| nine positives | and | eight | negatives. | This | contrast | | | | | | | |
fixingsubprocesscompatibility,patchformatting,
| identifies | an input-processing | | | failure, | rather than | | | | | | | |
| ---------- | ------------------- | --- | --- | -------- | ----------- | --- | --- | --- | --- | --- | --- | --- |
andenvironmentresetissues,thetracerecordspos-
anincorrectjudgmentoftheresultingorderstate.
| | | | | | | itive PASS | and negative | | FAIL | outcomes. | | Those |
| --- | --- | --- | --- | --- | --- | ---------- | ------------ | --- | ---- | --------- | --- | ----- |
Timeoutsdifferfromincorrectdecisions. On repairsmakethechosentestexecutablebutdonot
Django-11848,GPT-5.6-solandClaudeOpus4.8 challenge the file restriction. The relevant events
correctly judge all 15 test cases. Kimi K3 cor- are at zero-based native trace records 16, 26, 28,
rectlyjudgesnine,whileonepositiveandfiveneg- and 29, with the successful output at record 29.
ativestimeout. Therecordsestablishincomplete TheotherthreeAstraconfigurationsacceptallfour
execution, butdonotestablishthecauseofthese positives and reject all ten negatives on this task;
timeouts. Together,thecasesshowwhyscoredif- thisisanexistingconfigurationcomparison,notan
ferencesrequireseparateexaminationofinputhan- interventiononthefaultyverifier.
dling,task-specificchecks,andcompletionofexe-
| | | | | | | Matplotlib-14623. | | For | | | | |
| --- | --- | --- | --- | --- | --- | ----------------- | --- | --- | --- | --- | --- | --- |
cution.
| | | | | | | vgb-t-25932ca825bd7ab6, | | | | | both | Claude |
| --- | --- | --- | --- | --- | --- | ----------------------- | --- | --- | --- | --- | ---- | ------ |
I.2 AttributingRejectionstoUnjustified Code verifiers reject legitimate modifications to
lib/mpl_toolkits/mplot3d/axes3d.
Checks
| | | | | | | py. The | LOW | verifier’s | | allowed | directories | |
| --- | --- | --- | --- | --- | --- | ------- | --- | ---------- | --- | ------- | ----------- | --- |
Weinspectthepublictaskrequirements,generated
| | | | | | | omit lib/mpl_toolkits/; | | | | the | HIGH | ver- |
| ------- | -------- | ---- | ------ | -------- | ---------- | ----------------------- | ------ | ----- | ----- | --- | ---- | ------- |
| checks, | positive | test | cases, | recorded | judgments, | | | | | | | |
| | | | | | | ifier treats | Python | files | under | lib | but | outside |
25
| Model | | PositivePASS | | NegativeFAIL | | Novaliddecision | |
| ------------- | ----------------------------------- | ------------ | --- | ------------ | ----- | --------------- | --- |
| Tool-call: | updateauserprofile(ACEpreference17) | | | | | | |
| GPT-5.6-sol | | | 0/6 | | 18/18 | | 0 |
| ClaudeOpus4.8 | | | 6/6 | | 6/18 | | 0 |
| KimiK3 | | | 6/6 | | 16/18 | | 0 |
State: changeanorderaddress(τ2retail/34)
| GPT-5.6-sol | | | 0/9 | | 8/8 | | 0 |
| ------------- | --- | --- | --- | --- | --- | --- | --- |
| ClaudeOpus4.8 | | | 9/9 | | 8/8 | | 0 |
| KimiK3 | | | 9/9 | | 8/8 | | 0 |
Code: parseanHTTPdate(Django-11848)
| GPT-5.6-sol | | | 4/4 | | 11/11 | | 0 |
| ------------- | --- | --- | --- | --- | ----- | --- | --- |
| ClaudeOpus4.8 | | | 4/4 | | 11/11 | | 0 |
| KimiK3 | | | 3/4 | | 6/11 | | 6 |
Table24: Modelcomparisonsonidenticaltestcases. AllthreemodelsusethePiharness. Fractionscountcorrect
decisionsoverallcasesofthecorrespondingclass. ThesixmissingdecisionsforKimiK3onDjango-11848are
timeoutsononepositiveandfivenegatives,whichremaininthedenominators.
lib/matplotlib/ as protected. Each rejects only tool get_user_dispute_history_
four positive cases, yielding eight decisions 7291. Their checks reject legitimate auxiliary
on four distinct cases. LOW rejects locally readsasunsupportedoperations,producing18posi-
tiveFAILdecisions.
withoutenvironmentexecution;HIGHinvokesthe Theirself-testsomitthesame
environmentandrejectsattheprotected-filecheck read. These five verifiers attempted positive self-
before the relevant functional test. Successful tests,butnoneobtainedacompletepositivePASS;
generation-time tests modify allowed locations they therefore do not exemplify successful self-
| such as | axes/_base.py | or | ticker.py. | testing. | | | |
| -------- | ----------------- | --------------- | ---------- | -------- | --- | --- | --- |
| Changing | the repair within | those locations | does | | | | |
I.5 IncompleteCategoryVocabulary
| not test whether | the directory | restriction | itself is | | | | |
| ---------------- | ------------- | ----------- | --------- | ---------------------------------------- | --- | --- | --- |
| justified. | | | | ThePiharness/HIGHverifierforretail/33and | | | |
bothHIGHverifiersforretail/34implement
I.4 Tool-CallRestrictions
theoffice-itemcategorywithvocabulariesthatomit
The Pi harness/LOW verifiers for Headphones. Their positive cases contain this
banking_knowledge/task_037, item, causing the selected order to fail the gener-
task_040, and task_041 reject authorized atedcategory/countcondition. Acrossthesethree
FAIL
user-sidetoolcallsallowedbythepublictoolproto- verifiers, 27 positive decisions correspond
col. Theyrespectivelyrequiretheassistantrole,ex- to18distinctcases. Thetwotasksshareanorder
plicitlyrejecttheuser-callwrapper,oromititshan- background,sotheyarenotindependentsemantic
| dlingbranch. | Theirreleasedtaskidentifiersendin | | | replications. | | | |
| ------------ | --------------------------------- | --- | --- | ------------- | --- | --- | --- |
12aafca53ad113b0, 2de811fb047dcfa6, Forretail/34,thehiddenpositivecasescon-
and 36e8a311f52b2bb9. Each rejects tain Mechanical Keyboard, Headphones, and Of-
nine positives, for 27 decisions. For task fice Chair as the three office items. The Claude
041, the verifier permits authorization through Code/HIGH verifier recognizes only Mechanical
give_discoverable_user_tool, but KeyboardandOfficeChair,soitscountfallsbelow
omits call_discoverable_user_tool; the required three items. All nine positive cases
the positive cases use that wrapper to query card receive FAIL without public environment execu-
information. The intended-positive self-tests use tion. Theomittedcategorymemberissufficientfor
assistant calls and do not exercise this allowed rejection,althoughsomecasesmayalsoencounter
| variation. | | | | anearliercheck. | | | |
| ---------- | --- | --- | --- | --------------- | --- | --- | --- |
The Claude Code/LOW verifier for task 044, Across these three retail and five banking ver-
shown in Figure 2, and Pi harness/LOW ver- ifiers, all 72 positive judgments have zero envi-
ifier for task 045 omit the permitted read- ronment executions. All eight generation traces
26
acknowledgeincompletepositivevalidation. These reference acceptance rule and should not be gen-
eightverifiersformthelowerblockofTable4. eralized into an assumption that every redundant
operationisinvalid.
| Otherpermittedrepresentations. | | | | Theremain- | | | | | | | | |
| ------------------------------ | --- | --- | --- | ---------- | --- | --- | --- | --- | --- | --- | --- | --- |
Thevisibleself-testsusefullyspecifiedcalls,an
| ing cases | in Table | 4 concern | an | unjustified | ISO- | | | | | | | |
| --------- | -------- | --------- | --- | ----------- | ---- | --- | --- | --- | --- | --- | --- | --- |
incompletesubset,andaraw-textrepresentationof
onlycurrencyrestriction,numeric-onlydeviceiden-
| | | | | | | thefullyspecifiedcalls. | | | Theydonottestomission | | | |
| -------- | -------------- | -------- | --- | ------ | -------- | ----------------------- | --- | --- | ---------------------- | --- | --- | --- |
| tifiers, | and incomplete | decoding | | of TeX | spacing. | | | | | | | |
| | | | | | | ofthedefaultargument. | | | Thecasethereforelinksa | | | |
Theirtestsusetherepresentationsalreadyaccepted
specificcomparisonerrortoabehaviorspecifiedin
bythegeneratedchecks,leavingtheomittedalter-
thetooldefinitionsbutabsentfromtheself-tests.
| nativesuntested. | Aswiththecodecases,apositive | | | | | | | | | | | |
| ---------------- | ---------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
outcome observed during generation does not es- I.7 Self-TestsThatShareaParsing
tablishthatthefinalsourceversioncompletedfull
Assumption
| validation: | thesourceversionateveryself-testwas | | | | | | | | | | | |
| ----------- | ----------------------------------- | --- | --- | --- | --- | ---------------------- | --- | --- | ---- | ---- | ---- | ---- |
| | | | | | | The GPT-5.6-sol/Claude | | | Code | case | uses | task |
notreconstructed.
| | | | | | | vgb-t-0480046e6a2a021f, | | | | | an ACEBench | |
| ----------------------- | --- | --- | ----- | --- | ---------- | ----------------------- | --- | ---------------- | ----- | ------------ | ----------- | ------- |
| | | | | | | booking-cancellation | | | task. | Its recorded | | genera- |
| Compatibility-qualified | | | case. | The | four Astra | | | | | | | |
| | | | | | | tion includes | | a test described | | as a | function | call, |
verifiersforDjango-12155requiretheoldinternal
helpertrim_docstringtoremainavailable,al- constructed by placing a JSON dictionary inside
though the inspected repair replaces its handling the function’s parentheses. The test passes. The
| | | | | | | reference | solution | in | the test | suite | instead | ex- |
| --------------------- | --- | --- | ----------------- | --- | --- | --------- | -------- | --- | -------- | ----- | ------- | --- |
| withinspect.cleandoc. | | | Thischeckaccounts | | | | | | | | | |
for 16 positive rejections on four distinct cases, pressestheargumentsasPythonkeywords,includ-
butinterpretingitasunnecessarydependsoncom- ingbookingId=’R123’.
| | | | | | | The | generated | parser | extracts | | the text | inside |
| ------------------- | ------------ | ------------------------ | ----- | --------- | ------ | --------------- | --------- | ------------ | -------- | --- | -------- | ------- |
| patibility | requirements | not | fully | specified | in the | | | | | | | |
| | | | | | | the parentheses | | and attempts | | to | decode | it with |
| availablematerials. | | Wethereforeexcludeitfrom | | | | | | | | | | |
the12-task,16-verifierattributioncountinTable4. json.loads or ast.literal_eval. A
keyword-argumentlistisneitheraJSONvaluenor
I.6 IncorrectHandlingofaSpecifiedDefault a Python literal expression, so neither operation
TheQwen3.7Plus/PiharnesscaseinAppendixFig- parses that text. The verifier maps this failure to
ure7usestaskvgb-t-d78e768308367722,a FAIL;thedesignatedreferencesolutionreceives
| | | | | | | a recorded | rejection | with | confidence | | 0.99. | The |
| --- | --- | --- | --- | --- | --- | ---------- | --------- | ---- | ---------- | --- | ----- | --- |
DNA-analysistaskfromBFCL.Theactualgener-
ationpromptstatesthatmutation_typeisop- self-testpassesbecauseitusespreciselytherepre-
tionaland defaultstoinsertion; only thetwo sentationacceptedbytheparser.
sequence arguments are required. The generated The verifier input format provides raw and
checkercomparesexpectedargumentvalueswith parsedcallfields,butthegenerationpromptgives
valuesobtainedusing noconcretegrammarfortherawfield. Thiscase
supportsinputincompatibilityandinadequateself-
| actual_args.get(key, | | | "") | | | | | | | | | |
| -------------------- | --- | --- | --- | --- | --- | -------------- | --- | ----------- | --- | ----- | -------- | ----- |
| | | | | | | test coverage, | | rather than | a | claim | that the | model |
Consequently, an omitted optional parame- ignored an explicit syntax requirement. It comes
ter becomes the empty string rather than from the Claude Code configuration; Appendix
Figure9showsarelatedraw-inputparsingfailure
| its defined | default. | The | positive | | test case | | | | | | | |
| ---------------------- | --------------- | ---- | -------- | -------- | --------- | ----------- | --- | --- | --- | --- | --- | --- |
| vgb-c-e6ede484b58a7c02 | | | | | | underCodex. | | | | | | |
| | | | | omits | this pa- | | | | | | | |
| rameter | in an insertion | call | and | receives | FAIL, | | | | | | | |
I.8 CompatibilitywithCodeFormsofPositive
| while the | designated | fully | specified | solution | re- | | | | | | | |
| --------- | ---------- | ----- | --------- | -------- | --- | --- | --- | --- | --- | --- | --- | --- |
TestCases
ceivesPASS.
The same verifier searches for the presence of The GPT-6-astra/Pi harness LOW case uses
29caf6f_1,
sixexpectedcallsandrejectsextracallsonlywhen AppWorld task released as
| | | | | | | vgb-t-08167d1b4c9984dd. | | | | | The | task |
| -------------------------- | --- | --- | ---------------- | --- | --- | ----------------------- | --- | --- | --- | --- | --- | ---- |
| theirtoolnameisunexpected. | | | Negativetestcase | | | | | | | | | |
vgb-c-9c2f334fedb91aae requires sending a movie recommendation.
| | | | | preserves | those | | | | | | | |
| --- | --- | --- | --- | --------- | ----- | --- | --- | --- | --- | --- | --- | --- |
callsandaddsanothercallusingthesametool;the The candidate solution in the selected positive
| | | | | | | test case, | | vgb-c-2d2f17e345dbfd3b, | | | | |
| --- | --- | --- | --- | --- | --- | ---------- | --- | ----------------------- | --- | --- | --- | --- |
generatedverifierreturnsPASS.Thedefaultomis-
| | | | | | | imports | ApiCollection | | | from | appworld. | |
| ----------------------------------------- | --- | --- | --- | --- | --- | ------- | ------------- | --- | --- | ---- | --------- | --- |
| sionisexplicitlyjustifiedbythetoolschema. | | | | | The | | | | | | | |
treatment of extra calls is assessed relative to the collections.apis and Requester from
27
appworld.requester, then defines a
solutionfunction.
The verifier’s safe function walks the
candidate’s Python AST. Its check at source
line 9 returns false for any ast.Import
or ast.ImportFrom node, together
with ast.ClassDef, ast.With, and
ast.AsyncWith. Thecallerreturnsanegative
decision when safe(code) is false (line 23),
before reaching vgb_env.run_public (line
28). This positive test case therefore encounters
a rejection condition before the verifier executes
the proposed solution or examines its resulting
state. Thisidentifiesaspecificincompatibilitywith
a code form used by a positive test case; it does
notestablishthateveryAppWorldfailurehasthis
cause. Thearchivedpromptdoesnotfullyspecify
the solution-code entry convention, so this case
is not counted among the unjustified checks in
Table4.
Thegenerationtracealsorecordsasuccessfulre-
pairthatleavesthisrestrictionuntested. Themodel
constructsthreeself-tests: acorrectrecommenda-
tion,anincorrectrecommendation,andanempty
input. Their initial decisions are PASS, PASS,
and FAIL. The model identifies persistent state
betweeninvocations,insertsaresetbeforeexecu-
tion, and obtains PASS, FAIL, and FAIL on the
repeated tests. The positive self-test uses direct
APIcallswithouteitherimport,sothetestpasses
without exercising the condition that rejects the
referencesolutioninthetestsuite.
Native trace records 21–22 contain the initial
tests and results, record 23 adds the reset, and
records 25–26 contain the repeated tests and re-
sults.
I.9 ComparativeGenerationTrajectories
Thecomparisonsbelowexpandthethreefindings
inSection4. Figure6contrastsanoverbroadfile
restriction with checks that preserve legal source
changes. Figure 7 compares successful self-tests
with a valid variation they omit. Figure 8 shows
how an incomplete category vocabulary rejects
validsolutionsbymiscountingtheitemsspecified
by the task. Figures 9 and 10 then illustrate the
native-harnessdifferencesinTable3: inputrestric-
tionsandaccesstoausableexpectedstate.
28
Case1. Protectingtestswhileallowinglegalsourcechanges
| GPT-6-astra | | PiharnessandClaudeCode,LOW | | | Django-11734 | |
| ----------- | ---------------------------- | --- | -------------- | --- |
Taskrequirement
FixincorrectmodelreferencesinqueriesusingOuterRefwithexclude()ornegation. Thecandidateisa
source-codepatch;normalDjangosourcefilesmaybemodified. Thecomparisonaskswhetherprotectingtests
alsorequiresexcludinglegalsourcechanges.
Piharness: restrictfilenames ClaudeCode: protecttestsandcheckbehavior
Theagentaddsarejectionconditioncontaining: Theagentprotectsthetestinfrastructurewhile
| | | | allowingordinaryDjangosourcechanges. | Itchecks |
| --- | --- | --- | ------------------------------------ | -------- |
path.endswith((
thattestfilesremainunchanged,appliessource
’conftest.py’,’sitecustomize.py’,
changes,andrunsbothatask-specificOuterRef
’__init__.py’))
| | | | checkandexistingqueryandexpressiontests. | It |
| --- | --- | --- | ---------------------------------------- | --- |
Thisalsoexcludesordinarysourcemodulesnamed doesnotbanallfilesnamed__init__.py.
__init__.py,beforetheirbehaviorcanbe
checked.
Self-testing: executionrepairssucceed Self-testing: functionalcheckssucceed
Thepositiveself-testmodifiesquery.py. The Theagentobservessuccessfulexecutionofits
agentfixessubprocesscompatibility,malformed behaviorcheckandtheexistingregressiontests. Its
patchheaders,andmissingenvironmentreset. complete-verifierself-testsreturn:
Subsequentcomplete-verifiertestsreturn: Twointendedpositivepatches PASS
| | | PASS | | FAIL |
| ------------------- | --- | ---- | ----------------------------------------- | ---- |
| Chosenpositivepatch | | | Incorrectpatch | |
| Incorrectpatch | | FAIL | Emptyormalformedpatch | FAIL |
| Emptypatch | | FAIL | Bothpositiveself-testsalsomodifyquery.py; | |
Thesuccessfulpatchdoesnotexercisethefile-name broaderrepair-locationcoveragedoesnotexplain
| restriction. | | | thiscontrast. | |
| ----------------- | ------------------------- | --- | ------------- | --- |
| Hiddenevaluation: | thesamelegalsourcechanges | | | |
Allfourpositivetestcasesmodifydjango/db/models/fields/__init__.py. ThePiverifierrejects
thembeforeenvironmentexecution;theClaudeCodeverifieracceptsthemandstillrejectseverynegativecase.
Generatedverifier Positivecasesaccepted Negativecasesrejected
| Piharness | | | 0/4 | 10/10 |
| ---------- | --- | --- | --- | ----- |
| ClaudeCode | | | 4/4 | 10/10 |
Whatthecomparisonshows
Bothagentsobtainsuccessfulself-testsatthesamerepairlocation,butonlyoneverifierexcludesanotherlegal
location. ThecontrastconnectsthefindingsinSections4.1and4.2: anunnecessaryrestrictioncanrejectseveral
correctsolutionswhileremainingunchallengedbysuccessfulself-tests.
Figure6: Acceptinglegalsourcechangesneednotweakenrejectionoftestedviolations. Thefourpositives
compriseareferencerepairandsurfacevariants. Thecontrastconcernsthegeneratedchecks,notademonstrated
improvementinself-testcoverage.
Theseselectedcasesexplainobservedfailuresandsuccesses; theydonotestimatehowofteneachmechanism
occursorisolatetheeffectofaharnesscomponent. Eachcomparisonconnectsthetaskrequirements,generated
checks,self-tests,andhiddenjudgments. Pairedharnesscomparisonsusethesamemodel,task,andtestcases.
Self-testswerechosenduringgeneration;hiddentestcaseswereevaluatedafterward. Codeexcerptsomitunrelated
checks,andprosesummarizestheobservedgenerationsteps.
29
Case2. Explicitdefaultsversuspermittedomission
| Qwen3.7Plus | | Piharness | | DNAanalysis | | | | |
| ----------- | ----------- | ------------- | --- | --- | --- | --- |
Taskrequirement
Thetaskrequestsinsertion,deletion,andsubstitutionanalysesfortwoDNAsequences. Thetooldefinition
makesmutation_typeoptional,withdefaultinsertion;sequenceandreference_sequence
arerequired. Aninsertioncallthereforehasthesamemeaningwiththedefaultsuppliedoromitted.
| Generatedcheck: | missingargumentsbecomeemptystrings | | | | | |
| --------------- | ---------------------------------- | --- | --- | --- | --- | --- |
| for key, | val in expected_args.items(): | | | | | |
| actual_val | = actual_args.get(key, | | "") | | | |
Theverifiercomparesthisvaluewiththeexpectedargument. Foranomittedmutation_type,itcompares
""with"insertion"insteadofapplyingthespecifieddefault.
Generation-timepositive: explicitdefault Hiddenpositive: omitteddefault
Everycallinthepositiveself-testsupplies Avalidinsertioncallomitstheoptionalargument:
| mutation_type. | Foraninsertioncall,the | | | | | |
| -------------- | ---------------------- | --- | --- | --- | --- | --- |
"analyze_dna_sequence": {
relevantargumentis:
"reference_sequence": "AGCTTAGCTA",
| "mutation_type": | "insertion" | | | "sequence": | "AGCTTAGCTA" | |
| ---------------- | ----------- | --- | --- | ----------- | ------------ | --- |
}
Thecomparisonreceivestheexpectedvalue.
Thegeneratedcomparisoninsteadreceivesanempty
string.
Self-tests: allexpectedoutcomesobserved Hiddenevaluation: legalomissionrejected
| Fullyspecifiedcalls | | | PASS | Positivecasesaccepted | | 5/9 |
| -------------------- | --- | --- | ---- | --------------------- | --- | --- |
| Missingrequiredcalls | | | FAIL | Positivecasesrejected | | 4/9 |
PASS
Fullyspecifiedrawcalls Thefourrejectionscontaintwodistinctpayloads,
Theagenttestsanincompletesolutionandanother omittingthedefaultinthefirstorsecondinsertion
inputrepresentation,butbothpositivetestsexplicitly call. Theyrepeatedlytriggerthesamecomparison
| supplythedefault. | | | | error. | | |
| ----------------- | --- | --- | --- | ------ | --- | --- |
Whatthecomparisonshows
Thetestvariationstheagentselectedleavetheincorrectrequirementunchanged: everysuccessfulinsertion
exampleexplicitlysuppliesitsdefault. Thehiddenpositivesvaryexactlythatpermittedproperty. Thisillustrates
Section4.2: testsmustchallengewhetheraconditionisrequired,notonlydemonstrateacceptanceofexamples
thatsatisfyit.
Figure7:Successfulself-testsleavethespecifieddefaultuntested.TheQwen3.7Plusverifierpassesitsgeneration-
timetestsyetrejectsavariationexplicitlypermittedbythetooldefinition.
30
Case3. Anincompletecategoryvocabularyrejectsvalidsolutions
| GPT-6-astra | | ClaudeCode,HIGH | | retail/34 | | |
| ----------- | ----------------- | ----------- | --- | --- |
Taskrequirement
Removethreeofficeitemswhileretaininghikingitems. Ifpartialcancellationisunavailable,changetheorderto
theuser’sNYCaddressinstead. Thegeneratedverifierusesitemnamestoidentifytherelevantorder.
| Generatedcheck: | akeywordlistdeterminestheitemcategory | | | |
| --------------- | ------------------------------------- | --- | --- | --- |
Theverifierrequiresthreeofficeitems. Itscategoryvocabularyincludeslaptop,computer,monitor,
keyboard,chair,andheadset,butomitsheadphones.
| Hiddenpositivecases: | taskcategoryversusgeneratedcheck | | | |
| -------------------- | -------------------------------- | ------------------- | --- | -------------------- |
| Item | | Officeiteminthetask | | Matchedbytheverifier |
| MechanicalKeyboard | | | yes | yes |
| Headphones | | | yes | no |
| OfficeChair | | | yes | yes |
| Office-itemcount | | | 3 | 2 |
CyclingHelmetandHikingBootsarealsopresent. BecausethekeywordlistomitsHeadphones,thegenerated
checkfailstorecognizeallthreeofficeitemsandrejectstheorder.
Recordedjudgments
AllninepositivetestcasesreceiveFAILbeforepublicenvironmentexecution. Theomittedcategorymemberis
sufficientforstaticrejection,althoughsomecasesmayalsoencounteranearliercheck.
Whatthecomparisonshows
Thegeneratedvocabularyexcludesanitemthatbelongstothetask’soffice-itemcategory. Theresultingcount
failstheverifier’sowncondition,makingavalidorderappearinconsistentwiththetask. Thisillustratesthe
unjustifiedchecksanalyzedinSection4.1.
Figure 8: An omitted category member produces an incorrect item count. The task identifies three office
items,butthegeneratedvocabularyrecognizesonlytwo. AllninepositivetestcasesreceiveFAILwithoutpublic
environmentexecution.
31
Case4. Successfulself-testsmissanincompatibleparsingcondition
GPT-5.6-sol | CodexandPiharness | Bookingcancellation
Commontaskandverifierinput
Cancelfivereservationswiththespecifiedtimesandpolicies. Theinputprovidesparsed,containing
structuredcallsandarguments,andraw,containingthecallsasemitted. Thesupplieddescriptiondoesnot
requirerawtobeJSON.
Codex: requireanadditionalJSONparse Piharness: usethestructuredcalls
Aftercheckingthestructuredcalls,theverifier Theverifierusesparsedwhenitcanreadthat
decodestherawfieldusing: field,andfallsbacktorawotherwise. Itchecksthe
requestedbookingsandcancellationarguments
raw_value = json.loads(raw)
withoutmakingaseparaterawJSONparsea
Itrequiresthisresulttosatisfythetaskchecksand prerequisite. Therawrepresentationthereforedoes
matchthestructuredcalls. Arawstringthatisnot notoverridevalidstructuredcalls.
JSONcausesrejectionevenwhenthesupplied
structuredcallsarevalid.
Codexself-tests: observedsuccess Piharnessself-test: executionblocked
TheagentfixesanunavailablePythoncommandand Theagentalsoconstructsapositivecontainingonly
runsthecompleteverifier. parsedandattemptsacomplete-verifiertest. The
Correctcancellations,reordered PASS shellfailsbecauseunshareisunavailable;no
Incorrectcancellations FAIL successfulverifierresultisobserved. This
Malformedinput FAIL generationtracedoesnotdemonstratemore
Thepositivesuppliesonlyparsed;itbypassesthe completeself-testingthanCodex.
additionalraw-inputcheck.
Samehiddencases: validstructuredcallsandPython-stylerawcalls
Thepositivesuppliesthestructuredcallstogetherwithrawtextbeginning:
[RestaurantReservation_cancelBooking(bookingId=’R123’, ...), ...]
Theremainingargumentsandcallsareomittedhere. ThistextisnotJSON,soittriggerstheadditionalCodex
rejectioncondition.
Generatedverifier Positivecasesaccepted Negativecasesrejected
Codex 0/7 20/20
Piharness 7/7 20/20
Whatthecomparisonshows
Theverifierwithsuccessfulself-testsrejectseveryhiddenpositive;theotherverifieracceptsallpositiveswhile
rejectingthesamenegatives. Thecodeidentifiesanadditionalparsingcondition,andthesuccessfulself-tests
neverexerciseit. ThisconnectstheharnesscomparisontoSections4.1and4.2: thedifferenceconcernsthe
generatedchecks,whileneitherrunteststheraw-fieldassumption.
Figure9: Successfulself-testsbypasstheinputcheckthatrejectshiddenpositives. Thedifferenceisinput
compatibility,notadherencetoanexplicitlyspecifiedraw-callgrammar.
32
Case5. Constructinganexpectedstateversusobservingone
ClaudeOpus4.8 | PiharnessandClaudeCode | retail/49
Commontask
ExchangeamistakenlyorderedIPX7earbudforanearbudwiththerequestedwaterresistance,choosingthe
cheapestotherearbudinthesamedeliveredorder. Bothverifierschecktheexchangeargumentsandcompare
thecandidate’sresultingenvironmentstatewithanexpectedstate.
Piharness: constructatargetexchange ClaudeCode: executethetargetexchange
Thegeneratedverifierderivesexchangeparameters Theagentperformsthecorrectexchangeinthe
fromthecandidate’sreadresultsandconstructsan publicenvironmentandobtainstheresultingstate.
expectedexchangetranscript. Itreliesonreplaying Thegeneratedverifierincludesbothanexpected
thattranscripttoobtainausabletargetstatefor transcriptandtheobservedstateasafallback,then
comparison. Thereisnorecordedsuccessful comparescandidateexecutionwiththetargetand
executionestablishingthistargetduringgeneration. checkstherequiredexchangearguments.
Piharness: nosuccessfulexecutiontest ClaudeCode: testusingtheobservedstate
Shellcommandsfailbecauseunshareis Afterliveexecution,theagentcreatesalocal
unavailable. Theagent’sfinalresponsedescribesits substitutefortheenvironmentclientusingthe
sanitycasesas: observedstateandrunstheverifieronsixexamples.
Thecorrectexchangepasses;incorrect,empty,and
“reasoned,sincelocalexecisblocked”
malformedexamplesfail. Thesetestscheckverifier
Thesechecksdonotestablishthattheexpected logicwithasimulatedenvironmentresponse,notsix
transcriptcanbereplayedsuccessfully. completeliveexecutions.
Piharness: targetcomparisonunavailable ClaudeCode: samepositivesaccepted
Forthedesignatedhiddenpositive,therecorded Theverifieracceptsallninepositivesandrejectsall
decisionandfinalsourceidentifyrejectionwithouta eightnegatives. Thesuccessfuldecisionsare
usableexpected-statecomparison. Theprecise consistentwiththegeneratedcheckshavingaccess
internalreplayerrorisnotrecorded. Allnine toatargetstateobtainedfromactualenvironment
positivecasesarerejected. execution.
Comparisononthesametestcases
Generatedverifier Positivecasesaccepted Negativecasesrejected
Piharness 0/9 8/8
ClaudeCode 9/9 8/8
ThecontrastcomplementsSection4.3: constructingacheckdoesnotestablishthattheevidenceitrequirescan
beobtained. Here,liveexecutionestablishestheexpectedstateusedbythesuccessfulverifier;simulated
self-testsandhiddenevaluationthenprovidedifferentevidenceaboutitsdecisions.
Figure 10: Live execution establishes the target state used for verification. The paired traces distinguish a
constructedreplay,anobservedenvironmentresult,andtestsusingasimulatedresponse. Theydonotisolatewhich
harnesscomponentcausedthedifference.
33