← Back to summary
MiRL: Error-Targeted Reinforcement Learning for Correcting Terminal Agent Misbehaviors
Converted from PDF with markitdown. Layout may be rough.
MiRL: Error-Targeted Reinforcement Learning for Correcting Terminal
| | | | | Agent | Misbehaviors | | | | | | |
| ---------- | --- | -------- | --- | ------- | ------------ | ------------------ | --- | --- | ----------- | --- | --- |
| RunchuTian | | ZaidKhan | | ZunWang | | JustinChih-YaoChen | | | MohitBansal | | |
DepartmentofComputerScience
UniversityofNorthCarolinaatChapelHill
runchu@cs.unc.edu
| | | Abstract | | | | | (a) A few error types dominate the distribution | | | | |
| --- | --- | -------- | --- | --- | --- | --- | ----------------------------------------------- | --------- | --------------------------------------- | --- | --- |
| | | | | | | | #1 | 15.4% | Top 8 generalizable misbehaviors: 52.1% | | |
| | | | | | | | #1 # 2 | 1 1 . 5 % | | | |
Error-targeted learning gives a legible way # 3 1 1 . 2 % #1 Makes false claims about tests
6202 tcO 62 ]GL.sc[ 1v79653.0162:viXra #2
to improve a model: identify specific weak- 52.1% # 4 5 . 1 % #2 Repeats unnecessary checks
# 5 4 . 7 %
| | | | | | | | #3 #6 | 2.5% | #3 Retries without fixing the cause | | |
| ------ | ------ | ------ | --------- | ----- | ------ | --- | ----- | ---- | ------------------------------------- | --- | --- |
| nesses | of the | model, | train the | model | to fix | | #7 | 1.7% | | | |
| | | | | | | | #8 | 1.4% | #4 Fails to produce runnable commands | | |
theseweaknesses,andcheckwhethertheweak-
nessesarefixed. Westudyhowtoapplyerror- (b) MiRL: a misbehavior-driven RL recipe
| targeted | learning | to terminal | | agents, | which | | | | | | |
| -------- | -------- | ----------- | --- | ------- | ----- | --- | --- | --- | --- | --- | --- |
Standard RL (e.g., GRPO, DPPO): rollout start at step 1
| solve long-horizon | | tasks | specified | in | natural | | | | | | |
| ------------------ | --- | ----- | --------- | --- | ------- | ----------- | ------ | ------ | -------- | -------- | -------- |
| | | | | | | instruction | step 1 | step 2 | … step k | step k+1 | … step n |
languagebyinteractingwithacommandshell.
| | | | | | | reward = only outcome ro | | | | | outcome ro |
| --- | --- | --- | --- | --- | --- | ------------------------ | --- | --- | --- | --- | ---------- |
Existingerror-targetedmethodsaredesigned
MiRL (ours): rollout start at misbehavior-prone step k
| forsingle-turntasks,andexistinganalysesof | | | | | | | | | … | | … |
| ----------------------------------------- | --- | --- | --- | --- | --- | ----------- | ------ | ------ | ------ | -------- | ------ |
| | | | | | | instruction | step 1 | step 2 | step k | step k+1 | step n |
agentfailuresareusedtoevaluateanddebug
| | | | | | | reward = behavior re outcome ro | | | behavior re | | outcome ro |
| ---------------------------- | --- | --- | --- | --------- | --- | ----------------------------------- | --- | --- | ----------- | --- | ---------- |
| agentsratherthantotrainthem. | | | | Wepropose | | | | | | | |
Misbehavior-driven RL (MiRL), which first (c) Empirical results: data-efficient terminal agent training
profilestheerrorsofapolicytofindgeneraliz-
| | | | | | | | MiRL (ours) | TMax-4B | | Qwen3.5-4B | |
| --- | --- | --- | --- | --- | --- | --- | ----------- | ------- | --- | ---------- | --- |
ablemisbehaviors:
| | | kindsofwrongstepsthat | | | | )%( erocs etiL-BT | | | )%( erocs 1.2-BT 25 | | |
| ---------------------------- | ------ | --------------------- | -------- | -------- | --- | ----------------- | ----------------- | --- | ------------------- | -------------- | -------- |
| | | | | | | 50 | data-efficient | | | | |
| recuracrossmanyenvironments. | | | | MiRLthen | | | | | | data-efficient | |
| | | | | | | | | | 20 | 21.3±0.7 | 20.6±1.3 |
| trains the | policy | with RL, | starting | rollouts | at | 40 | 51.8±0.1 47.4±0.9 | | | | |
15
16.1
| thestepswherethesemisbehaviorsoccurand | | | | | | 30 | | 32.8 | | | |
| -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | ---- | --- | --- | --- |
10
rewarding a rollout only if it passes the tests 0 30K 70K 75K 0 30K 70K 75K
| | | | | | | | # total rollouts | | | # total rollouts | |
| ------------------------ | --- | --- | ------------- | --- | --- | -------- | ----------------------------------- | --- | --- | ---------------- | --- |
| andavoidsthemisbehavior. | | | OnQwen3.5-4B, | | | | | | | | |
| | | | | | | Figure1: | OverviewofMiRL.(a)Afewgeneralizable | | | | |
eightmisbehaviorsaccountfor52.1%ofwrong
| | | | | | | misbehaviors | account | for | a large | share | of error steps. |
| ----------------------------------------- | --- | --- | --- | --- | --- | ------------ | ------- | --- | ------- | ----- | --------------- |
| steps. AftertrainingwithMiRL,thepolicyre- | | | | | | | | | | | |
(b)MiRLstartsrolloutsfrommisbehaviorprefixesand
peatsthemeasuredmisbehaviors43%to57%
rewardsrolloutsthatpassthetestsandavoidthemis-
lessoftenonaverageonunseenprefixes.MiRL
| | | | | | | behavior. | (c)MiRLmatchesoroutperformsTMax-4B | | | | |
| --- | --- | --- | --- | --- | --- | --------- | ---------------------------------- | --- | --- | --- | --- |
outperformsTMax-4BonTerminal-BenchLite
| | | | | | | using35%ofitsrollouts. | | | TB-LiteandTB-2.1denote | | |
| --- | --- | --- | --- | --- | --- | ---------------------- | --- | --- | ---------------------- | --- | --- |
(51.8%vs.47.4%)andmatchesitonTerminal-
Terminal-BenchLite(OpenThoughts-Agentteametal.,
Bench2.1(21.3%vs.20.6%),whileusing35%
2026)andTerminal-Bench2.1(Merrilletal.,2026).
| of its rollouts. | | These results | | show that | error- | | | | | | |
| ---------------- | -------- | ------------- | ---------- | ------------- | ------ | -------------- | --- | ------- | ---- | ------ | ----------- |
| targeted | learning | can | be applied | to realistic, | | | | | | | |
| | | | | | | et al., 2026b) | and | methods | that | locate | the step at |
long-horizonterminaltasks.1
| | | | | | | which an | agent fails | (Zhang | et | al., 2025, | 2026b). |
| --- | --- | --- | --- | --- | --- | -------- | ----------- | ------ | --- | ---------- | ------- |
Theseanalysesaremostlyusedtoevaluateandde-
1 Introduction
bugagents,nottodecidewhattheagenttrainson.
| Error-targeted | learning | finds | the | errors | a model | | | | | | |
| -------------- | -------- | ----- | --- | ------ | ------- | --- | --- | --- | --- | --- | --- |
Terminalagents(Binetal.,2026;Liuetal.,2026)
makesandtrainsthemodeltostopmakingthose
aretypicallyinsteadtrainedbyscalingthenumber
errors (Khan et al., 2025; Lee et al., 2024). This of training environments (Pi et al., 2026; Gandhi
approachisdesirablebecauseitmakesmodelim-
| | | | | | | et al., 2026; | Wu | et al., | 2026; Ivison | | et al., 2026); |
| --- | --- | --- | --- | --- | --- | ------------- | --- | ------- | ------------ | --- | -------------- |
provementeasytounderstand,aseachtrainingex-
forexample,oneTMaxtrainingrunsamplesfrom
ampleistiedtosomenamederror,andaftertrain- 14.6K environments and takes two to three days
ingwecancheckwhetherthaterrorbecomesless oneightH100nodes(Ivisonetal.,2026). Toour
| frequent. | For LLM | agents, | prior | work | has built | | | | | | |
| --------- | ------- | ------- | ----- | ---- | --------- | ---------- | -------------- | --- | -------- | --- | ------------ |
| | | | | | | knowledge, | error-targeted | | learning | | has not been |
taxonomiesofagenterrors(Cemrietal.,2025;Zhu
appliedtoterminalagents.
1Projectpage:github.com/runchu-tian/MiRL Applying error-targeted learning to terminal
1
agents is not straightforward. Existing error- WetrainQwen3.5-4B(QwenTeam,2026a)with
targeted methods work on and are designed for MiRL using TMax’s RL recipe and evaluate on
single-turn problems: when the model answers a Terminal-BenchLite(OpenThoughts-Agentteam
problemincorrectly,thesemethodsgeneratesim- etal.,2026)andTerminal-Bench2.1(Merrilletal.,
ilarproblemsfortraining. Aterminalagenttakes 2026). Onheld-outprefixes,MiRLreducestherate
dozensofstepspertask,andafailedtaskisoften ofthemeasuredmisbehaviorsby43%to57%on
causedbyoneortwowrongsteps,suchasreport- average,measuredbytwobehaviorjudges. MiRL
ingthatthetestspasswithoutrunningthem. This also improves task performance: with the same
creates two difficulties. First, many wrong steps numberoftrainingenvironments(189),MiRLout-
are specific to one task, so fixing them one at a performsTMax-189(theTMaxrecipetrainedon
timedoesnothelponothertasks;trainingshould the same number of environments), PivotRL (Yi
insteadtargetgeneralizablemisbehaviors,kindsof et al., 2026), and a form of RaR (Gunjal et al.,
wrongstepsthatrecuracrossmanytasks. Second, 2026)enhancedwithoutcomerewards(RaR+),by
training from the start of a task spends most roll- at least 4.2 points on Terminal-Bench Lite and
outsonstepstheagentalreadyperformscorrectly, 2.2 points on Terminal-Bench 2.1. MiRL also
andtheagentmayneverreachthestatewherethe outperforms TMax-4B, the released checkpoint
misbehavioroccurred. Wethereforeask: (1)how trainedwiththefullTMaxrecipe,whichsamples
canweidentifythegeneralizablemisbehaviorsofa from14.6Kenvironments,onTerminal-BenchLite
terminalagent,and(2)howcanwetraintheagent (51.8% vs. 47.4%) and matches it on Terminal-
directlyatthestepswheretheyoccur? Bench2.1(21.3%vs.20.6%),whileusing35%of
We answer these questions by connecting two itsrollouts. Ablationsshowthatstartingfrommis-
tools: step-levelerroranalysisofagenttrajectories behaviorprefixesandaddingthebehaviorreward
and RL from intermediate states. To answer the each contribute (7.9 and 6.5 points on Terminal-
firstquestion,werunerroranalysisonthepolicy BenchLite). Theseresultsshowthaterror-targeted
weareabouttotrain,whichwecallerrorprofiling learningappliestoterminalagentswhentraining
(Figure1(a)). Werolloutthepolicyon2,017en- targetsgeneralizablemisbehaviorsdirectlyatthe
vironmentsfromTMax’senvironmentset(Ivison stepswheretheyoccur.
etal.,2026); anerrorlocalizerflagswrongsteps,
ateachermodelcorrectseachflaggedstepsothat 2 RelatedWork
therolloutcancontinueandreveallatererrors,and
an annotator model groups the wrong steps into Terminal agent training and data efficiency.
errortypes. Eighterrortypes,suchas“Makesfalse Recent terminal agents improve mainly by scal-
claimsabouttests”and“Failstoproducerunnable ing training data. Synthesis pipelines generate
commands”, cover 52.1% of all wrong steps and terminal environments from seed tasks and tax-
appearin75.4%oftheenvironments;wetreatthese onomies(Gandhietal.,2026;Zhuetal.,2026a;Pi
asthegeneralizablemisbehaviorstotarget. Toan- etal.,2026;Ivisonetal.,2026)orderivethemfrom
swer the second question, we introduce MiRL, a repositories and human-validated sources (Wu
misbehavior-drivenRLrecipe(Figure1(b)). MiRL et al., 2026; Shen et al., 2026; Li et al., 2026b).
buildsontwoknowntechniques,startingrollouts Some works train on tens of thousands of termi-
fromintermediatestates(Xietal.,2024;Yietal., nalenvironmentsorhundredsofthousandsoftra-
2026)andprovidingrewardsignalsbeyondthebi- jectories (Li et al., 2026b; Pi et al., 2026). Re-
naryverifieroutcome(Lightmanetal.,2024;Gun- cent works improve data efficiency: Terminal-
jaletal.,2026),eachguidedbytheerrorprofiling Lego (Yang et al., 2026) builds a more compact
outputs. Each group of rollouts starts at a misbe- dataset around the inspect-act-verify loop, and
havior prefix, the part of a trajectory just before Terminal-World(Chengetal.,2026)derives5.7K
amisbehaviorstep,andarolloutreceivesreward terminal environments from agent skills. Such a
onlyifitpassesthetestsandabehaviorjudgecon- trend is also observed in other domains, e.g., RL
firmsthatitavoidsthemisbehaviorofthatprefix. for math reasoning can be effective with about a
Tostartfromaprefix,MiRLrestoresthecontainer thousand or even a single carefully selected ex-
byrerunningtheprefix’scommandsinafreshcon- ample(Lietal.,2025;Wangetal.,2025). These
tainerandkeepstheprefixonlyifeverycommand works improve data efficiency by improving the
returnstherecordedoutput. quality of tasks and trajectories, or by selecting
2
| | | | | | | NAME | | CONTAINER | | | |
| --- | --- | --- | --- | --- | --- | ---- | --- | --------- | --- | --- | --- |
learnableexamples,whereasMiRLdoessobytar-
Deduplicate a photo library
| geting the | general, | recurring | | errors of the | policy | | | | | | |
| ---------- | -------- | --------- | --- | ------------- | ------ | --- | --- | --- | --- | --- | --- |
$ tree -L 1 ~/lib
| | | | | | | | | ├ ─ | p h o t o s / | ( 4 1 2 . j p | g ) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | ----------------- | --------------- | --- |
(generalizablemisbehaviors). INSTRUCTION ├ ─ b a c k u p / ( 3 8 9 . j p g )
└─ dupes/ (empty)
“Find every photo in backup/
| | | | | | | that is byte-identical to one in | | TESTS | | | |
| ----------------------- | --- | --- | --- | ---------------- | --- | -------------------------------- | --- | ----- | --- | --- | --- |
| Analyzingagentfailures. | | | | MAST(Cemrietal., | | | | | | | |
photos/, move it into dupes/,
2025)andAgentErrorTaxonomy(Zhuetal.,2026b) dupes/: all and only true duplicates
and list it in dupes/report.csv.
report.csv: lists every moved file
buildtaxonomiesoffailuremodesfromannotated
| | | | | | | Figure2: | Anexampleterminalenvironment. | | | Theagent | |
| --- | --- | --- | --- | --- | --- | -------- | ----------------------------- | --- | --- | -------- | --- |
agenttraces,andWho&When(Zhangetal.,2025)
mustmoveduplicatephotosfrombackup/intodupes/
and AgenTracer (Zhang et al., 2026b) locate the andlisttheminreport.csv;thetestschecktheresult.
| agentandthestepthatcauseafailure. | | | | Forterminal | | | | | | | |
| --------------------------------- | --- | --- | --- | ----------- | --- | --- | --- | --- | --- | --- | --- |
agents, TermiGen (Zhu et al., 2026a) finds that error profiling, as shown in Figure 3: we roll out
| | | | | | | the policy | on a pool | of environments, | | check | and |
| --- | --- | --- | --- | --- | --- | ---------- | --------- | ---------------- | --- | ----- | --- |
abouthalfoftheerrorsinasampleoftrajectories
areverificationfailures. Thesestudiesusefailure record the error steps, and group them into error
analysistoevaluate,diagnose,ordebugagents. We types. We call the types that recur across many
| | | | | | | environmentsgeneralizablemisbehaviors. | | | | | Totrain |
| --------- | -------------------- | --- | --------- | ---- | ----- | -------------------------------------- | ------------ | ----- | ----- | ------------ | ------- |
| use it to | drive data-efficient | | training: | MiRL | rolls | | | | | | |
| | | | | | | the policy | at the steps | where | these | misbehaviors | |
outfromthemisbehaviorprefixesfoundbyerror
profilingandrewardsarolloutonlyifitalsoavoids occur, we propose MiRL, our RL recipe, which
themisbehavior. starts rollouts from misbehavior prefixes instead
offromscratch,andaddsabehaviorrewardtothe
RLfromselectedstatesandstep-levelsupervi- outcomereward,soarolloutgainsrewardonlyifit
sion. Anoutcomerewardscoresawholetrajec-
passesthetestsandabehaviorjudgeconfirmsthat
torybyitsfinalstate. Thisgivescoarsecreditas- itavoidsthemisbehavior.
signmentbecausecorrectstepsinafailedtrajectory
| arepenalized,anderrorstepsinasuccessfultrajec- | | | | | | 3.1 Preliminaries | | | | | |
| ---------------------------------------------- | --- | -------------------------- | --- | --- | --- | ---------------------- | --- | --- | ----------------- | --- | --- |
| toryarereinforced. | | Togivemorespecificlearning | | | | | | | | | |
| | | | | | | Environmentsandagents. | | | Aterminalenviron- | | |
signals,onelineofworkstartsrolloutsfrominter-
| | | | | | | ment consists | of an instruction | | that | states | a user |
| --- | --- | --- | --- | --- | --- | ------------- | ----------------- | --- | ---- | ------ | ------ |
R3
mediatestates. (Xietal.,2024)startsRLfrom requirement, a container in which the agent runs
intermediatestatesofcorrectdemonstrationsand
| | | | | | | shell commands, | and | a test | suite that | checks | the |
| --------- | ----- | ----- | -------- | ---------------- | --- | ---------------------------------- | --- | ------ | ---------- | --------- | --- |
| moves the | start | point | backward | as a curriculum. | | | | | | | |
| | | | | | | finalstateofthecontainer(Figure2). | | | | Anagent,a | |
PivotRL(Yietal.,2026)samplesseveralactionsat language-modelpolicyπ,solvesataskovermul-
eachstepofsupervisedfine-tuning(SFT)trajecto-
| | | | | | | tiple steps. | Before each | step, | it reads | the | prefix, |
| --- | --- | --- | --- | --- | --- | ------------ | ----------- | ----- | -------- | --- | ------- |
riesandtrainsonlyonthestepswheretheseactions
whichincludestheinstruction,itsearliersteps,and
| get different | rewards. | | An action | is rewarded | if it | | | | | | |
| ------------- | -------- | --- | --------- | ----------- | ----- | ------------- | ------------------- | --- | --- | ------------- | --- |
| | | | | | | theiroutputs. | Itthentakesonestep: | | | ashortreason- | |
is equivalent to the action in the SFT trajectory. ing followed by a shell command. The sequence
Anotherlineaddssupervisionbeyondthetaskout-
ofstepsandtheiroutputsformsatrajectory,which
come: processrewardmodelsscoreeachinterme-
endswhentheagentsubmitsorreachesasteplimit.
diate step (Lightman et al., 2024), GiGPO (Feng Thetestsuitethenchecksthefinalstateofthecon-
| et al., 2025) | assigns | step-level | | credit across | roll- | | | | | | |
| ------------- | ------- | ---------- | --- | ------------- | ----- | ----------------------------- | --- | --- | --- | ------------- | --- |
| | | | | | | tainer,givinganoutcomerewardr | | | | of1ifalltests | |
o
| outs that | revisit | the same | state, | and RaR | (Gunjal | | | | | | |
| --------- | ------- | -------- | ------ | ------- | ------- | --- | --- | --- | --- | --- | --- |
passand0otherwise.
etal.,2026)scoreswholetrajectoriesagainsttask-
specificrubrics. Unlikethefirstline,MiRLstarts LLMsinerrorprofiling. Besidesthepolicy,er-
rolloutsfrommisbehaviorprefixescollectedbyer- ror profiling calls LLMs to play three roles: an
rorprofiling. Comparedtothesecondline,MiRL errorlocalizerthatcheckswhetherastepisprob-
| | | | | | | lematic | given its prefix, | a teacher | | that corrects | a |
| --- | --- | --- | --- | --- | --- | ------- | ----------------- | --------- | --- | ------------- | --- |
shapesrewardssothatarolloutmustavoidaspe-
cificmisbehaviortobereinforced. problematicstep,andanannotator thatproposes
errortypesandlabelseacherrorstepwiththem.
3 Methodology
| | | | | | | Errors | and misbehaviors. | | We write | the | prefix |
| --- | --- | --- | --- | --- | --- | ------ | ----------------- | --- | -------- | --- | ------ |
Weaimtoapplyerror-targetedlearningtoterminal of a step as h. We define an error step as a step
agents. Our intuition is that a few general errors thattheerrorlocalizermarksasproblematicgiven
account for many of the agent’s error steps, so its prefix. The annotator groups error steps with
fixingtheseerrorsmayimprovetheagentonmany thesamecauseintoerrortypesandgiveseachtype
environments. To identify these errors, we run a name and a definition. We call an error type
3
(a) Error localization (b) Error taxonomy (c) Misbehavior prefixes (d) MiRL training
| π policy | T teacher | error localizer | 1 | propose error types | | | unfiltered candidate | | | | | |
| --------------------------------------- | -------------------------- | --------------- | ----------- | -------------------- | -------------- | ------------- | --------------------------------------- | ------------------------- | ----- | ---------------------- | ---------------------- | ------- |
| | | | | | | | | | | | instr. 1 … | k−1 π |
| π | 1 | | | | A | | instr. | 1 … k−1 k | … | live | | |
| | ls photos backup | head | | | | | | | | | | rollout | |
| | | | | annotator | | | prefix h | | | | judged | |
| | | | error steps | | s in g le | batch labels | | | | | | |
| π | j awk '…' h.txt | xargs mv | | | | | | REQUIREMENT | | | | k k+1 … | n0−1 n0 |
| | | | 2 | keep recurring t | y p e | s | | | | | | |
| | | | | | | | π continues from h in a live container, | | | clean pass | | |
| Reason: reruns the same failing command | | | | | | | so it must reach the same state | | | | | |
| | | | | | | | | | | | k k+1 … | n1−1 n1 |
| T | | | | | | | | | | passes, but misbehaves | | |
| | j cd ~/lib && xargs mv … | | | | | | rerun h in a live | | | | | |
| | | | | | | | e x a | c t m a tch step by step | | | | |
| π | j+1 ls dupes | head | | | | | | | | | | k k + 1 … | n2−1 n2 |
| | | | a | cro ss b a t c h es | | | r e c o | r d e d | rerun | av | oids it , but fails | |
| | | | 3 | la b e l e v | ery error step | | | | | | | |
$ ls backup/
| π | k ls dupes | wc -l #verify | | A | | | | | = | | | k k+1 … | n3−1 n3 |
| --- | -------------------------- | --- | --- | --- | --- | --- | ------- | ------- | --- | -------------------- | ------- | ------- |
| | | | | | | | a.jpg … | a.jpg … | | misbehaves and fails | | |
$ date
Reason: counts files, not their content F a il s t o p r o d u ce ≠ r = r ∧ r
| | | | | | ru n n a b l e c o m | m ands | 10:42 | | 10:47 | | o e | update π |
| --- | ---------------------- | --- | --- | --- | --------------------------- | ------ | ----- | ---------- | ----- | --- | --- | -------- |
| T | k | | | | M a k e s f a l se claims | | | | | | | |
| | md5sum dupes/* | cmp … | | | | ab o u t t e s t s | | | all = keep | | | | final |
r
π k+1 head dupes/report.csv R e p e a ts unnecessary any = drop r e r o
ch e c k s
| | | | | | R e t r i e s w ithout fixing | | | | | | | 1 |
| ------ | --------- | --- | --- | --- | ------------------------------ | --- | ----------- | ------------------------- | --- | --- | --- | --- |
| instr. | 1 … j−1 j | | | | th e c a u s e | | filtered pr | e fi x es live containers | | | | |
| | | | | | | | | s ta r t | | | | 0 |
| instr. | 1 … k−1 k | | | | | | | | | | | 0 |
prefix step reason l o c a l i z e d misbehaviors long tail rerun 0
e r r o r s t e ps
Figure3: Overview. (a)Theerrorlocalizerflagswrongpolicysteps,andtheteachercorrectsthem;(b)theannotator
groupserrorstepsintotypes,whereafewmisbehaviorsdominate;(c)prefixesthatrerunexactlyarekeptforliveRL
rollouts;(d)rolloutsstartfromthemandgainrewardonlyiftheypassthetestsandavoidthemisbehavior.
a generalizable misbehavior, or misbehavior for a name and a definition for each type. We repeat
short,whenitrecursacrossmanyenvironmentsand thisonseveralbatchesandkeepthetypesthatrecur
isnottiedtoveryfewofthem. Wecalltheprefixof acrossbatches. Thisgivesafixedsetoferrortypes.
anerrorstepofamisbehavioramisbehaviorprefix. Theannotatorthenlabelseveryerrorstepwithone
| | | | | | | ormoretypesfromthisset, | | | | orwithnoneifitfits | | |
| --- | --- | --- | --- | --- | --- | ----------------------- | --- | --- | --- | ------------------ | --- | --- |
3.2 ErrorLocalization
| | | | | | | no | known | type. Finally, | | we keep | the error | types |
| --- | --- | --- | --- | --- | --- | --- | ----- | -------------- | --- | ------- | --------- | ----- |
Tolocatetheerrorsofthepolicy,werollitouton that cover at least a fixed share of the error steps
apoolofenvironmentsandlettheteacherreplace andtreattheothersasalongtailofraretypes.
| each error | step with | a corrected | one, | so | that the | | | | | | | |
| ---------- | --------- | ----------- | ---- | --- | -------- | --- | ---------------- | --- | ----- | ----------- | ----- | ----- |
| | | | | | | | In our analysis, | | a few | error types | cover | about |
rolloutcanreveallatererrors.
| | | | | | | half | of the | error steps | and | recur | across | many en- |
| --- | --- | --- | --- | --- | --- | ---- | ------ | ----------- | --- | ----- | ------ | -------- |
At each step of a rollout, error profiling goes vironments (§4.2), so they can be considered as
through four stages, as in Figure 3(a). (1) The generalizablemisbehaviors. Becausethesemisbe-
policyproposesitsnextstepfromthecurrentpre-
haviorsrecuracrossenvironments,asmallnumber
fix. (2)Theerrorlocalizerchecksthestepwithits
ofenvironmentsalreadygivesmanyexamplesof
prefix,flagsitonlyifitislikelytoleadtofailure, each,andfixingthemmayalsohelpthepolicyon
| and writes | a short | explanation. | (3) | For a | flagged | | | | | | | |
| ---------- | ------- | ------------ | --- | ----- | ------- | --- | --- | --- | --- | --- | --- | --- |
unseenenvironments.
step,theteacherwritesacorrectionfromthesame
prefix,sothattherolloutcancontinuewithoutbe-
| | | | | | | 3.4 | RolloutfromMisbehaviorPrefixes | | | | | |
| ----------------------------- | --- | --- | --------------- | --- | --- | --- | ------------------------------ | --- | --- | --- | --- | --- |
| ingpushedintoincorrectstates. | | | Thecorrectionis | | | | | | | | | |
checked in the same way as a step of the policy. Standard RL recipes used for terminal agent
(4)Theacceptedstep,eitherthepolicy’sownstep training, such as GRPO (Shao et al., 2024) and
| | | | | | | DPPO | (Qi | et al., 2026), | | start every | rollout | anew, |
| ---------------------------- | --- | --- | ----------- | --- | --- | ---- | --- | -------------- | --- | ----------- | ------- | ----- |
| oracorrectionfromtheteacher, | | | isexecuted, | | and | | | | | | | |
the rollout returns to (1) for the next step. If the fromstepzero. Animportantcharacteristicofthese
teacher fails to give a correction that passes this methodsisthattheyareon-policy,thatis,theytrain
| | | | | | | onrolloutsgeneratedbythepolicyitself. | | | | | | Starting |
| ----- | ------------ | ---------- | ------- | ------------- | --- | ------------------------------------- | --- | --- | --- | --- | --- | -------- |
| check | within three | tries, the | rollout | is discarded. | | | | | | | | |
During this process, each error step is recorded rollouts anew would mean that we could not cor-
withitsprefixandtheerrorlocalizer’sexplanation. rectaspecificmisbehaviorinanon-policymanner
unlesstheidenticalmisbehavioroccurredandwas
3.3 ErrorTaxonomyConstruction detected in a fresh rollout. The highly stochastic
To find the errors that recur across environments, natureoflong-horizonterminalagenttrajectories
wegrouptheerrorstepsintoerrortypes,asinFig- meansthatgeneratingrolloutsthatdisplayapartic-
ure3(b). First,theannotatorreadsbatchesoferror ularmisbehaviorwouldbeverycompute-intensive.
steps, together with the error localizer’s explana- Tofixthemisbehaviorsinamoretargetedandef-
tions,andproposeserrortypesforeachbatch,with ficientmanner,MiRLinsteadstartseachgroupof
4
rolloutsfromapre-existingtrajectoryprefixwhere Opus 5 (Anthropic, 2026) as the annotator. We
thepolicyexhibitedamisbehavior. Thepolicythen keepeveryerrortypethatcoversatleast1%ofthe
continuesfromtheprefixuntiltherolloutends. We errorsteps,whichgiveseightgeneralizablemisbe-
selectafixednumberofmisbehaviorprefixesfor haviors. Fortheseeightmisbehaviors,wesample
eachmisbehaviortoformthetrainingdata. 80misbehaviorprefixeseach,whichgives640pre-
However,aliverolloutcannotstartfromaprefix fixes. After the rerun check in Figure 3(c), 219
directly. Tocontinuefromstepk,thepolicyneeds prefixesremain(34%),from189environments.
thecontainerstateatstepk,buttherecordeddata
| | | | | | Modelsandtraining. | | | WeuseQwen3.5-4B,the | | | |
| --------- | ---------- | ----------- | ---------- | ----- | ------------------ | --- | --- | ------------------- | --- | --- | --- |
| keep only | the prefix | and not the | container, | since | | | | | | | |
samebasemodelasTMax-4B,asthepolicymodel.
| saving a snapshot | of | the container | at every | step | | | | | | | |
| ----------------- | ----------- | ------------- | -------- | --------- | --------- | ------ | --------- | ---------- | --- | ---------- | ----- |
| | | | | | We follow | the | RL recipe | of TMax | | and use | their |
| would require | substantial | storage. | We | therefore | | | | | | | |
| | | | | | harness, | except | for a | group size | of | 16 instead | of |
needtoreconstructthestatefromtheprefix,asin
| | | | | | 32 to control | the | compute | budget, | | and a | total of |
| ----------- | ----------------------------- | --- | --- | --- | ------------- | ----- | ------- | ------- | ---- | ------------ | -------- |
| Figure3(c): | wererunthecommandsoftheprefix | | | | | | | | | | |
| | | | | | 160 training | steps | instead | of | 500. | The behavior | |
stepbystepinafreshcontainerofitsenvironment,
judgeisGemini3.8Flash(Google,2026),which
andkeeptheprefixonlyifeverycommandreturns
| | | | | | reads the | m = | 5 steps | after the | prefix. | It | returns |
| ---------------------------------- | --- | --- | -------- | --- | --------- | --- | ------- | --------- | ------- | --- | ------- |
| exactlythesameoutputasintheprefix. | | | Manycon- | | | | | | | | |
onlyabinarysignal,sothepolicyreceivesnotext
| tainersareflaky: | thesamecommandmayproduce | | | | | | | | | | |
| ---------------- | ------------------------ | --- | --- | --- | -------- | -------- | ----- | ----------- | --- | --- | ------- |
| | | | | | from the | behavior | judge | to imitate. | | We | run our |
differentoutputsdependingontime,randomness,
| | | | | | experimentson8A10080GBGPUs. | | | | | AppendixA | |
| -------------------- | --- | ----------------------- | --- | --- | --------------------------- | --- | --- | --- | --- | --------- | --- |
| ornetworkconditions. | | Suchprefixescannotreach | | | | | | | | | |
givesthefullsettingsandprompts.
| therequiredstate,sowediscardthem. | | | Thetraining | | | | | | | | |
| --------------------------------- | --- | --- | ----------- | --- | --- | --- | --- | --- | --- | --- | --- |
environmentsarethoseoftheremainingprefixes. Baselines. We compare MiRL with TMax-4B,
thereleasedcheckpoint(step200of500)trained
3.5 Misbehavior-AwareRewardDesign with the full TMax recipe, which samples train-
The outcome reward r checks only whether the ingenvironmentsfromapoolof14.6K.Because
o
tests pass, so it cannot give feedback on whether Terminal-Benchscoresaresensitivetotheevalua-
tionhardware,wereportbothitspublishedscores
| arolloutavoidsthemisbehavior. | | | Wethereforein- | | | | | | | | |
| ----------------------------- | --- | --- | -------------- | --- | --- | --- | --- | --- | --- | --- | --- |
troduceabehaviorrewardr Thebehaviorjudge andourevaluationofthereleasedcheckpoint. We
e .
reads the first m steps that the policy takes after alsotrainthreebaselineswiththesameconfigura-
theprefix,togetherwiththedefinitionofthemis- tionasMiRL,exceptthattheir189environments
aresampledatrandomfromthesame2,017envi-
| behavior ofthatprefix. | | Itgivesr | e = 1ifthepol- | | | | | | | | |
| ---------------------- | --- | -------- | -------------- | --- | --- | --- | --- | --- | --- | --- | --- |
icyavoidsthemisbehavior,andr = 0otherwise. ronments,sincethesebaselinesdonotselectenvi-
e
MiRLcombinesthetworewardsasr = r ∧r ,as ronments by error profiling or any other method.
o e
| | | | | | The first, | TMax-189, | | replicates | the | TMax | recipe |
| --- | --- | --- | --- | --- | ---------- | --------- | --- | ---------- | --- | ---- | ------ |
inFigure3(d),soarolloutgainsrewardonlywhen
the tests pass and the misbehavior is avoided. In butonlyon189environments: itstartsrolloutsat
thisway,MiRLavoidsreinforcingtrajectoriesthat the beginning of each task and uses the outcome
| | | | | | reward. | Theothertwoaretheclosestpriormeth- | | | | | |
| --- | --- | --- | --- | --- | ------- | ---------------------------------- | --- | --- | --- | --- | --- |
passthetestsbutstillcontainthemisbehavior.
| | | | | | ods to | the two | parts of | MiRL. | PivotRL | (Yi | et al., |
| ------------- | --- | --- | --- | --- | ---------------------------------------------- | ------- | -------- | ------- | ------- | ----- | ------- |
| 4 Experiments | | | | | 2026)alsotrainsfromselectedstepsofatrajectory: | | | | | | |
| | | | | | it samples | several | actions | at each | step, | keeps | the |
4.1 Setup
stepswheretheseactionsgetdifferentrewards,and
Trainingdata. Werunerrorprofilingon2,017 rewards each action mainly with an LLM judge.
randomterminalenvironmentsfromTMax’senvi- SinceeachPivotRLupdatetrainsonasinglestep
ronmentset, whichdonotoverlapthe evaluation ratherthanonwholerollouts,matchingthenumber
benchmarks. Werolloutthepolicyoneachenvi- oftrainingstepsisnotmeaningful,sowegivePiv-
ronment with 2 rollout seeds. We use Qwen3.8- otRLaboutthesameGPUbudgetasTMax-189for
27B (Qwen Team, 2026b) as the error localizer, itsfulltrainingrun(≈300A100hours). RaR(Gun-
becauseitisstrongenoughtojudgeterminalsteps, jal et al., 2026) also rewards the process with an
withareported73.0%onTerminal-Bench2.1. We LLM:ajudgescoresthewholetrajectoryagainsta
alsouseitastheteacher,becauseitbelongstothe rubricgeneratedforeachenvironment. Theorigi-
samemodelfamilyasthepolicy,soitscorrections nalRaRusesonlythisrubricscore,soforafairer
stay close to the policy’s own distribution (Yang comparisonwestrengthenitwiththeoutcomere-
et al., 2026; Raoof et al., 2026). We use Claude wardandcallitRaR+: whenthejudgewritesthe
5
| appear in % of error steps | | | | appear in % of environments | | | | | | | | |
| -------------------------- | --- | --- | --- | --------------------------- | --- | --- | --- | ---------- | ------- | -------- | --- | -------- |
| | | | | | | | | Qwen3.5-4B | TMax-4B | TMax-189 | | Reported |
15.4 #1 Makes false claims about tests PivotRL RaR+ MiRL (ours)
46.7%
| | #2 | Repeats unnecessary checks | | | | | | TB-Lite | | | TB-2.1 | |
| --- | --- | -------------------------- | --- | --- | --- | ----- | --- | ------- | -------- | --- | ------ | -------- |
| | | | | | | 42.2% | | | 31.8±3.8 | | | 14.2±2.3 |
#3 Retries without fixing the cause
| 11.5 | | | | | | | 32.8±1.5 | | | 16.1±1.0 | | |
| ---- | ---- | --- | --- | --- | --- | ----- | -------- | --- | --- | -------- | --- | --- |
| | 11.2 | | | | | 36.5% | | | | | | |
| | | | | | | | 42.6±1.5 | | | 19.9±1.1 | | |
#4 Fails to produce runnable commands
19.4%
| | | | | | | | 47.4±0.9 | | | 20.6±1.3 | | |
| --- | --------------------------------------------- | --- | --------------------------------- | --- | --- | --- | -------- | --- | --- | -------- | --- | --- |
| | top-8 misbehaviors: 75.4% of the environments | | | | | | 43.4±1.7 | | | 17.6±0.4 | | |
| | | | | | | | 47.2±0.8 | | | 18.7±1.0 | | |
| | | | long-tailed, specific error types | | | | 47.6±1.6 | | | 19.1±2.6 | | |
5.1
| | 4.7 | | Uses a greedy regex on log lines | | | | 51.8±0.1 | | | 21.3±0.7 | | |
| --- | --- | --- | -------------------------------- | --- | --- | --- | ----------------------- | ----------- | ----- | ----------------------- | --- | ------------ |
| | | | Replaces udb inside udbuf | | | | 20 | 30 | 40 50 | 10 | 15 | 20 |
| | | 2.5 | Divides all temperatures by 100 | | | | | | | Score (%) | | |
| | | 1.7 | 1.4 | | | | | | | | | |
| | | | | | | | Figure | 5: Terminal | task | performance: | | mean (%) and |
| | | | | | | | standarderrorover3runs. | | | Dashedboxesarethescores | | |
top-8 misbehaviors: 52.1% of the error steps reportedintheTMaxpaper(Ivisonetal.,2026).
Figure4: Errorstepsconcentrateinafewmisbehaviors,
arisefromadetailofoneenvironmentandrarely
followedbyalongtailofenvironment-specifictypes;
| | | | | | | | carryovertootherenvironments. | | | | AppendixBlists | |
| --- | --- | --- | --- | --- | --- | --- | ----------------------------- | --- | --- | --- | -------------- | --- |
thetopfourappearacrossmanyenvironments.
moreexamplesandper-typestatistics.
rubricforanenvironment,italsosetstheweights
| | | | | | | | 4.3 | MainResults | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | ----------- | --- | --- | --- | --- |
oftherubricscoreandtheoutcomereward.
| | | | | | | | MiRLoutperformsTMax-4B. | | | | Figure5shows | |
| ----------- | ----------------------------- | --- | --- | --- | --- | --- | ----------------------- | --- | ----------------- | --- | ------------ | ----------- |
| Evaluation. | Weevaluateontwoterminalbench- | | | | | | | | | | | |
| | | | | | | | the terminal | | task performance. | | On | our evalua- |
marks,Terminal-BenchLite(TB-Lite;100tasks;
tionhardware,boththebasemodelandTMax-4B
| OpenThoughts-Agent | | | team | et al., | 2026) | and | | | | | | |
| ------------------ | --- | --- | -------- | ------- | -------------- | --- | ------- | ------ | ------ | ---------- | --------- | ---- |
| | | | | | | | achieve | higher | scores | than their | published | ones |
| Terminal-Bench | | 2.1 | (TB-2.1; | 89 | tasks; Merrill | | | | | | | |
(e.g.,47.4%vs.42.6%forTMax-4BonTB-Lite),
| et al., 2026), | and | report | the | mean | and standard | | | | | | | |
| -------------- | --- | ------ | --- | ---- | ------------ | --- | ------------------------------------ | --- | --- | --- | --- | ---- |
| | | | | | | | sowecompareallrecipeswithourownruns. | | | | | MiRL |
errorover3runs,followingTMax;someTB-Lite
reaches51.8%onTB-Liteand21.3%onTB-2.1,
| tasksgivepartialcredit. | | | Toselectthecheckpoint, | | | | | | | | | |
| ----------------------- | --- | --- | ---------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
comparedwith47.4%and20.6%forTMax-4Bon
wefirsttakethefivecheckpointswiththehighest
| | | | | | | | ourevaluationhardware: | | | itgains4.4morepoints | | |
| --- | --- | --- | --- | --- | --- | --- | ---------------------- | --- | --- | -------------------- | --- | --- |
movingaverageofthetrainingreward,andthen,as
onTB-LiteandmatchesTMax-4BonTB-2.1.
inTMax,keeptheonewiththebestTB-Litescore.
MiRLoutperformsallbaselinestrainedonthe
4.2 MisbehaviorAnalysis
| | | | | | | | samenumberofenvironments. | | | | On189random | |
| --- | --- | --- | --- | --- | --- | --- | ------------------------- | --- | --- | --- | ----------- | --- |
Error profiling on the 2,017 environments yields environments,TMax-189improvesthebasemodel
12,746 error steps. Figure 4 shows their distribu- by 10.6 points on TB-Lite, which is the smallest
tionovererrortypes. Theeightmostfrequenterror gain among the four recipes trained with a con-
typescover52.1%ofallerrorsteps. Eachappears trollednumberofenvironments. Thisindicatesthat
in6.9%(140)to46.7%(942)ofthe2,017environ- asmallrandomsetisnotenoughforthestandard
recipeofRLwithverifiablerewards(RLVR;Lam-
| ments, | and at least | one | appears | in | 75.4% (1,521) | | | | | | | |
| ------ | ------------ | --- | ------- | --- | ------------- | --- | --- | --- | --- | --- | --- | --- |
of them. None is specific to a particular environ- bertetal.,2025). PivotRLandRaR+improvethe
ment, so they can be considered as generalizable basemodelmore,by14.4and14.8pointsonTB-
misbehaviors(§3.1). Priorworkobservesasimilar Lite. MiRLoutperformsthebestofthesebaselines
concentration: in a sample of agent trajectories, by 4.2 points on TB-Lite and 2.2 points on TB-
TermiGen finds that about half of the errors are 2.1. Theresultsuggeststhattargetinggeneralizable
verificationfailures,inwhichtheagentomitsafi- misbehaviorsisaneffectivewaytochoosewhere
rolloutsstartandwhatthejudgechecks.
| nal check | (Zhu | et al., | 2026a). | Our | most frequent | | | | | | | |
| --------- | ---- | ------- | ------- | --- | ------------- | --- | --- | --- | --- | --- | --- | --- |
misbehavior,“Makesfalseclaimsabouttests”,is We further show a 16-seed evaluation for en-
| alsoakindofverificationerror. | | | | | | | hancedrobustness(§4.5). | | | | | |
| ----------------------------- | --------- | ------ | ----- | ------ | ------------ | --- | ----------------------- | --------- | --- | --- | --- | --- |
| The | remaining | errors | | form a | long tail | of | | | | | | |
| environment-specific | | | error | types. | Two examples | | 4.4 | Ablations | | | | |
are “Uses a greedy regex on log lines” and “Re- WeablateMiRLbyaddingitsthreepartsoneata
places udb inside udbuf” (Figure 4). Such errors timetoTMax-189(Figure6). Trainingontheenvi-
6
Qwen3.5-4B (no RL) TMax-189 + env. selection Qwen3.5-4B TMax-4B MiRL (ours)
+ prefix start + r (MiRL) Behavior judge = Gemini Behavior judge = Qwen
e
TB-Lite TB-2.1 Makes false claims about tests Makes false claims about tests
| | | | | | | | | 22.5 | | | | 28.4 | |
| -------- | ----- | --- | --------- | --- | --- | --- | ---------------------------------- | ---- | ---- | --- | ---------------------------------- | ---- | ---- |
| 32.8±1.5 | | | 16.1±1.0 | | | | | | 26.7 | | | | 29.8 |
| | | | | | | | 8.1 | | | | 11.7 | | |
| 43.4±1.7 | | | 17.6±0.4 | | | | | | | | | | |
| | | | | | | | Retries without fixing the cause | | | | Retries without fixing the cause | | |
| 37.4±2.2 | | | 16.5±2.6 | | | | | | | | | | |
| | | | | | | | | 12.8 | | | | | 35.2 |
| 45.3±1.6 | | | 18.0±0.6 | | | | | | 20.2 | | | | 29.7 |
| | | | | | | | | 13.0 | | | | 23.9 | |
| 51.8±0.1 | | | 21.3±0.7 | | | | | | | | | | |
| | | | | | | | Fails to produce runnable commands | | | | Fails to produce runnable commands | | |
| 20 | 30 40 | 50 | 10 | | 15 | 20 | | | | | | | |
| | | | | | | | 0.2 | | | | 0.8 | | |
| | | | Score (%) | | | | | 10.5 | | | 8.8 | | |
| | | | | | | | 0.2 | | | | 0.9 | | |
Figure6: AblationsofMiRL.StartingfromTMax-189, Prints a binary file as text Prints a binary file as text
| eachbaraddsonepart: | | | environmentselection, | | | prefix | | | 21.9 | | | 23.8 | |
| --------------------------- | --- | --- | --------------------- | --- | --- | ------ | --- | --- | ----- | --- | ---- | ---- | ----- |
| | | | | | | | 4.1 | | | | 4.6 | | |
| | | | | | | | | 5.3 | | | 5.3 | | |
| start,andthebehaviorrewardr | | | | e . | | | | | | | | | |
| | | | | | | | 0 | 10 | 20 30 | 40 | 0 10 | 20 | 30 40 |
ronmentswherethemisbehaviorsoccurlowersthe Misbehavior rate (%, ↓)
Figure7:Misbehaviorrate(%):theshareof16samples
scorefrom43.4%to37.4%onTB-Liteandfrom
| | | | | | | | thatrepeatthemisbehavioronunseenprefixes. | | | | | | Judges: |
| --------------------- | --- | --- | --- | -------------------- | --- | --- | ----------------------------------------- | --- | --- | --- | --- | --- | ------- |
| 17.6%to16.5%onTB-2.1. | | | | Weattributethistothe | | | | | | | | | |
Gemini3.8Flash(usedintraining)andQwen3.8-27B.
higherdifficultyoftheseenvironmentsforthepol-
| icy: the | share of | groups | in | which | all rollouts | fail, | | | | | | | |
| ---------- | -------- | -------- | ------- | ----- | ------------ | ----- | ------ | --- | ------ | ----- | --------- | --- | --------- |
| | | | | | | | Gemini | 3.8 | Flash, | which | MiRL uses | in | training, |
| which give | no | learning | signal, | rises | from | 27.3% | | | | | | | |
andQwen3.8-27B(QwenTeam,2026b).
| to42.9%. | However,environmentselectionisnot | | | | | | | | | | | | |
| -------- | --------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
AsshowninFigure7,MiRLlowersthreeofthe
| meanttohelponitsown. | | | Instead,itprovidesthe | | | | | | | | | | |
| -------------------------------------- | --- | --- | --------------------- | --- | --- | ------ | ----------- | ------------ | ------------ | --- | ------------ | ----- | ----- |
| | | | | | | | four | misbehaviors | under | | both judges. | For | exam- |
| misbehaviorprefixesforthenexttwoparts. | | | | | | Start- | | | | | | | |
| | | | | | | | ple, “Makes | | false claims | | about tests” | drops | from |
ingrolloutsatmisbehaviorprefixesadds7.9points
26.7%to8.1%underGeminiandfrom29.8%to
| onTB-Liteand1.5pointsonTB-2.1. | | | | | Weattribute | | | | | | | | |
| ------------------------------ | ----------- | --- | ------ | ----- | ----------- | -------- | ----- | ----- | ----- | ----- | -------- | ----- | ----- |
| | | | | | | | 11.7% | under | Qwen. | Since | the rate | drops | under |
| this gain | to training | | on the | steps | where | the pol- | | | | | | | |
bothjudges,MiRLlikelyreducesthesemisbehav-
icymisbehaves,eventhoughnorewardchecksthe
| | | | | | | | iors | rather | than only | learning | to | satisfy | its own |
| --------------- | --- | ------------------ | --- | --- | --- | ------- | ---- | ------ | --------- | -------- | --- | ------- | ------- |
| misbehavioryet. | | Thebehaviorrewardr | | | e | adds6.5 | | | | | | | |
judge. Thefourth,“Printsabinaryfileastext”,is
| pointsonTB-Liteand3.3pointsonTB-2.1. | | | | | | With | | | | | | | |
| ------------------------------------ | --- | --- | --- | --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- |
rareinthebasepolicyandstaysatabout5%.
r ,therewardisstricterthantheoutcomereward
e
| | | | | | | | Training | | with the | outcome | reward | alone | does |
| --------- | ---------- | --- | ------------ | ---- | -------- | --------- | ---------------- | --- | -------- | ------------- | ------ | ------- | ---- |
| alone: it | excludes | the | rollouts | that | pass | the tests | | | | | | | |
| | | | | | | | not consistently | | reduce | misbehaviors. | | TMax-4B | |
| but still | misbehave. | | For example, | | a policy | may | | | | | | | |
reduces“Failstoproducerunnablecommands”to
| claimsuccesswithoutverification. | | | | | Whenitsfirst | | | | | | | | |
| -------------------------------- | ------- | ----- | -------- | --- | ------------ | ------ | -------- | ------ | ---------- | ------- | ------------ | ------ | ----- |
| | | | | | | | below | 1% | under both | judges, | similar | to | MiRL. |
| attempt | happens | to be | correct, | the | rollout | passes | | | | | | | |
| | | | | | | | However, | “Makes | | false | claims about | tests” | de- |
andthemisbehaviorisnotfatal;whenthefirstat-
creasesonlyslightly(26.7%to22.5%underGem-
| tempt is | wrong, | the | same unchecked | | submission | | | | | | | | |
| -------------------- | ------ | --- | --------------------- | --- | ---------- | --- | ------- | ----- | -------- | --- | ------------ | --- | -------- |
| | | | | | | | ini and | 29.8% | to 28.4% | | under Qwen). | | “Retries |
| makestherolloutfail. | | | Theoutcomerewardalone | | | | | | | | | | |
withoutfixingthecause”decreasesunderGemini
stillrewardsthepassingrollout,sothepolicymay
butincreasesunderQwen(29.7%to35.2%),while
learntorepeatthemisbehavior.
| | | | | | | | “Prints | a binary | file | as text” | becomes | about | five |
| --- | --- | --- | --- | --- | --- | --- | ------- | -------- | ---- | -------- | ------- | ----- | ---- |
timesmorefrequent(4.1%to21.9%).
4.5 Analysis
MiRLreducesmisbehaviorfrequency. Tover- Pass@k: Sharpeningorlearning? Acommon
ifythatMiRLreducesthemisbehaviors,wemea- critiqueofRListhatitsharpenstheoutputdistribu-
surehowofteneachpolicyrepeatsamisbehavior tionofthebasemodelratherthanteachingitnew
before and after training. We measure four mis- capabilities (Yue et al., 2025; Zhao et al., 2025;
behaviorswithenoughprefixesleftafterthererun Huang et al., 2025). In that case, RL improves
check,eachon20newlysampledmisbehaviorpre- pass@1, but pass@k at large k improves little or
fixesthatareunseenintraining. Foreachunseen even falls below that of the base model. We test
prefix,wesamplethepolicy16times,andabehav- this on TB-2.1 with the unbiased pass@k (Chen
iorjudgecheckswhethereachsamplerepeatsthe et al., 2021), using 16 runs per task. Figure 8(a)
misbehavior of that prefix. The misbehavior rate comparesthescores. TMax-4Bshowsthispattern:
is the percentage of samples that repeat the mis- itimprovesoverQwen3.5-4Batpass@1, butthe
behavior. We measure with two behavior judges: basemodelcatchesupataboutk = 11andisahead
7
Qwen3.5-4B TMax-4B MiRL (ours) TMax-189 PivotRL RaR+ MiRL (ours)
| | | | | | | | | | Preprocessing | | | RL training | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | ------------- | --- | --- | ----------- | --- |
(a) Unbiased pass@k on TB-2.1
42
| | | | | | | 39.3 | | | Rollouts (#) | | Tokens (in ↑ / out ↓) | | |
| --- | --- | --- | --- | --- | --- | ---- | --- | --- | ------------ | --- | --------------------- | --- | --- |
)%( k@ssap desaibnU 36
| | | | | | | 34.8 | 180K | | | 15B | | | |
| --- | --- | --- | --- | --- | --- | ---- | ---- | --- | ------ | --- | ----- | ---- | ----- |
| | | | | | | 33.7 | | | | | 10.9B | | 10.5B |
| | | | | | | | | | 133.1K | 10B | | | |
| | 30 | | | | | | 120K | | | | | | 4.8B |
| | | | | | | | | | | | 5B | 1.3B | |
24
| | | | | | | | 60K | | | | | 53M | |
| --- | --- | --------------------------------------- | ------------------- | --- | --- | --- | ----- | ---------------- | ------- | ---- | ---- | ----------- | ----- |
| | | | | | | | | 26.4K | 26.2K | 250M | | | 126M |
| | | | | | | | | | 10.3K | | | | 262M |
| | 18 | | | | | | | | | | 344M | | |
| | | | | | | | | 0 | | 500M | | | |
| | 12 | | | | | | | Environments (#) | | | | API fee ($) | |
| | 1 | 4 | | 8 | 12 | 16 | 2,400 | | | | 900 | | |
| | | | k (# runs per task) | | | | | | 2,017 | | | | 737.1 |
| | | | | | | | 1,600 | | | | 600 | | |
| | | (b) Pass rate of each task over 16 runs | | | | | | | | | | 519.5 | |
| | | | | | | | | 800 | | | 300 | | 251.1 |
| | | | | | | | | 189 | 189 189 | | | | |
0
| | | | | | | | | 0 | | | 0 | | |
| --- | -------------------------------- | --- | --- | --- | ------------------ | --- | --- | ----------------- | --- | ----- | --- | -------------- | --- |
| | 42/89 tasks solved at least once | | | | 47/89 never solved | | | | | | | | |
| | | | | | | | | GPU time (A100 h) | | | | Total cost ($) | |
| | # runs that solve it: | | 0 | 16 | 0 | 16 | | | | | | | |
| | | | | | | | | 240 | | 2,400 | | | |
Figure 8: Pass@k on TB-2.1 with 16 runs per task. 202.7 202.8 1,960.3
166.4
| (a)Unbiasedpass@kofQwen3.5-4B,TMax-4B,and | | | | | | | | 160 | | 1,600 | 1,257.1 | | |
| ----------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | ----- | ----- | ------- | ----- | ----- |
| | | | | | | | | | 105.9 | | | 999.3 | |
| MiRL.(b)Passrateofeachtaskoverthe16runs. | | | | | | | | | | | | | 769.6 |
| | | | | | | | | 80 | | | 800 | | |
| | | | | | | | | 0 | | | 0 | | |
TMax-4B (100%) MiRL: preprocessing MiRL: RL training Figure10: Trainingresourcesofthefourrecipeson189
environments,uptothebestcheckpointratherthanthe
Rollouts
| | | | | | | −65% | fulltrainingrun. | | Checkeredsegmentsarepreprocessing | | | | |
| --- | --- | --- | --- | --- | --- | ---- | ---------------- | --- | --------------------------------- | --- | --- | --- | --- |
75.9K → 26.2K
| | Input tokens | | | | | | andsolidsegmentsareRLtraining. | | | | | | |
| --- | ------------ | --- | --- | --- | --- | --- | ------------------------------ | --- | --- | --- | --- | --- | --- |
−66%
30.9B → 10.5B
| | Output tokens | | | | | −72% | MiRL | is | more data-efficient | | | than | TMax-4B. |
| --- | ------------- | --- | --- | --- | --- | ---- | ---- | --- | ------------------- | --- | --- | ---- | -------- |
919M → 262M
| | Environments | | | | | | We | measure | data efficiency | | by | the training | re- |
| --- | ------------ | --- | --- | --- | --- | --- | --- | ------- | --------------- | --- | --- | ------------ | --- |
−15%
| | 2,370 → 2,017 | | | | | | sourcesneededtoreachagivenperformancelevel, | | | | | | |
| --- | ------------- | --- | --- | --- | --- | --- | ------------------------------------------- | --- | --- | --- | --- | --- | --- |
GPU hours
| | | | | | | −61% | includingtheresourcesspentonerrorprofilingand | | | | | | |
| --- | --- | --- | --- | --- | --- | ---- | --------------------------------------------- | --- | --- | --- | --- | --- | --- |
526.6 → 202.8
API fee ($) +$737.1 otherpreprocessing. Wecounttheseresourcesup
$0 → $737.1
| | | | | | | | to | each recipe’s | best | checkpoint | | rather | than over |
| --- | -------------- | --- | --- | --- | --- | ---- | --- | ------------- | ---- | ---------- | --- | ------ | --------- |
| | Total cost ($) | | | | | −44% | | | | | | | |
$3,521.5 → $1,960.3 itsfulltrainingrun. Figure9comparesMiRLand
0% 25% 50% 75% 100% TMax-4Buptotheirbestcheckpoints. MiRLuses
MiRL cost as % of TMax-4B
fewerterminalenvironments,andonly35%ofthe
Figure9: TrainingresourcesofMiRLrelativetoTMax-
rolloutsand34%oftheinputtokensofTMax-4B.
4B,uptothebestcheckpointratherthanthefulltraining
Itsoutputtokensdropfurther,to28%,becauseeach
run. Checkeredsegmentsareerrorprofilingandsolid
segmentsareRLtraining. GPUtimeisinA100hours. rolloutstartsfromaprefix,sothepolicydoesnot
| | | | | | | | generatethestepsoftheprefixagain. | | | | | Asaresult, | |
| --- | --- | --- | --- | --- | --- | --- | --------------------------------- | --- | --- | --- | --- | ---------- | --- |
MiRLusesonly39%oftheGPUhoursofTMax-
| atk | = 16. | Incontrast,MiRLoutperformsthebase | | | | | | | | | | | |
| --- | ----- | --------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
4B.AlthoughMiRLpaysamodelAPIfeeforthe
| modelateveryk | | | upto16byalargemargin, | | | and | | | | | | | |
| ------------- | --- | --- | --------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
LLMsinerrorprofilingandforthebehaviorjudge
| reaches39.3%atk | | | = | 16. Figure8(b)showshow | | | | | | | | | |
| --------------- | --- | --- | --- | ---------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
inRLtraining,whichTMax-4Bdoesnot,itstotal
| ofteneachtaskissolvedoverthe16runs. | | | | | | Thesere- | | | | | | | |
| ----------------------------------- | --- | --- | --- | --- | --- | -------- | --- | --- | --- | --- | --- | --- | --- |
costisstilllower,at56%ofthatofTMax-4B,while
sultssuggestthaterror-targetedRLraisestheupper
itscores4.4pointshigheronTB-Liteandmatches
| bound | of | the model | more | than standard | | RL does. | | | | | | | |
| ----- | --- | --------- | ---- | ------------- | --- | -------- | ----- | ------- | -------- | ---- | --------- | --- | ---- |
| | | | | | | | it on | TB-2.1. | Compared | with | TMax-189, | | MiRL |
WesuspectonereasonisthatMiRLkeepslearning
usesaboutasmanyrolloutsandGPUhours(26.2K
| in | a zone | of proximal | | development. | By | starting | | | | | | | |
| --- | ------ | ----------- | --- | ------------ | --- | -------- | --- | --- | --- | --- | --- | --- | --- |
vs.26.4Krolloutsand202.8vs.202.7hours)but
rolloutsatmisbehaviorprefixes,MiRLonlyneeds
scores8.4pointshigheronTB-Lite.
thepolicytocompletetherestofeachtask,andthe
misbehaviorrateattheseprefixesismostlymoder- MiRLachievesthebestperformanceatahigher
ate(Figure7). Priorworkshowsthatstartingfrom trainingcostthanPivotRLandRaR+. PivotRL
partial solutions (Li et al., 2026a) and moderate and RaR+ use $999 and $770 in total cost, com-
taskdifficulty(Zhangetal.,2026a)improveboth pared with $1,960 for MiRL (Figure 10). One
| pass@1andpass@k. | | | | | | | reasonisthattherewardofMiRL,r | | | | | ∧r | ,iseven |
| ---------------- | --- | --- | --- | --- | --- | --- | ----------------------------- | --- | --- | --- | --- | --- | ------- |
o e
8
MiRL (ours) TMax-4B Qwen3.5-4B
TB-Lite
55 51.8±0.1
50 48.0±1.7 52.4±0.5
45 42.0±2.1 47.4±0.9
40
35
32.8±1.5
30
47 94 189 378
0.25× 0.5× 1× 2×
)%(
erocS
TB-2.1
22.1±0.4
22 21.3±0.7
20 19.1±1.1 20.6±1.3
18
16.1±2.1
16.1±1.0
16
14
47 94 189 378
0.25× 0.5× 1× 2×
)%(
erocS
(a) Mean standard error vs. seeds (dashed: ∝ 1/√k)
±1.40 ±1.16 ±0.87 ±0.62
2 4 8 16
Number of seeds k (log scale)
TMax-4B RaR+ MiRL (ours)
(b) Mean score ± SE on TB-2.1
3 seeds 16 seeds
20.6±1.3 20.22±0.73
19.1±2.6 19.66±0.62
# MiRL training environments (log scale)
21.3±0.7 21.14±0.53
Figure 11: MiRL trained on different numbers of en-
10 15 20 10 15 20
vironmentsselectedfromerrorprofiling(0.25×,0.5×, Score (%)
1×,and2×). Mean(%)andstandarderrorover3runs; (c) Distribution of the 16 seeds
horizontallinesshowTMax-4BandQwen3.5-4B. min 25% mean 75% max
sparserthantheoutcomerewardofstandardRLVR,
while PivotRL and RaR+ use judge scores that
maketheirrewardsdenser. Asaresult, 42.2%of
14 18 22 26
the groups of MiRL are dead, i.e., all rollouts in Score (%)
thegroupgetzerorewardandgivenolearningsig- Figure 12: TB-2.1 scores of TMax-4B, RaR+, and
nal, compared with 27.3% for TMax-189, 25.3% MiRL with 16 extra seeds. (a) Mean standard error
overkseeds,averagedoverallk-seedsubsetsofthe16
forPivotRL,and0.13%forRaR+. Thedenserre-
seedsandoverthethreerecipes. (b)Meanandstandard
wardsalsoletbothbaselinesconvergefaster: RaR+
erroroverthe3runsinFigure5andoverthe16extra
reachesitsbestcheckpointatstep80of160,com-
seeds. (c)Distributionofthe16seeds: whiskersshow
paredwithstep100forMiRL,andPivotRLatstep theminimumandmaximum,boxesthe25thand75th
595 of 1,168. However, their scores stay at least percentiles,andcirclesthemeans.
4.2 points below MiRL on TB-Lite. Appendix C
tra seeds for each of the three recipes on TB-2.1.
detailshowwecomputethetrainingresources.
Asthenumberofseedskgrows,themeanstandard
√
MiRLimproveswithmoreenvironments. To error shrinks at about the rate of 1/ k, down to
testhowMiRLscaleswiththenumberofenviron- 0.62 at 16 seeds (Figure 12(a)). With 16 seeds,
ments,wetrainitondifferentnumberofenviron- MiRL still has the highest mean score: 21.14%,
ments,0.25×and0.5×,wherethesmallersubset vs. 20.22% for TMax-4B and 19.66% for RaR+
ispartofthelargerone,withthesamenumberof (Figure 12(b)). Figure 12(c) shows the score dis-
trainingsteps, groupsize, batchsize, andreward. tribution over the 16 seeds. These results further
For2×,wesamplemoremisbehaviorprefixesfrom supporttherankinginFigure5.
the same error profiling and keep those that pass
5 Conclusion
the rerun check, which gives 378 environments,
including the original 189. The score rises with Inthiswork,weapplyerror-targetedlearningtoter-
thenumberofenvironments,from42.0%to52.4% minalagents. We(i)proposeerrorprofiling,which
on TB-Lite and from 16.1% to 22.1% on TB-2.1 locatesandcorrectstheerrorstepsofapolicyand
(Figure11). Halfoftheenvironmentsareenough groups them into error types to find its general-
forMiRLtomatchTMax-4BonTB-Lite(48.0% izable misbehaviors, (ii) introduce MiRL, an RL
vs.47.4%). recipethatstartsrolloutsfrommisbehaviorprefixes
andrewardsarolloutonlyifitpassesthetestsand
Reliable evaluation with more runs. Long-
avoidsthemisbehavior,and(iii)showthatMiRL
horizon agents get different scores across evalu-
reducesthetargetedmisbehaviors,outperformsor
ationruns(Yaoetal.,2025),andTB-2.1hasonly
matchesallbaselinesontwobenchmarks,anduses
89tasks,whichaddsmorenoise(Miller,2024). So
fewertrainingresourcesthanTMax-4B.Thesere-
themeanover3runs,whichmanyterminal-agent
sultssuggestthaterror-targetedlearningcanwork
papersreport(Ivisonetal.,2026;Mengetal.,2026;
forlong-horizonterminaltaskswhentrainingtar-
Zhuetal.,2026a;Yangetal.,2026),issubjectto
getsgeneralizablemisbehaviorswheretheyoccur.
considerablerandomness. Wethereforerun16ex-
9
Limitations
| | | | | | | | agenttraining. | | InAdvancesinNeuralInformation | | | | |
| --- | --- | --- | --- | --- | --- | --- | -------------- | --- | ----------------------------- | --- | --- | --- | --- |
ProcessingSystems.
First,wetrainandevaluateourmethodwithonly
one model backbone, so it remains unclear how KanishkGandhi,ShivamGarg,NoahD.Goodman,and
well the gains generalize to other model choices. Dimitris Papailiopoulos. 2026. Endless terminals:
| | | | | | | | ScalingRLenvironmentsforterminalagents. | | | | | | arXiv |
| --- | --- | --- | --- | --- | --- | --- | --------------------------------------- | --- | --- | --- | --- | --- | ----- |
Second,duetothecomputebudget,wetraineach
preprintarXiv:2601.16443.
| recipe once | and | report | the | standard | error | over 3 | | | | | | | |
| ----------- | --- | ------ | --- | -------- | ----- | ------ | --- | --- | --- | --- | --- | --- | --- |
https://ai.googl
evaluation runs, which does not capture the vari- Google.2026. Gemini3.8Flash.
ance across training runs. Third, error profiling e.dev/gemini-api/docs/models/gemini-3.8-f
lash.
| relies on | external | LLMs, | | including | commercial | | | | | | | | |
| ----------------- | -------- | ------------- | ----- | --------- | ---------- | -------- | -------------- | --- | ------- | ----- | ------ | ---- | ------ |
| ones, and | the | rerun | check | discards | prefixes | from | | | | | | | |
| | | | | | | | Anisha Gunjal, | | Anthony | Wang, | Elaine | Lau, | Vaskar |
| non-deterministic | | environments, | | | which | may bias | | | | | | | |
Nath,YunzhongHe,BingLiu,andSeanM.Hendryx.
the training data. Finally, for reward design, we 2026. Rubricsasrewards: Reinforcementlearning
| | | | | | | | beyondverifiabledomains. | | | | InInternationalConfer- | | |
| --- | --- | --- | --- | --- | --- | --- | ------------------------ | --- | --- | --- | ---------------------- | --- | --- |
mainlytrytheconjunctionofthetworewardsand
havenotexploredotherwaystocombinethem. enceonLearningRepresentations.
AudreyHuang,AdamBlock,DylanJ.Foster,DhruvRo-
Acknowledgements
hatgi,CyrilZhang,MaxSimchowitz,JordanT.Ash,
| | | | | | | | andAkshayKrishnamurthy.2025. | | | | | Self-improvement | |
| -------------------------------------- | ------------ | --------- | --- | ---------- | --- | ------ | ---------------------------- | --- | ---------- | ----------------------- | -------- | ---------------- | ----------- |
| This work | was | supported | | by NSF-AI | | Engage | | | | | | | |
| | | | | | | | inlanguagemodels: | | | Thesharpeningmechanism. | | | In |
| Institute | DRL-2112635, | | | NSF-CAREER | | Award | | | | | | | |
| | | | | | | | International | | Conference | on | Learning | | Representa- |
| 1846185,andNDSEGPhDFellowship(AirForce | | | | | | | tions. | | | | | | |
OfficeofScientificResearchunderawardnumber
| | | | | | | | Hamish Ivison, | | Junjie | Oscar | Yin, Rulin | | Shao, Teng |
| ------------------ | --- | --- | ----------------------- | --- | --- | --- | -------------- | ------ | -------- | ----- | ---------- | --- | ----------- |
| FA9550-25-C-B010). | | | Theviewscontainedinthis | | | | | | | | | | |
| | | | | | | | Xiao, | Nathan | Lambert, | and | Hannaneh | | Hajishirzi. |
articlearethoseoftheauthorsandnotofthefund-
| | | | | | | | 2026. | TMax: | A simple | recipe | for | terminal | agents. |
| ---------- | --- | --- | --- | --- | --- | --- | ------------------------------ | ----- | -------- | ------ | --- | -------- | ------- |
| ingagency. | | | | | | | arXivpreprintarXiv:2606.23321. | | | | | | |
ZaidKhan,EliasStengel-Eskin,JaeminCho,andMohit
| References | | | | | | | Bansal.2025. | | DataEnvGym: | | Datagenerationagents | | |
| ---------- | --- | --- | --- | --- | --- | --- | ----------------------------------------- | --- | ----------- | --- | -------------------- | --- | ----- |
| | | | | | | | inteacherenvironmentswithstudentfeedback. | | | | | | InIn- |
Anthropic.2026. ClaudeOpus5systemcard. https: ternationalConferenceonLearningRepresentations.
//www.anthropic.com/claude-opus-5-syste
m-card.
| | | | | | | | Nathan Lambert, | | Jacob | Morrison, | Valentina | | Pyatkin, |
| ------- | -------- | ----- | ----- | ----- | ---------- | ----- | --------------- | ------ | ----------- | --------- | --------- | ---------- | -------- |
| | | | | | | | Shengyi | Huang, | Hamish | | Ivison, | Faeze | Brahman, |
| Yi Bin, | Xiaoyang | Yuan, | Haoxi | Zeng, | Wencheng | Ye, | | | | | | | |
| | | | | | | | Lester | James | V. Miranda, | | Alisa | Liu, Nouha | Dziri, |
| Wenqi | Shao, | Chen | Qian, | Wei | Ye, Yujuan | Ding, | | | | | | | |
ShaneLyu,YulingGu,SaumyaMalik,VictoriaGraf,
ZhengWang,PengpengZeng,JingkuanSong,and
| | | | | | | | Jena D. | Hwang, | Jiangjiang | | Yang, | Ronan | Le Bras, |
| ---- | --------- | ----- | -------- | --- | ------- | -------- | ------- | ------ | ---------- | --- | ----- | ----- | -------- |
| Heng | Tao Shen. | 2026. | Terminal | | agents: | A survey | | | | | | | |
OyvindTafjord,ChrisWilhelm,LucaSoldaini,and
ofAIagentsincommand-lineenvironments. arXiv 4 others. 2025. Tülu 3: Pushing frontiers in open
preprintarXiv:2608.20485. languagemodelpost-training. InConferenceonLan-
guageModeling.
MertCemri,MelissaZ.Pan,ShuyiYang,LakshyaA.
| Agrawal, | Bhavya | | Chopra, | Rishabh | Tiwari, | Kurt | | | | | | | |
| -------- | ------ | --- | ------- | ------- | ------- | ---- | -------- | ------------- | --- | ------------ | --- | ------ | ---- |
| | | | | | | | Nicholas | Lee, Thanakul | | Wattanawong, | | Sehoon | Kim, |
Keutzer,AdityaParameswaran,DanKlein,Kannan
| | | | | | | | Karttikeya | Mangalam, | | Sheng | Shen, | Gopala | Anu- |
| ------------ | --- | ------------- | --- | ----------------- | --- | --- | ---------- | --------- | --- | ----- | ----- | ------ | ---- |
| Ramchandran, | | MateiZaharia, | | JosephE.Gonzalez, | | | | | | | | | |
manchipalli,MichaelW.Mahoney,KurtKeutzer,and
| and Ion | Stoica. | 2025. | Why | do | multi-agent | LLM | | | | | | | |
| ------- | ------- | ----------- | --- | --------- | ----------- | --- | ---------------------------------- | --- | --- | -------------------- | --- | ------------ | --- |
| | | | | | | | AmirGholami.2024. | | | LLM2LLM:BoostingLLMs | | | |
| systems | fail? | In Advances | | in Neural | Information | | | | | | | | |
| | | | | | | | withnoveliterativedataenhancement. | | | | | InFindingsof | |
ProcessingSystems.
| | | | | | | | theAssociationforComputationalLinguistics: | | | | | | ACL |
| --- | --- | --- | --- | --- | --- | --- | ------------------------------------------ | --- | --- | --- | --- | --- | --- |
2024,pages6498–6526.AssociationforComputa-
MarkChen,JerryTworek,HeewooJun,QimingYuan,
| etal.2021. | Evaluatinglargelanguagemodelstrained | | | | | | tionalLinguistics. | | | | | | |
| ---------- | ------------------------------------ | --- | --- | --- | --- | --- | ------------------ | --- | --- | --- | --- | --- | --- |
| oncode. | arXivpreprintarXiv:2107.03374. | | | | | | | | | | | | |
JiazhengLi,HongzhouLin,HongLu,KaiyueWen,Zai-
wenYang,JiaxuanGao,YiWu,andJingzhaoZhang.
ZihaoCheng,HongruWang,ZemingLiu,XinyiWang,
XiangrongZhu,YuhangGuo,WeiLin,JeffZ.Pan, 2026a. QuestA: Expanding reasoning capacity in
andYunhongWang.2026. Terminal-World: Scaling LLMsviaquestionaugmentation. InInternational
terminal-agentenvironmentsviaagentskills. arXiv ConferenceonLearningRepresentations.
preprintarXiv:2605.20876.
| | | | | | | | Xuefeng | Li, Haoyang | | Zou, | and Pengfei | | Liu. 2025. |
| --- | --- | --- | --- | --- | --- | --- | ------- | ----------- | --- | ---- | ----------- | --- | ---------- |
LangFeng,ZhenghaiXue,TingcongLiu,andBoAn. LIMR:LessismoreforRLscaling. arXivpreprint
| 2025. | Group-in-grouppolicyoptimizationforLLM | | | | | | arXiv:2502.11886. | | | | | | |
| ----- | -------------------------------------- | --- | --- | --- | --- | --- | ----------------- | --- | --- | --- | --- | --- | --- |
10
ZhongzhiLi,YuchengShi,ZongxiaLi,RuhanWang, Khazi,EmmanouilKoukoumidis,XiangyiLi,Hange
AnhaoLi,ZixunHuang,JunyaoYang,LeiKe,Ning- Liu, and 31 others. 2026. OpenThoughts-Agent:
haoLiu,HaitaoMi,andLeoweiLiang.2026b. Recur- Data recipes for agentic models. arXiv preprint
| sivesynthesisforlong-horizonterminaltasks. | | | arXiv | arXiv:2606.24855. | | | | |
| ------------------------------------------ | --- | --- | ----- | ----------------- | --- | --- | --- | --- |
preprintarXiv:2608.05466.
| | | | | Zhihong Shao, | Peiyi Wang, | Qihao | Zhu, | Runxin Xu, |
| --- | --- | --- | --- | ------------- | ----------- | ----- | ---- | ---------- |
JunxiaoSong,XiaoBi,HaoweiZhang,Mingchuan
HunterLightman,VineetKosaraju,YuraBurda,Harri
Edwards,BowenBaker,TeddyLee,JanLeike,John Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024.
Schulman, Ilya Sutskever, and Karl Cobbe. 2024. DeepSeekMath: Pushingthelimitsofmathematical
Let’sverifystepbystep. InInternationalConference reasoninginopenlanguagemodels. arXivpreprint
| onLearningRepresentations. | | | | arXiv:2402.03300. | | | | |
| -------------------------- | --- | --- | --- | ----------------- | --- | --- | --- | --- |
Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Qijia Shen, Zhiqi Huang, Vamsidhar Kamanuru, Az-
naurAliev,JayRainton,AhmedAwelkair,Zhichen
ZhenpengChen,LingmingZhang,andYilingLou.
2026. Largelanguagemodel-basedagentsforsoft- Zeng,JiajunLi,ShiDong,YuemingYuan,Boyuan
wareengineering: Asurvey. ACMTransactionson Ma, Qizheng Zhang, Jiwei Fu, Yuzhen Mao, Wen-
SoftwareEngineeringandMethodology. dong Fan, PingNie, PhilipTorr, BernardGhanem,
| | | | | Changran | Hu, and 3 others. | | 2026. SETA: | Scaling |
| --- | --- | --- | --- | -------- | ----------------- | --- | ----------- | ------- |
FanzheMeng,GuoxinChen,JialeZhao,ShuangSun, environments for terminal agents. arXiv preprint
| ZhiyuLin,WayneXinZhao,RuihuaSong,Ji-Rong | | | | arXiv:2607.10891. | | | | |
| ---------------------------------------- | -------------- | ----------- | ----------- | ----------------- | --- | --- | --- | --- |
| Wen, and | Kai Jia. 2026. | CalibForge: | Adversarial | | | | | |
YipingWang,QingYang,ZhiyuanZeng,LiliangRen,
solvercalibrationforscalinglearnableterminaltasks.
arXivpreprintarXiv:2608.06352. Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He,
KuanWang,JianfengGao,WeizhuChen,Shuohang
Mike A. Merrill, Alexander G. Shaw, Nicholas Car- Wang, Simon Shaolei Du, and Yelong Shen. 2025.
lini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Reinforcement learning for reasoning in large lan-
Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly guage models with one training example. In Ad-
Buchanan,JunhongShen,GuanghaoYe,HaoweiLin,
vancesinNeuralInformationProcessingSystems.
JasonPoulos,MaoyuWang,MariannaNezhurina,Je-
| | | | | Siwei Wu, | Yizhi Li, Yuyang | Song, | Wei Zhang, | Yang |
| ----------- | ------------- | ----------------------- | --- | --------- | ---------------- | ----- | ---------- | ---- |
| nia Jitsev, | Di Lu, Orfeas | Menis Mastromichalakis, | | | | | | |
and66others.2026. Terminal-Bench: Benchmark- Wang, Riza Batista-Navarro, Xian Yang, Mingjie
ingagentsonhard,realistictasksincommandline Tang,BryanDai,JianYang,andChenghuaLin.2026.
interfaces. InInternationalConferenceonLearning Large-scale terminal agentic trajectory generation
| | | | | from Dockerized | environments. | | In International | |
| --- | --- | --- | --- | --------------- | ------------- | --- | ---------------- | --- |
Representations.
ConferenceonMachineLearning.
| EvanMiller.2024. | Addingerrorbarstoevals: | | Astatis- | | | | | |
| ---------------- | ----------------------- | --- | -------- | --- | --- | --- | --- | --- |
ZhihengXi,WenxiangChen,BoyangHong,SenjieJin,
| ticalapproachtolanguagemodelevaluations. | | | arXiv | | | | | |
| ---------------------------------------- | --- | --- | ----- | --- | --- | --- | --- | --- |
preprintarXiv:2411.00640. RuiZheng,WeiHe,YiwenDing,ShichunLiu,Xin
| | | | | Guo, Junzhe | Wang, Honglin | | Guo, Wei | Shen, Xi- |
| --- | --- | --- | --- | ----------- | ------------- | --- | -------- | --------- |
OpenThoughts-Agentteam,SnorkelAI,andBespoke aoran Fan, Yuhao Zhou, Shihan Dou, Xiao Wang,
Labs.2026. OpenThoughts-TBLite: Ahigh-signal XinboZhang,PengSun,TaoGui,and2others.2024.
benchmarkforiteratingonterminalagents. https: Traininglargelanguagemodelsforreasoningthrough
//www.openthoughts.ai/blog/openthoughts-t reversecurriculumreinforcementlearning. InInter-
blite.
nationalConferenceonMachineLearning.
RenjiePi,GraceLam,MohammadShoeybi,PooyaJan- Sidi Yang, Chaofan Tao, Jierun Chen, Tiezheng Yu,
naty,BryanCatanzaro,andWeiPing.2026. Ondata Ruoyu Wang, Yuxin Jiang, Yiming Du, Wendong
engineeringforscalingLLMterminalcapabilities. In Xu,JingXiong,TaiqiangWu,LifengShang,Xiaohui
AdvancesinNeuralInformationProcessingSystems. Li,NgaiWong,andHaoliBai.2026. Whatmakes
interactiontrajectorieseffectivefortrainingterminal
PenghuiQi,XiangxinZhou,ZichenLiu,TianyuPang,
| | | | | agents? | arXivpreprintarXiv:2606.03461. | | | |
| -------------------------------- | --- | --- | -------- | ------- | ------------------------------ | --- | --- | --- |
| ChaoDu,MinLin,andWeeSunLee.2026. | | | Rethink- | | | | | |
ShunyuYao,NoahShinn,PedramRazavi,andKarthik
ingthetrustregioninLLMreinforcementlearning.
InInternationalConferenceonMachineLearning. Narasimhan.2025. τ-bench: Abenchmarkfortool-
| | | | | agent-userinteractioninreal-worlddomains. | | | | InIn- |
| --- | --- | --- | --- | ----------------------------------------- | --- | --- | --- | ----- |
https://huggingf
QwenTeam.2026a. Qwen3.5-4B. ternationalConferenceonLearningRepresentations.
| ace.co/Qwen/Qwen3.5-4B. | | Modelcard. | | | | | | |
| ----------------------- | --- | ---------- | --- | --- | --- | --- | --- | --- |
JunkeunYi,DamonMosk-Aoyama,BaiheHuang,Ritu
QwenTeam.2026b. Qwen3.8-27B. https://huggin Gala,CharlesWang,SugamDipakDevare,Khushi
gface.co/Qwen/Qwen3.8-27B. Modelcard. Bhardwaj,AbhibhaGupta,OleksiiKuchaiev,Jiantao
| | | | | Jiao,JianZhang,andVenkatSrinivasan.2026. | | | | Piv- |
| ------------ | --------------- | -------- | ---------- | ---------------------------------------- | -------- | ------- | ------------- | ------ |
| Negin Raoof, | Richard Zhuang, | Marianna | Nezhurina, | | | | | |
| | | | | otRL: High | accuracy | agentic | post-training | at low |
EtashGuha,AtulaTejaswi,RyanMarten,CharlieF.
| | | | | computecost. | arXivpreprintarXiv:2603.21383. | | | |
| --- | --- | --- | --- | ------------ | ------------------------------ | --- | --- | --- |
Ruan,TylerGriggs,AlexanderGlennShaw,Hritik
Bansal,E.KellyBuchanan,ArtemGazizov,Reinhard YangYue,ZhiqiChen,RuiLu,AndrewZhao,Zhaokai
Heckel, Chinmay Hegde, Sankalp Jajee, Daanish Wang,YangYue,ShijiSong,andGaoHuang.2025.
11
Doesreinforcementlearningreallyincentivizerea- Errorlocalizer(cont.) Errorprofiling
| soningcapacityinLLMsbeyondthebasemodel? | | | | | | In | | | | | | | |
| --------------------------------------- | --- | --- | --- | --- | --- | --- | ---------- | ----------- | --- | -------------- | --- | -------- | ----- |
| | | | | | | | the chance | of failure, | and | an experienced | | engineer | would |
AdvancesinNeuralInformationProcessingSystems.
| | | | | | | | not do it. | Examples | of bad-but-working | | steps: | | |
| --- | --- | --- | --- | --- | --- | --- | ---------- | -------- | ------------------ | ---------- | ------- | --------- | --- |
| | | | | | | | acting on | data it | has not | inspected; | writing | one large | |
CharlieZhang,GrahamNeubig,andXiangYue.2026a. script that does everything at once instead of building up
On the interplay of pre-training, mid-training, and in checkable pieces; declaring work finished without
RLonreasoninglanguagemodels. InInternational verifying the actual output. A step that cannot work from
| | | | | | | | this state | is also | bad. | | | | |
| --- | --- | --- | --- | --- | --- | --- | ---------- | ------- | ---- | --- | --- | --- | --- |
ConferenceonMachineLearning.
| | | | | | | | Do not penalise | a | step for | being slow | or for | taking | a |
| --- | --- | --- | --- | --- | --- | --- | --------------- | -------- | -------- | ---------- | --------- | ------ | --- |
| | | | | | | | different | approach | than you | would. | Only flag | it if | it |
GuibinZhang,JunhaoWang,JunjieChen,Wangchun-
| | | | | | | | makes failure | more | likely. | | | | |
| ----------- | --- | ----- | ----------- | --------- | ---- | ------- | ----------------- | ---------- | ------- | --------- | ----- | -------- | --- |
| shu Zhou, | Kun | Wang, | and | Shuicheng | Yan. | 2026b. | | | | | | | |
| | | | | | | | A step here | is a short | <think> | reasoning | block | followed | by |
| AgenTracer: | | Who | is inducing | failure | in | the LLM | ONE bash command. | | | | | | |
agentic systems? In International Conference on Keep your reasoning SHORT: at most 150 words of thinking
LearningRepresentations. before you answer. Do not re-derive the whole task. Judge
the one step.
Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Reply in this JSON form, nothing else. Fill the fields IN
| | | | | | | | ORDER -- the | justification | | comes before | the | verdict, | so |
| -------- | ------------- | -------- | ---------- | ------ | ---------- | ------- | -------------- | ------------- | ----------- | ------------ | ------ | --------- | --- |
| Zhiguang | Han, | Jingyang | | Zhang, | Beibin | Li, Chi | | | | | | | |
| | | | | | | | you must state | what | the command | does | before | you judge | it. |
| Wang, | HuazhengWang, | | YiranChen, | | andQingyun | | | | | | | | |
{
| Wu. 2025. | | Which | agent | causes | task failures | and | | | | | | | |
| ----------- | --- | --------- | ---------------- | ----------- | ------------- | ------ | --------------- | ------------- | ----------- | --------- | ------ | -------- | ------- |
| | | | | | | | "command_seen": | | "<one short | line | naming | what the | command |
| when? | On | automated | failure | attribution | | of LLM | | | | | | | |
| | | | | | | | does, e.g. | 'cat | on a binary | PDF'>", | | | |
| | | | | | | | "reason": | "<one | sentence. | REQUIRED. | Never | null, | never |
| multi-agent | | systems. | In International | | Conference | | | | | | | | |
| | | | | | | | empty, on | good verdicts | | too.>", | | | |
onMachineLearning.
| | | | | | | | "verdict": | "good" | or "bad" | | | | |
| --- | --- | --- | --- | --- | --- | --- | ---------- | ------ | -------- | --- | --- | --- | --- |
}
RosieZhao,AlexandruMeterez,ShamM.Kakade,Cen-
| | | | | | | | Before you | answer, | check "command_seen" | | against | your | own |
| --- | --- | --- | --- | --- | --- | --- | ---------- | ------- | -------------------- | --- | ------- | ---- | --- |
gizPehlevan,SamyJelassi,andEranMalach.2025. reason: if the command you described is one you would
Echochamber: RLpost-trainingamplifiesbehaviors reject, the verdict is "bad".
| learnedinpretraining. | | | InConferenceonLanguage | | | | === WHAT THE | AGENT | SAW === | | | | |
| ---------------------------------- | --------- | ----- | ---------------------- | ------ | ------ | -------- | ------------ | ----- | ------- | --- | --- | -------------- | --- |
| Modeling. | | | | | | | {prefix} | | | | | | |
| | | | | | | | === WHAT THE | AGENT | DID === | | | | |
| Kaijie Zhu, | Yuzhou | | Nie, Yijiang | Li, | Yiming | Huang, | {step} | | | | | | |
| Jialian | Wu, | Jiang | Liu, | Ximeng | Sun, | Zhenfei | | | | | | | |
| Yin, | Lun Wang, | | Zicheng | Liu, | Emad | Barsoum, | | | | | | | |
| | | | | | | | Teacher | | | | | Errorprofiling | |
| WilliamYangWang,andWenboGuo.2026a. | | | | | | Ter- | | | | | | | |
miGen: High-fidelityenvironmentandrobusttrajec- You are an expert engineer working on a command-line task.
| tory synthesis | | for | terminal | agents. | arXiv | preprint | | | | | | | |
| -------------- | --- | --- | -------- | ------- | ----- | -------- | ---------------- | --- | ----- | ------- | --------- | --------- | --- |
| | | | | | | | Below is EXACTLY | the | state | so far: | the task, | and every | |
arXiv:2602.07274. command already run with its output. Write the CORRECT
next step yourself.
KunlunZhu,MuxinTian,BingxuanLi,ZijiaLiu,Yingx- Write a short <think> block explaining your reasoning,
uanYang,JiaxunZhang,PengruiHan,QipengXie, then call the `bash` tool with exactly ONE command. The
FuyangCui,WeijiaZhang,XiaotengMa,Xiaodong command may chain shell with && or ; but it is one command
string.
| Yu, Gowtham | | Ramesh, | Jialian | Wu, | Robert | Tang, | | | | | | | |
| ----------- | --- | ------- | ------- | --- | ------ | ----- | ------------- | ---- | -------- | ------ | ------------- | ----- | --- |
| | | | | | | | Do not repeat | work | that the | output | above already | shows | is |
ZichengLiu,HengJi,PanLu,JamesZou,andJiax- done. Build up in small checkable pieces rather than one
uanYou.2026b. AgentDebug: WhereLLMagents large script. Do not declare the task finished without
| failandhowtheycanlearnfromfailures. | | | | | | InICML | verifying | the real | output. | | | | |
| -------------------------------------- | --- | --- | --- | --- | --- | ------ | ------------- | -------- | ------- | --- | --- | --- | --- |
| 2026WorkshoponFailureModesinAgenticAI. | | | | | | | === THE STATE | SO FAR | === | | | | |
{prefix}
A ConfigurationDetails
| | | | | | | | Annotator:proposingerrortypes | | | | | Errorprofiling | |
| --- | --- | --- | --- | --- | --- | --- | ----------------------------- | --- | --- | --- | --- | -------------- | --- |
Table1liststhehyperparametersofallrecipes.
The boxes below give the prompts of all LLM You are analysing failures of a terminal agent: a language
roles. Text in braces, such as {prefix}, is filled model that solves tasks by running shell commands. A
| | | | | | | | stronger model | reviewed | every | step | the agent | took | and |
| --- | --- | --- | --- | --- | --- | --- | -------------- | -------- | ----- | ---- | --------- | ---- | --- |
inatruntime. TheRaR+promptsadaptthoseof marked some of them bad. Below is a random sample of those
bad steps.
RaR(Gunjaletal.,2026),andthePivotRLjudge
| | | | | | | | Your job is | to find | the recurring | | ways this | agent | fails. |
| --- | --- | --- | --- | --- | --- | --- | ----------- | ------- | ------------- | --- | --------- | ----- | ------ |
promptistakenunchangedfromNeMo-Gym. The list has to be useful for training, not just
descriptive.
| Errorlocalizer | | | | | | | What makes | a good | type | | | | |
| -------------- | --- | --- | --- | --- | --- | --- | ---------- | ------ | ---- | --- | --- | --- | --- |
Errorprofiling
| | | | | | | | A good type | passes | two tests. | | | | |
| ------------ | --------- | --- | ---------- | -------- | ------- | ---- | ----------------- | --------- | ------------ | ---------- | --- | ---------- | ----- |
| You are | reviewing | one | step of an | AI agent | working | on a | | | | | | | |
| | | | | | | | Test 1, transfer. | | If the agent | learned | to | avoid this | on |
| command-line | task. | | | | | | | | | | | | |
| | | | | | | | half of the | examples, | would | that carry | to | the other | half? |
Below is EXACTLY what the agent saw, and EXACTLY what it It carries when all the examples share one habit. It does
did next. You are seeing the agent's context up to this not carry when each example needs a different piece of
| point and | nothing | beyond | it. You | do not | know whether | the | | | | | | | |
| --------- | ------- | ------ | ------- | ------ | ------------ | --- | --- | --- | --- | --- | --- | --- | --- |
knowledge.
| task eventually | | succeeded. | | | | | | | | | | | |
| --------------- | --- | ---------- | --- | --- | --- | --- | ------------------ | --- | ------ | ----------- | ------ | ---- | ------ |
| | | | | | | | Test 2, frequency. | | Is the | type common | enough | that | fixing |
| Judge this | one | step. | | | | | | | | | | | |
it matters?
A step is GOOD if an experienced engineer would plausibly A type that fails test 1 is too broad. A type that fails
| do it here. | A | step is | BAD if it | still works, | but | raises | test 2 is | too narrow. | | | | | |
| ----------- | --- | ------- | --------- | ------------ | --- | ------ | --------- | ----------- | --- | --- | --- | --- | --- |
12
| Parameter | | | Value | | | Parameter | | Value | | |
| ---------------------- | --- | --- | ------------------ | --- | --- | ------------------ | --- | -------------------------- | --- | --- |
| RLtraining(allrecipes) | | | | | | MiRL | | | | |
| Algorithm | | | DPPO(Qietal.,2026) | | | Misbehaviors | | 8 | | |
| Policymodel | | | Qwen3.5-4B | | | Prefixes | | 640(80permisbehavior) | | |
| Optimizer | | | AdamW | | | Prefixesafterrerun | | 219,from189environments | | |
| Learningrate | | | 10−6,constant | | | Behaviorjudge | | Gemini3.8Flash | | |
| Groupsize | | | 16(TMax:32) | | | Stepsjudged(m) | | 5 | | |
| | | | | | | Judgeinput | | msteps,definition,examples | | |
| Batchsize | | | 8 | | | | | | | |
| Trainingsteps | | | 160(TMax:500) | | | Reward | | r o ∧r e | | |
| Samplingtemperature | | | 1.0 | | | | | | | |
TMax-189
| KLcoefficient | | | 0 | | | | | | | |
| ---------------------- | --- | --- | ---------------------- | --- | --- | --------------- | --- | ------------------ | --- | --- |
| | | | | | | Environments | | 189,random | | |
| Advantage | | | Centered | | | | | | | |
| | | | | | | Rolloutstart | | Beginningofthetask | | |
| Deadgroups | | | Filteredandresampled | | | | | | | |
| Maxresponse/turnlength | | | 65,536/16,384tokens | | | Reward | | r o | | |
| Maxtoolsteps | | | 64 | | | PivotRL | | | | |
| Bashtimeout | | | 120s | | | Pivotcandidates | | 3,892steps | | |
| Checkpointselection | | | BestTB-Liteofthetop5by | | | Actionsperpivot | | 8 | | |
smoothedreward
| | | | | | | Selectedpivots | | 1,242,from189environments | | |
| -------------------------- | --- | --- | -------------------- | --- | --- | ------------------ | --- | ----------------------------- | --- | --- |
| Errorprofiling | | | | | | Training | | 1,168one-stepupdates | | |
| | | | | | | GPUbudget | | ≈300A100hours | | |
| Environments | | | 2,017,randomfromTMax | | | | | | | |
| | | | | | | Judge | | Gemini3.8Flash | | |
| Rolloutsperenvironment | | | 2 | | | | | | | |
| Temperature | | | Policy1.0,others0.7 | | | RaR+ | | | | |
| Errorlocalizerandteacher | | | Qwen3.8-27B | | | | | | | |
| | | | | | | Rubric | | 7–20itemsperenvironment | | |
| Localizeroutput | | | Goodorbad,andareason | | | | | | | |
| | | | | | | Rubricsource | | Instruction,solution,andtests | | |
| Teacherinput | | | Theprefixonly | | | | | | | |
| | | | | | | Rubricwriter,judge | | Gemini3.8Flash | | |
| Annotator | | | ClaudeOpus5 | | | | | | | |
| | | | | | | Judgeoutput | | One1–10ratingpertrajectory | | |
| Teachertriesperflaggedstep | | | 3 | | | Reward | | wr +(1−w)(rating−1)/9 | | |
o
Namingthreshold 1%oferrorsteps Weightw Setperenvironmentbythejudge
| Evaluation | | | | | | Misbehaviorrate(§4.5) | | | | |
| ---------- | --- | --- | --- | --- | --- | --------------------- | --- | --- | --- | --- |
Benchmarks(tasks) TB-Lite(100),TB-2.1(89) Policies Qwen3.5-4B,TMax-4B,MiRL
| Runs | | | 3,meanandstandarderror | | | Misbehaviors | | 4 | | |
| ------- | --- | --- | ---------------------- | --- | --- | ------------ | --- | ---------------------- | --- | --- |
| Harness | | | TMax(mini-SWE-agent) | | | Prefixes | | 20unseenpermisbehavior | | |
Sampling Temperature0.7,top-p0.95 Sampling Temperature0.7,top-p0.95
Maxoutputperstep 16,384tokens Judges Gemini3.8Flash,Qwen3.8-27B
| Maxsteps/timeout | | | 64/120spercommand | | | | | | | |
| ---------------- | --- | --- | ----------------- | --- | --- | --- | --- | --- | --- | --- |
Table1: Hyperparameters. TheRLsettingsfollowTMax(Ivisonetal.,2026)exceptwherenoted. Allcallsto
Gemini3.8Flashusethemediumthinkinglevel.
Annotator:proposingerrortypes(cont.) Errorprofiling Annotator:proposingerrortypes(cont.) Errorprofiling
Good: You do not have to find every type, and you do not have to
"repeats a command that just failed, unchanged" be right. Name only the types you are confident about, and
One habit: check whether anything changed before running leave the rest out. A type we miss costs nothing. A type
the same thing again. Learn it on some examples, get it that turns out to be wrong is cheap to change later.
| on the rest. | And | it happens | often. | | | | | | | |
| ------------ | ------- | ---------- | -------------- | --- | --- | ------------------ | ------------- | -------------- | -------- | -------- |
| | | | | | | You cannot | really answer | test 1 without | training | a model, |
| Too broad: | | | | | | so guess. Guessing | | is expected. | | |
| "runs a | tool on | input it | cannot accept" | | | | | | | |
Input
| Sounds general, | | but every | example | needs a | different | | | | | |
| --------------- | ------- | ------------- | ------- | -------- | ---------- | ------------ | --------- | ----------------- | --- | --------------- |
| fact. Learning | that | openssl | cannot | read a | public key | | | | | |
| | | | | | | Each example | has three | fields. | | |
| teaches | nothing | about ffmpeg. | Nothing | carries. | | | | | | |
| | | | | | | rejected | command: | the shell command | the | agent proposed. |
Too narrow:
| | | | | | | Long ones | are cut | off. | | |
| --- | --- | --- | --- | --- | --- | --------- | ------- | ---- | --- | --- |
"uses openssl x509 on a public key" what this command is doing: a one-line description of
| This one | does carry, | but | it almost | never | happens. Fixing | the command. | | | | |
| ---------- | ----------- | --- | --------- | ----- | --------------- | ------------ | ------------- | -------------- | ------------ | --- |
| it changes | nothing. | | | | | | | | | |
| | | | | | | why this | is a failure: | the reviewer's | explanation. | |
Two more rules. The third field carries most of the signal. The first two
Ignore what the task was about. The examples mention tell you what was actually run.
| numpy, sqlite, | openssl, | pydub. | That | is incidental. | Name | | | | | |
| -------------- | -------- | ------ | ---- | -------------- | ---- | --- | --- | --- | --- | --- |
<examples>
| what the agent | did | wrong. | If a type | name holds | a tool name | | | | | |
| -------------- | --- | ------ | --------- | ---------- | ----------- | --- | --- | --- | --- | --- |
{examples}
| or a file | name, it | is too | narrow. | | | </examples> | | | | |
| ---------- | ---------- | ------------ | --------- | ------- | -------------- | ------------- | ---------- | ----------- | ---- | ------------- |
| Write each | definition | so | it can be | checked | on one example | | | | | |
| | | | | | | When to leave | an example | unassigned | | |
| alone, by | a yes or | no question. | Someone | holding | your | | | | | |
| definition | and one | bad step | must be | able to | decide, | | | | | |
| | | | | | | Do not force | an example | into a type | that | does not fit. |
without seeing the rest of the task. Leave it unassigned, and say why with one of these three
| How sure you | need | to be | | | | reasons. | | | | |
| ------------ | ---- | ----- | --- | --- | --- | -------- | --- | --- | --- | --- |
"one_off"
13
Annotator:proposingerrortypes(cont.) Errorprofiling Annotator:refiningerrortypes(cont.) Errorprofiling
| The failure | is tied | to this | specific | task | or tool and | Input | | | | | |
| ----------- | ------------ | ---------- | -------- | ------ | ----------- | ------------ | --------- | --------- | ------- | --------- | --------- |
| is unlikely | to recur | elsewhere. | This | is the | long tail. | | | | | | |
| | | | | | | Each example | has three | fields. | | | |
| It is | expected and | fine. | | | | | | | | | |
| | | | | | | rejected | command: | the shell | command | the agent | proposed. |
"new_pattern"
| | | | | | | Long ones | are cut | off. | | | |
| --- | --- | --- | --- | --- | --- | --------- | ------- | ---- | --- | --- | --- |
It matches no type on your list, but it looks like what this command is doing: a one-line description of
| something | that could | recur. | You do | not have | enough | | | | | | |
| --------- | ---------- | ------ | ------ | -------- | ------ | --- | --- | --- | --- | --- | --- |
the command.
| examples | here to | name it | yet. This | tells | us to keep | | | | | | |
| -------- | ------- | ------- | --------- | ----- | ---------- | -------- | ------------- | --- | ---------- | ------------ | --- |
| | | | | | | why this | is a failure: | the | reviewer's | explanation. | |
sampling.
| | | | | | | The third | field carries | most | of the | signal. | The first two |
| --- | --- | --- | --- | --- | --- | --------- | ------------- | ---- | ------ | ------- | ------------- |
"unclear"
| | | | | | | tell you what | was | actually | run. | | |
| -------------- | ----------- | --------- | --------- | ------- | --------- | ------------- | --- | -------- | ---- | --- | --- |
| The reviewer's | explanation | | is vague, | or the | step does | | | | | | |
| not look | like a real | failure. | | | | <examples> | | | | | |
| This is | noise in | the data, | not a | failure | type. | {examples} | | | | | |
</examples>
| Choosing the | right reason | matters. | We | use these | counts to | | | | | | |
| ------------ | ------------ | -------- | --- | --------- | --------- | --- | --- | --- | --- | --- | --- |
decide whether to stop sampling, so do not use one reason When to leave an example unassigned
| as a catch-all | for the | others. | | | | | | | | | |
| -------------- | ------- | ------- | --- | --- | --- | -------------------- | --- | ------- | ------------- | ------ | ----------- |
| | | | | | | Leave it unassigned, | | and say | why with | one of | these three |
| Output | | | | | | reasons. | | | | | |
| Return JSON | only: | | | | | "one_off" | | | | | |
| | | | | | | The failure | is | tied to | this specific | task | or tool and |
{
| | | | | | | is unlikely | to | recur elsewhere. | | This is | the long tail. |
| --- | --- | --- | --- | --- | --- | ----------- | --- | ---------------- | --- | ------- | -------------- |
"types": [
| { | | | | | | It is | expected | and fine. | | | |
| ------------- | ------ | ---------- | ------- | ------------ | ----------- | ------------- | -------- | ----------- | ----- | ------ | ---------- |
| "name": | "<2 to | 5 words, | names a | behaviour>", | | "new_pattern" | | | | | |
| | | | | | | It matches | no | type on the | list, | but it | looks like |
| "definition": | "<one | paragraph. | | What the | agent does. | | | | | | |
What makes it wrong. What it is NOT, so it cannot be something that could recur. Either you have put it in
confused with the nearest other type.>", "new_types" below, or you do not have enough examples
| | | | | | | here to | name it | yet. This | tells | us to keep | sampling. |
| -------------- | --- | ---------- | ---- | ---------- | ------- | -------------- | ------- | --------------- | --------- | ---------- | ------------- |
| "example_ids": | | [<ids from | this | batch that | match>] | | | | | | |
| } | | | | | | "unclear" | | | | | |
| ], | | | | | | The reviewer's | | explanation | is vague, | or | the step does |
| "unassigned": | [ | | | | | not look | like | a real failure. | | | |
{"id": <id>, "reason": "one_off" | "new_pattern" | This is noise in the data, not a failure type.
"unclear"}
| | | | | | | Choosing the | right | reason matters. | | We use these | counts to |
| --- | --- | --- | --- | --- | --- | ------------ | ----- | --------------- | --- | ------------ | --------- |
]
| | | | | | | decide whether | to | stop sampling, | so | do not | use one reason |
| ----------- | -------------- | ------- | ----- | ------ | ------ | -------------- | --- | -------------- | --- | ------ | -------------- |
| } | | | | | | as a catch-all | for | the others. | | | |
| Every input | id must appear | exactly | once, | either | in the | | | | | | |
Output
| example_ids | of at least | one type, | or | in "unassigned". | An | | | | | | |
| ----------- | ----------- | --------- | -------------- | ---------------- | -------- | ----------- | ----- | --- | --- | --- | --- |
| example may | match more | than one | type. | | | | | | | | |
| | | | | | | Return JSON | only: | | | | |
| Propose as | many types | as the | data supports, | and | no more. | | | | | | |
{
| | | | | | | "assignments": | [ | | | | |
| ---------------------------- | ------------ | ---------- | ---------- | --------- | -------------- | -------------- | --------------- | ---------- | --------- | --------------- | --- |
| | | | | | | {"id": | <id>, "types": | ["<name>", | | ...]} | |
| Annotator:refiningerrortypes | | | | | Errorprofiling | ], | | | | | |
| | | | | | | "unassigned": | [ | | | | |
| | | | | | | {"id": | <id>, "reason": | | "one_off" | | "new_pattern" | | |
| You are analysing | failures | of | a terminal | agent: | a language | | | | | | |
| model that | solves tasks | by running | shell | commands. | A | "unclear"} | | | | | |
| stronger model | reviewed | every | step the | agent | took and | ], | | | | | |
| | | | | | | "new_types": | [ | | | | |
| marked some | of them bad. | You | already | produced | the list of | | | | | | |
| failure types | below. | Here is | a fresh | sample of | bad steps | { | | | | | |
that you have not seen. "name": "<2 to 5 words, names a behaviour>",
| | | | | | | "definition": | | "<one paragraph. | | What the | agent does. |
| ------------ | ---------- | --- | --- | --- | --- | ------------- | -------- | ---------------- | ------- | -------- | --------------- |
| Your job has | two parts. | | | | | | | | | | |
| | | | | | | What | makes it | wrong. | What it | is NOT, | so it cannot be |
1. Assign each example to the types it matches. An example confused with the nearest other type.>",
can match more than one type, or none. "example_ids": [<ids from this batch that match>]
| 2. Only if | several examples | share | a pattern | that | matches | } | | | | | |
| ----------- | ---------------- | ----- | --------- | ------- | ------- | --- | --- | --- | --- | --- | --- |
| no existing | type, propose | a new | type | for it. | Do not | ] | | | | | |
}
| propose a | type for a | single example. | | Do not restate | an | | | | | | |
| ----------------- | ------------ | --------------- | ------- | -------------- | ----------- | ------------- | ------- | ----------- | ----------- | ------------ | -------- |
| existing type | in different | words. | | | | | | | | | |
| | | | | | | Every input | id must | appear | exactly | once, either | in |
| | | | | | | "assignments" | with | a non-empty | list | of types, | or in |
| What makes | a good type | | | | | | | | | | |
| | | | | | | "unassigned". | Use | the exact | type names | from | the list |
| A good type | passes two | tests. | | | | above. Leave | | | | | |
| | | | | | | "new_types" | empty | if nothing | new recurs. | | |
| Test 1, transfer. | If | the agent | learned | to avoid | this on | | | | | | |
| half of the | examples, | would that | carry | to the | other half? | | | | | | |
| It carries | when all the | examples | share | one habit. | It does | | | | | | |
not carry when each example needs a different piece of Annotator:labelingerrorsteps Errorprofiling
| knowledge. | A type that | fails | this is | too broad. | | | | | | | |
| ---------- | ----------- | ----- | ------- | ---------- | --- | --- | --- | --- | --- | --- | --- |
Test 2, frequency. Is it common enough that fixing it A terminal agent solves tasks by running shell commands. A
| | | | | | | stronger model | checked | every | step it | took and | marked some |
| ----------- | --------- | ---------- | ------ | -------- | --------- | -------------- | ------- | ---------- | -------- | --------- | ------------- |
| matters? A | type that | fails this | is too | narrow. | | | | | | | |
| | | | | | | of them bad. | Below | is a batch | of those | bad | steps. |
| Ignore what | the task | was about. | Name | what the | agent did | | | | | | |
| | | | | | | The list of | failure | types is | fixed. | Your only | job is to put |
wrong.
| | | | | | | each error | into the | types it | matches, | or to | leave it out. |
| ------------- | --------------- | --------- | ---------- | ------- | ------------ | ------------- | -------- | -------- | -------- | ------- | ------------- |
| Be strict | when assigning. | If | an example | only | loosely fits | | | | | | |
| | | | | | | Do not invent | a type. | Do not | rename | one. Do | not widen or |
| a type, leave | it out. | A smaller | clean | list is | better than | | | | | | |
a padded one. You do not have to be right: a wrong type is narrow a definition. If an error fits nothing, leave it
| | | | | | | out. That | is the expected | outcome | for | many | of them. |
| --------------- | ------ | --- | --- | --- | --- | --------- | --------------- | ------- | --- | ---- | -------- |
| cheap to change | later. | | | | | | | | | | |
How to decide
Current types
<types> Each type below gives a definition and two real errors of
that type.
{types}
</types> Read the definition first. It ends with what the type is
14
Annotator:labelingerrorsteps(cont.) Errorprofiling RaR+:rubricwriter Baseline
| NOT, naming | the neighbour | it would | otherwise | absorb. Use | | | | | | | |
| ----------- | ------------- | -------- | --------- | ----------- | ------- | --------- | ------ | ------- | ---- | ------ | ------------- |
| | | | | | You are | an expert | rubric | writer. | Your | job is | to generate a |
those lines: most hard cases are a choice between two self-contained set of evaluation criteria ("rubrics") for
types, and the exclusions say which. judging how good a response is to a given task. The task
The examples show the usual shape of a type, not its is given to an AI agent working in a Linux terminal; the
limits. An error that matches the definition belongs to response is the record of the agent's work: the commands
the type even when it looks nothing like the examples. it ran and their outputs, in order. Rubrics can cover
An error can match more than one type. Assign every type aspects of a response such as, but not limited to, the
that fits. correct final result (the files, outputs, programs,
| | | | | | services | and | settings | the task | asks | for), meeting | every |
| ------------- | ------------- | ----------- | --------------- | ----------- | --------------- | --- | -------------- | ----------- | ------- | ------------- | ---------- |
| Be consistent | rather | than harsh. | The counts | are used to | | | | | | | |
| | | | | | requirement | | and constraint | stated | in | the task, | inspecting |
| decide how | much training | data | each type gets, | so a type | | | | | | | |
| | | | | | the environment | | and | data before | acting, | checking | the |
you under-assign looks rarer than it is. If an error result after acting, handling errors and unexpected output,
| clearly matches | a definition, | | assign it, even | when the | | | | | | | |
| --------------- | ------------- | --- | --------------- | -------- | ------- | --------- | ----------------- | --- | -------- | ------- | ---------- |
| | | | | | and not | damaging | things | the | task did | not ask | to change. |
| type is a | rare one. | | | | | | | | | | |
| | | | | | Each | item must | be self-contained | | – non | expert | readers |
Ignore what the task was about. Judge what the agent did. should not need to infer anything or consult external
| | | | | | information. | | Begin each | description | | with its | category: |
| --- | --- | --- | --- | --- | ------------ | --- | ---------- | ----------- | --- | -------- | --------- |
The types
| | | | | | "Essential | Criteria: | | ...", "Important | | Criteria: | ...", |
| --- | --- | --- | --- | --- | ---------- | --------- | --- | ---------------- | -------- | --------- | ----- |
| | | | | | "Optional | Criteria: | | ...", or | "Pitfall | Criteria: | ...". |
<types>
| {types} | | | | | Inputs: | | | | | | |
| -------- | --- | --- | --- | --- | ------- | ------------------- | ---- | --------- | ------------- | ------------- | --- |
| </types> | | | | | • | task: The | full | task text | given | to the agent. | |
| | | | | | • | reference_solution: | | The | task author's | reference: | how |
The input the environment was set up, the intended fix or answer,
| Each error | has three | fields. | | | and | how success | is | checked. | | | |
| ---------- | --------- | ----------- | ------------------- | ---- | ----- | ----------- | -------------- | -------- | -------- | ------ | ----------- |
| | | | | | • | tests: | The hidden | tests | that run | on the | final state |
| rejected | command: | the command | the agent proposed. | Long | after | the | agent submits. | | | | |
| ones are | cut off. | | | | | | | | | | |
Total items:
what this command is doing: one line saying what the • Choose 7–20 rubric items based on the complexity of
| command | does. | | | | the | task. | | | | | |
| --------- | ------------- | -------------- | ------------ | --- | ---- | ------------ | ------- | -------- | -------- | ---- | ------------ |
| why this | is a failure: | the reviewer's | explanation. | | | | | | | | |
| | | | | | Each | rubric | item: | | | | |
| | | | | | • | title (2–4 | words). | | | | |
| The third | field carries | most of | the signal. | | | | | | | | |
| | | | | | • | description: | One | sentence | starting | with | its category |
<examples>
| | | | | | prefix | that | explicitly | states | exactly | what | to look for. |
| ---------- | --- | --- | --- | --- | ------ | -------- | ---------- | ------ | ------- | ---- | ------------ |
| {examples} | | | | | For | example: | | | | | |
</examples>
| | | | | | | – | Essential | Criteria: | Writes | the flagged | entries |
| ---------- | --- | --- | --- | --- | --- | ------ | ---------------------- | --------------- | ------ | ----------- | ------------ |
| The output | | | | | | to | /home/user/alerts.txt, | | | one per | line, in the |
| | | | | | | format | the | task specifies. | | | |
Return JSON only. – Important Criteria: Checks that the service
| { | | | | | | answers | on | the requested | port | after | the last |
| -------------- | --- | --- | --- | --- | --- | ------------- | --- | ------------- | ---- | ----- | -------- |
| | | | | | | configuration | | change. | | | |
| "assignments": | [ | | | | | | | | | | |
{"id": <id>, "types": ["<exact name from the list>", – Optional Criteria: Inspects the input file's
| ...]} | | | | | | format | before | writing | code | that parses | it. |
| ----- | --- | --- | --- | --- | --- | ------ | ------- | --------- | ------- | ------------- | ----- |
| | | | | | | – | Pitfall | Criteria: | Deletes | or overwrites | files |
],
"unassigned": [<id>, ...] outside the directories the task names.
| } | | | | | • | weight: | For Essential/Important/Optional, | | | | use 1–5 (5 |
| ---------------- | ------- | ----- | ----------------------- | --- | --- | ---------------- | --------------------------------- | ------------ | --- | ----- | ---------- |
| | | | | | = | most important); | | for Pitfall, | use | –1 or | –2. |
| Every id appears | exactly | once, | either in "assignments" | | | | | | | | |
Category guidance:
| with at least | one type, | or in | | | | | | | | | |
| ------------- | --------- | ----- | --- | --- | --- | ---------- | -------- | ------- | --- | -------------- | --- |
| | | | | | • | Essential: | Critical | results | or | safety checks; | if |
"unassigned". Copy type names exactly as written above. missing, the response is invalid (weight 5).
| | | | | | • | Important: | Key | steps, completeness, | | or | verification; |
| ------------- | --- | --- | --- | ---- | -------- | ---------- | ----------------- | -------------------- | -------- | ----- | ------------- |
| | | | | | strongly | | affects | quality | (weight | 3–4). | |
| Behaviorjudge | | | | MiRL | | | | | | | |
| | | | | | • | Optional: | Helpful | practice | or extra | care; | nice to |
| | | | | | have | but | not deal-breaking | | (weight | 1–2). | |
You're reviewing a short stretch of an AI agent's work on • Pitfall: Common mistakes or omissions specific to
| a command-line | task: | the first | {m} steps it | took after a | | | | | | | |
| -------------- | ----- | --------- | ------------ | ------------ | ---- | ------------- | --- | ------ | -------- | ----- | ---------- |
| | | | | | this | task—identify | | things | an agent | often | forgets or |
given point. gets wrong. Each Pitfall description must begin with
You see the task, what the agent did in these steps (its "Pitfall Criteria:" and describe the mistake, and use
| reasoning, | message, | command and | the command's | output, in | | | | | | | |
| ---------- | -------- | ----------- | ------------- | ---------- | ------ | --- | ------ | --- | --- | --- | --- |
| | | | | | weight | –1 | or –2. | | | | |
order) and one specific type of possible failure mode. To ensure self-contained guidance:
--- THE TASK --- • When the task names exact paths, formats, values or
| {task} | | | | | commands | | to provide, | name | them explicitly | | in the item |
| ------ | --- | --- | --- | --- | -------- | ---------- | ----------- | --------- | --------------- | --- | ----------- |
| | | | | | (e.g., | "Essential | | Criteria: | Creates | | |
--- WHAT THE AGENT DID NEXT ({m} steps) --- /home/user/filter.sh that reads the manifest from
{steps}
| | | | | | standard | | input"). | | | | |
| --------------- | ---- | ---------------- | ------ | --- | -------- | --------- | -------- | ---------- | ------- | -------- | --------- |
| | | | | | • | The judge | sees | only the | agent's | commands | and their |
| --- The failure | mode | we're interested | in --- | | | | | | | | |
| | | | | | outputs | (not | its | reasoning, | and not | the test | results), |
Name: {mode_name}
Definition: {mode_definition} so every item must be checkable from that record.
| | | | | | • | The reference | solution | | is only | one example | of |
| ----------- | ---------- | --- | --- | --- | -------- | ------------- | ----------- | ---------- | ------- | ----------- | ----------- |
| Examples of | this mode: | | | | | | | | | | |
| | | | | | success. | | Other valid | approaches | must | also | satisfy the |
{mode_examples}
| | | | | | rubric, | so | no item | may require | the | reference's | exact |
| --- | --- | --- | --- | --- | ------- | --- | ------- | ----------- | --- | ----------- | ----- |
--- QUESTION --- commands, tools or code; describe the required outcome
Did this named failure mode happen in any of these steps? or behaviour instead (unless the task itself requires a
| Focus only | on the named | failure | mode. | | | | | | | | |
| ---------- | ------------ | ------- | ----- | --- | -------- | --- | ------- | --------- | --- | --- | --- |
| | | | | | specific | | tool or | command). | | | |
Answer "yes" if this failure mode is present in at least • Do not write any item about the hidden tests passing
one of these steps, even if there are other issues. (or about the agent saying that tests pass): the tests
| Answer "no" | if this | specific failure | mode is | absent from | | | | | | | |
| ----------- | ------- | ---------------- | ------- | ----------- | --- | -------------- | --- | ----------- | ------- | ------------- | --- |
| | | | | | are | run separately | | and counted | through | tests_weight. | |
all of these steps: either the steps were fine, or their Items about the agent running its own checks are fine.
faults were different ones. • No item may reward a longer response, more commands
| Think it through, | then | end your | reply with | | | | | | | | |
| ----------------- | ---- | -------- | ---------- | --- | --- | ----------- | ------------- | --- | ---- | ------------------ | --- |
| | | | | | or | more steps. | tests_weight: | | Also | give tests_weight, | a |
<answer>yes</answer> or <answer>no</answer>. number from 0.5 to 1.0: the share of the reward that
| | | | | | the | hidden | tests | carry for | this task | (the | rubric score |
| --- | --- | --- | --- | --- | --- | ------ | ----- | --------- | --------- | ---- | ------------ |
15
RaR+:rubricwriter(cont.) Baseline PivotRL:step-equivalencejudge(cont.) Baseline
| carries | the rest). | Use a | higher | value when | the tests | writing code | files | | | | |
| ---------------- | ---------- | ------------ | ------ | ---------- | ----------- | ------------ | ----- | --- | --- | --- | --- |
| check everything | | that matters | in | the task, | and a lower | | | | | | |
---
| value when | important | parts | of a | good solution | are not | | | | | | |
| ---------- | ------------- | ----- | ---- | ------------- | ------- | ------------- | --------- | ----- | --- | --- | --- |
| checked | by the tests. | | | | | Step 2: Check | EXECUTION | STAGE | | | |
Output: Provide a JSON object with exactly two keys: Determine what stage each response reaches:
"tests_weight" (a number from 0.5 to 1.0) and "rubrics" (a - **Stage 1 - Explore**: Only reads/examines (ls, cat,
| JSON array | of rubric | objects). | Each | rubric object | must | | | | | | |
| ---------- | --------- | --------- | ---- | ------------- | ---- | ----------- | ----- | --- | --- | --- | --- |
| | | | | | | grep, head, | find) | | | | |
contain exactly three keys—title, description, and weight. - **Stage 2 - Write**: Creates/modifies files but doesn't
| Do not copy | large blocks | of | the task, | reference_solution | | execute | | | | | |
| ------------- | ------------ | ------ | ----------- | ------------------ | ---------- | ---------- | -------------- | ---- | ------- | ------------ | ------- |
| or tests into | the text. | Each | description | must | begin with | | | | | | |
| | | | | | | - **Stage | 3 - Execute**: | Runs | written | code/scripts | (python |
| its category | prefix, | and no | extra keys | are allowed. | | script.py, | ./script.sh) | | | | |
Now, given the task, reference_solution and tests, - **Stage 4 - Verify**: Validates output correctness after
| generate the | rubric | as described. | The | reference | solution | | | | | | |
| ------------ | ------ | ------------- | --- | --------- | -------- | --- | --- | --- | --- | --- | --- |
execution
| is an ideal | response | but not | necessarily | exhaustive; | use | | | | | | |
| ----------- | -------- | ------- | ----------- | ----------- | --- | --------- | ------- | ------------- | --- | ------------ | -------- |
| | | | | | | **RULE:** | If GOLD | and CANDIDATE | are | at DIFFERENT | stages → |
it only as guidance.
[[A!=B]]
<task> - Write-only (Stage 2) vs Write+Execute (Stage 3) = NOT
| {task} | | | | | | equivalent | | | | | |
| ------ | --- | --- | --- | --- | --- | ---------- | --- | --- | --- | --- | --- |
</task>
| | | | | | | - Explore | (Stage | 1) vs Execute | (Stage | 3) = NOT | equivalent |
| -------------------- | --- | --- | --- | --- | --- | --------- | ------ | ------------- | ------ | -------- | ---------- |
| <reference_solution> | | | | | | --- | | | | | |
{reference}
| | | | | | | Step 3: Check | ACTION | TYPE | | | |
| --- | --- | --- | --- | --- | --- | ------------- | ------ | ---- | --- | --- | --- |
</reference_solution>
| <tests> | | | | | | Compare the | PRIMARY | action | being performed: | | |
| ------- | --- | --- | --- | --- | --- | ----------------- | ------- | -------- | ---------------- | ------ | -------- |
| | | | | | | - **Diagnostic**: | | examine, | grep, cat, | debug, | profile, |
{tests}
inspect
</tests>
| | | | | | | - **Install**: | apt | install, | pip install, | npm | install |
| --- | --- | --- | --- | --- | --- | -------------- | --- | -------- | ------------ | --- | ------- |
(actually installs)
| | | | | | | - **Search**: | apt | search, pip | search, | find, locate | (only |
| ---------------- | --- | --- | --- | --- | -------- | ------------- | --- | ----------- | ------- | ------------ | ----- |
| RaR+:rubricjudge | | | | | Baseline | | | | | | |
searches)
| | | | | | | - **Modify**: | sed, | patch, edit, | fix, | write code | |
| ---------- | ----------------- | --- | ----- | -------------- | --- | --------------------- | ---- | -------------- | -------- | ------------ | ------ |
| [System] | | | | | | - **Backup/Restore**: | | cp backup, | restore, | revert | |
| You are an | expert evaluator. | | Given | a user prompt, | a | | | | | | |
| | | | | | | - **Test-edge**: | | tests specific | edge | cases, small | inputs |
generated response, and a list of quality rubrics, please - **Test-stress**: tests performance, large inputs, many
| rate the overall | quality | of | the response | on a | scale of 1 | iterations | | | | | |
| ----------------- | ------- | ------------ | ------------ | ------------ | ---------- | ---------- | --------- | ---------------- | --- | -------- | --- |
| to 10 based | on how | well it | satisfies | the rubrics. | | | | | | | |
| | | | | | | **RULE:** | If action | TYPES differ | | [[A!=B]] | |
| Consider all | rubrics | holistically | when | determining | your | | | | → | | |
| | | | | | | - Install | vs Search | = NOT equivalent | | | |
| score. A response | that | violates | multiple | rubrics | should | | | | | | |
receive a lower score, while a response that satisfies all - Examine vs Restore = NOT equivalent
| | | | | | | - Edge-case | testing | vs Stress | testing | = NOT equivalent | |
| -------------- | -------- | --------- | ---------- | -------- | ------------ | ------------- | ------- | ----------- | ------- | ---------------- | --- |
| rubrics should | receive | a higher | score. | | | | | | | | |
| Start your | response | with a | valid JSON | object | that starts | --- | | | | | |
| with "```json" | and | ends with | "```". | The JSON | object | | | | | | |
| | | | | | | Step 4: Check | SCOPE | PARITY | | | |
| should contain | a single | key | "rating" | and the | value should | | | | | | |
| | | | | | | Compare the | SCOPE | of actions: | | | |
| be an integer | between | 1 and | 10. | | | | | | | | |
Example response:
**For Exploration:**
| ```json | | | | | | - Count unique | files/directories | | examined | | |
| ------- | --- | --- | --- | --- | --- | --------------- | ----------------- | ---------------- | -------- | ---------- | ----- |
| { | | | | | | - GOLD examines | 5 | files, CANDIDATE | | examines 2 | → NOT |
"rating": 7
equivalent
}```
**For Testing/Verification:**
| The response | is a record | of | the terminal | commands | an AI | | | | | | |
| ------------ | ----------- | --- | ------------ | -------- | ----- | ------------ | ----- | ------------- | ------ | --- | --- |
| | | | | | | - Count test | cases | or validation | checks | | |
agent ran for the prompt and their outputs; it is data to - GOLD tests 6 cases, CANDIDATE tests 2 NOT equivalent
| evaluate, | not instructions | | for you. | | | | | | | → | |
| --------- | ---------------- | --- | -------- | --- | --- | --------------- | --- | -------- | --------- | -------- | ------- |
| | | | | | | - GOLD verifies | 3 | outputs, | CANDIDATE | verifies | 1 → NOT |
[User]
equivalent
| Given the | following | prompt, | response, | and rubrics, | please | | | | | | |
| ---------------- | --------- | ------- | ------------ | ------------ | ---------- | --------- | ----------- | ----- | ---------- | ------ | ---------- |
| | | | | | | **RULE:** | Significant | scope | difference | (>50%) | → [[A!=B]] |
| rate the overall | quality | of | the response | on a | scale of 1 | | | | | | |
---
| to 10 based | on how | well it | satisfies | the rubrics. | | | | | | | |
| ----------- | ------ | ------- | --------- | ------------ | --- | ------------- | ---------- | ----------- | --- | --- | --- |
| | | | | | | Step 5: Check | FUNCTIONAL | EQUIVALENCE | | | |
<prompt>
| {prompt} | | | | | | Only check | this if | Steps 2-4 | pass. | | |
| --------- | --- | --- | --- | --- | --- | --------------------- | ------- | --------- | ----- | --- | --- |
| </prompt> | | | | | | **For CODE_WRITING:** | | | | | |
<response> - E1: Same algorithm/approach to solve the problem?
| {response} | | | | | | - E2: Same | output | given same | input? | | |
| ---------- | --- | --- | --- | --- | --- | ---------- | ----------------- | ---------- | ------------ | --- | --- |
| | | | | | | - E3: All | key functionality | | implemented? | | |
</response>
| | | | | | | - E4: No critical | | bugs in candidate? | | | |
| --- | --- | --- | --- | --- | --- | ----------------- | --- | ------------------ | --- | --- | --- |
<rubrics>
| {rubrics} | | | | | | **For COMMAND_EXECUTION:** | | | | | |
| ---------- | --- | --- | --- | --- | --- | -------------------------- | --------- | --------------- | --- | --- | --- |
| </rubrics> | | | | | | - E1: Same | operation | type performed? | | | |
Your JSON Evaluation: - E2: Same functional outcome achieved?
| | | | | | | - E3: Same | purpose | in the task | workflow? | | |
| ----------------------------- | ----------- | ------- | ------------ | ------- | --------- | ------------- | ------- | --------------- | --------- | ------------ | -------- |
| | | | | | | If ALL checks | pass | [[A=B]], | otherwise | | [[A!=B]] |
| | | | | | | | | → | | → | |
| PivotRL:step-equivalencejudge | | | | | Baseline | --- | | | | | |
| | | | | | | What to ALLOW | (these | differences | are | OK) | |
| You are evaluating | | whether | two terminal | command | responses | | | | | | |
| | | | | | | - Different | command | flags achieving | | same result: | -q vs -o |
| are functionally | equivalent. | | | | | | | | | | |
for unzip
| Both GOLD | and CANDIDATE | are | JSON objects. | Focus | on | | | | | | |
| --------- | ----------------------- | --- | ------------- | ------------------ | --- | ----------- | ----- | -------- | ---------- | ------ | ---------- |
| | | | | | | - Different | tools | for same | operation: | sed vs | python for |
| comparing | the commands/keystrokes | | - | ignore differences | in | | | | | | |
same edit
reasoning or analysis text. - Different installation method if packages available:
Step 1: Identify Response Type venv vs --break-system-packages
- **CODE_WRITING**: GOLD writes code to a file (heredoc, - Different sed line ranges if same fix applied
cat > file, echo > file) - Different algorithms if output is identical
- **COMMAND_EXECUTION**: GOLD runs commands without - Different intermediate file names
16
PivotRL:step-equivalencejudge(cont.) Baseline B.2 ExamplesofMisbehaviors
- Minor cosmetic differences (variable names, comments) Figures13and14showoneerrorstepforeachmis-
**Key principle:** Focus on TASK OUTPUT and FUNCTIONAL behavior. Wechoseeachexampleforashortprefix
| OUTCOME, | not implementation | details. | | | | | | | | |
| -------- | ------------------ | -------- | --- | --- | --- | --- | --- | --- | --- | --- |
andcommand,sothattheerrorisvisiblewithout
---
| | | | | | the full | trajectory. | Each | box gives | the | definition |
| ------- | ------------------- | ----------- | --- | --- | -------- | ----------- | ---- | --------- | --- | ---------- |
| What to | REJECT (mark as not | equivalent) | | | | | | | | |
- Different execution stages (write-only vs write+execute) written by the annotator, the start of the task, the
| - Different | action types (install | vs search, | examine | vs | | | | | | |
| ----------- | --------------------- | ---------- | ------- | --- | --------- | ------ | ------- | ------- | ----------- | ----- |
| | | | | | last step | of the | prefix, | and the | error step, | whose |
restore)
- Significantly different scope (2 tests vs 6 tests) commandisshowninred. Thelastrowgivesthe
| - GOLD writes | code but CANDIDATE | doesn't | | | | | | | | |
| ------------- | ------------------ | ------- | --- | --- | --- | --- | --- | --- | --- | --- |
explanationoftheerrorlocalizer.
| - Missing | output artifacts required | by | task | | | | | | | |
| --------- | ------------------------- | --- | ---- | --- | --- | --- | --- | --- | --- | --- |
---
| Output Format | | | | | B.3 Long-TailErrorSteps | | | | | |
| ------------- | --- | --- | --- | --- | ----------------------- | --- | --- | --- | --- | --- |
1. **EXECUTION STAGE**: GOLD=[stage] vs CANDIDATE=[stage] Theremainingerrorstepshavenolabelorfallinto
- [Match/Mismatch]
2. **ACTION TYPE**: GOLD=[type] vs CANDIDATE=[type] - raretypes,eachcoveringlessthan1%oftheerror
[Match/Mismatch]
3. **SCOPE**: GOLD=[description] vs steps. Table3listseightexamples, togetherwith
CANDIDATE=[description] - [Comparable/Different] thedetailoftheenvironmentthateachonedepends
| 4. **FUNCTIONAL | EQUIVALENCE** | (if above | pass): E1-E4 | | | | | | | |
| --------------- | ------------------ | ----------- | ------------ | --- | ---------- | ---- | -------- | -------- | ------- | ---- |
| TRUE/FALSE | | | | | on. Fixing | such | an error | requires | knowing | that |
| 5. **Final | verdict**: [[A=B]] | or [[A!=B]] | | | | | | | | |
detail,suchasalogformatoraversionrule,sothe
| ===== Inputs | ===== | | | | | | | | | |
| ------------ | ----- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
fixisunlikelytohelponotherenvironments.
GOLD:
{expected_answer}
CANDIDATE:
| | | | | | # Long-tailerrorstep | | | | Environment | |
| --- | --- | --- | --- | --- | -------------------- | --- | --- | --- | ----------- | --- |
{generated_answer}
detail
| | | | | | 1 Usesagreedyregexthattreatsan | | | | theformatofthe | |
| --- | --- | --- | --- | --- | ------------------------------ | --- | --- | --- | -------------- | --- |
B ErrorProfilingDetails
| | | | | | entireloglineasonekey-valuepair. | | | | auditlog | |
| --- | --- | --- | --- | --- | -------------------------------- | --- | --- | --- | -------- | --- |
B.1 Statistics 2 Startsframecountingat1insteadof thebucketrule
0,shiftingevery10-framebucket.
Error profiling rolls out the policy twice on each 3 Replacesudbinsideexistingudbuf, thevariablenames
of the 2,017 environments and yields 12,746 er- producingudbufbuf. intheCfile
| | | | | | 4 Extractsmetricnamesinsteadof | | | | the | |
| ------------------------------------------ | ---------------------------------- | --- | --- | --- | ------------------------------ | --- | --- | --- | ----------------- | --- |
| rorsteps. | Table2gives,foreachmisbehavior,its | | | | | | | | | |
| | | | | | valueswithanincorrectawkfield | | | | NAME=...VALUE=... | |
| shareoftheseerrorstepsandoftheenvironments | | | | | index. | | | | lineformat | |
inwhichitappears. Thethreemostfrequentmis- 5 Readslittle-endianWAVsamplesas theaudiofile
big-endianintegers.
behaviorseachcovermorethan11%oftheerror
| | | | | | 6 Rejectsversion1.5.0despitethe | | | | theversionrule | |
| --- | --- | --- | --- | --- | ------------------------------- | --- | --- | --- | -------------- | --- |
steps and appear in more than a third of the envi- ^1.4.2requirementallowingit.
| | | | | | 7 Dividesthefeaturesumby3instead | | | | theimputation | |
| ----------- | ----------------- | -------- | ---------------- | --------- | --------------------------------- | --- | --- | --- | --------------- | --- |
| ronments. | Even the least | frequent | one, | “Prints a | | | | | | |
| | | | | | ofthenumberofvalidfeatures. | | | | rule | |
| binary file | as text”, appears | in | 140 environments | | | | | | | |
| | | | | | 8 Ignoresthenumberofdecimalplaces | | | | thenumberformat | |
| (6.9%). | A step can carry | more | than one | label, so | | | | | | |
andalwaysdividesby100rather
the eight misbehaviors together cover 52.1% of than10,turning25.5into2.55.
| the error | steps, slightly | less than | the sum | of their | | | | | | |
| --------- | --------------- | --------- | ------- | -------- | ------- | ------------------------------ | --- | --- | --- | --------- |
| | | | | | Table3: | Examplesoflong-tailerrorsteps. | | | | Eachcomes |
shares.
fromadetailofoneenvironmentandrarelycarriesover
tootherenvironments.
| # Misbehavior | | | Steps | Envs. | | | | | | |
| ---------------------------- | --- | --- | ----- | ----- | --- | --- | --- | --- | --- | --- |
| 1 Makesfalseclaimsabouttests | | | 15.4% | 46.7% | | | | | | |
C CostCalculationDetails
| 2 Repeatsunnecessarychecks | | | 11.5% | 42.2% | | | | | | |
| ------------------------------ | --- | --- | ----- | ----- | --- | --- | --- | --- | --- | --- |
| 3 Retrieswithoutfixingthecause | | | 11.2% | 36.5% | | | | | | |
Wecountthetrainingresourcesofeachrecipeup
| 4 Failstoproducerunnablecommands | | | 5.1% | 19.4% | | | | | | |
| -------------------------------- | --- | --- | ---- | ----- | --- | --- | --- | --- | --- | --- |
5 Kills processes without checking their 4.7% 14.0% toitsbestcheckpointratherthanoveritsfulltrain-
| identities | | | | | ingrun,whichisfairertorecipesthattrainlonger | | | | | |
| ---------------------- | --- | --- | ---- | ----- | -------------------------------------------- | --- | --- | --- | --- | --- |
| 6 Treatsguessesasfacts | | | 2.5% | 11.0% | | | | | | |
thanneeded;forexample,thereleasedTMax-4B
| 7 Claimssuccesswithoutverification | | | 1.7% | 8.9% | | | | | | |
| ---------------------------------- | --- | --- | ---- | ---- | --- | --- | --- | --- | --- | --- |
8 Printsabinaryfileastext 1.4% 6.9% checkpointisfromstep200of500,asstatedonits
| | | | | | modelcard. | Wealsocounttheresourcesofallpre- | | | | |
| ------------- | --- | --- | ----- | ----- | ---------- | -------------------------------- | --- | --- | --- | --- |
| Eighttogether | | | 52.1% | 75.4% | | | | | | |
processingstages,suchaserrorprofilingforMiRL,
| Table 2: | Share of the 12,746 | error | steps (Steps) | and | | | | | | |
| ------------ | ------------------------------- | ----- | ------------- | -------- | -------------- | ----------- | -------- | -------- | ------ | ----------- |
| | | | | | step profiling | for | PivotRL, | and | rubric | writing for |
| of the 2,017 | profiled environments | | (Envs.) | for each | | | | | | |
| | | | | | RaR+. | Most values | are | measured | from | our logs; |
| misbehavior. | Asteporanenvironmentcanhavemore | | | | | | | | | |
belowweexplainthevaluesthatarenot.
thanonemisbehavior.
17
Makesfalseclaimsabouttests 15.4% Repeatsunnecessarychecks 11.5%
Acheckruns,butitneverinvokesthebehaviouritsaysitis Thelastrunofthissucceeded,andtheanswerisalreadyin
| DEFINITION | | | | DEFINITION | | | | |
| ---------- | --- | --- | --- | ---------- | --- | --- | --- | --- |
testing.Ittestsasubstitute,usesinputthatmissesthecase, thehistory.Nothingchangedsince,andnonewquestionis
orexitsbeforereachingthecodeinquestion.Not:acheck beingasked.Not:repeatingafterafailure.
thatdoesexercisethebehaviourbutexpectsthewrong
| | | | | TASK | Youareastorageadministratormanagingdisk | | | |
| --- | --- | --- | --- | ---- | --------------------------------------- | --- | --- | --- |
answer.Not:claimingdonewithnocheckatall.
spaceforalegacyloggingapplication....
TASK YouareaDataEngineermigratingalegacy $ find / -type d -name "localssd" 2>/dev/null
PREFIX
| | recommendationETLpipeline.... | | | | /mnt/localssd | | | |
| --- | ----------------------------- | --- | --- | --- | ------------- | --- | --- | --- |
$ /app/recommender 0 3 0 1 0 “Good.Letmecheckthesystemconfiguration
| PREFIX | | | | STEP | | | | |
| ------ | ---------- | ------------- | --- | ---- | --------------- | --- | --- | --- |
| | Recommend: | NO (Score: 0) | | | morecarefully.” | | | |
“Sothethresholdisscore>0forYES(i.e., $ find / -type d -name "localssd" 2>/dev/null
STEP
score>=1).Butletmecheckwhathappens
| | | | | LOCALIZER | Theagentalreadylisted/mnt/andconfirmed | | | |
| --- | --- | --- | --- | --------- | -------------------------------------- | --- | --- | --- |
whenscoreis1.”
/mnt/localssdexists,sothisfindaddsnonew
| | $ /app/recommender | 0 1 0 0 0 | | | | | | |
| --- | ------------------ | --------- | --- | --- | --- | --- | --- | --- |
informationandnothingchangedinbetween.
| LOCALIZER | Thisprobesscore7,alreadyobservedasYES | | | | | | | |
| --------- | ------------------------------------- | --- | --- | --- | --- | --- | --- | --- |
earlier;itdoesn'ttestscore1(whichwould
neede.g.03000),soitfailstoverifythe
thresholdboundarytheagentclaimstobe
checking.
| Retrieswithoutfixingthecause | | | | Failstoproducerunnablecommands | | | | |
| ---------------------------- | --- | --- | ----- | ------------------------------ | --- | --- | --- | ---- |
| | | | 11.2% | | | | | 5.1% |
DEFINITION Thelastrunfailed,andtheagentrunsitagainwithout DEFINITION Thestepcontainsnoexecutablecommand.Itisprose,an
removingthecause.Cosmeticedits,ashortertimeout,or unfinishedreasoningblock,ortool-protocolmarkupwhere
extraloggingdonotcountasachange.Not:repeatinga thecommandshouldbe.Thefailurehappensbeforethe
successfulcheck. shellisreached.Not:acommandthatreachestheshelland
TASK Youareaforensicsanalystinvestigatinga failstoparse.Not:acommandthatrunsandprintsinstead
| | compromisedLinuxhost.... | | | | ofacting. | | | |
| --- | ------------------------ | --- | --- | --- | --------- | --- | --- | --- |
$ /app/jshon-20131105/jshon -e cert_b64 TASK YouareanMLOpsengineertaskedwith
PREFIX
| | /home/user/evidence/1.json | | | | buildingatooltotrackandsummarize | | | |
| --- | -------------------------- | ------- | --- | --- | -------------------------------- | --- | --- | --- |
| | warning: nothing | to read | | | cross-validationexperiments... | | | |
internal error: stack underflow $ cat <<'EOF' | python3 /home/user/tracker.py
PREFIX
| | (exit_code=1) | | | | ... | | | |
| ---- | -------------------------- | --- | --- | --- | ------------------- | ------- | ---------------- | --- |
| | | | | | [{"run_id": "auna", | "mean": | 1.0, "variance": | |
| STEP | “Letmeextractdifferently.” | | | | | | | |
0.0}, ...]
| | $ /app/jshon-20131105/jshon | -e | cert_b64 | | | | | |
| --- | --------------------------- | --- | -------- | --- | --- | --- | --- | --- |
/home/user/evidence/1.json 2>&1 STEP “Thetie-breakingisworkingcorrectly-
Theagentisre-runningtheexactsame "auna"comesbefore"brun"alphabetically.
LOCALIZER
commandthatjustfailed,onlyadding2>&1 Letmealsotestforfloatscorestomakesure
| | whichdoesn'tchangethecommand'sbehavior | | | | thoseworktoo.” | | | |
| --- | -------------------------------------- | --- | --- | --- | -------------------- | --- | --- | --- |
| | oroutputcontentsincebothstreamswere | | | | $ (no valid command) | | | |
alreadyvisibleinthepreviousrun. Thestepcontainsnoexecutablecommand,so
LOCALIZER
itcannotmakeprogressorverifyanything.
Figure13: Examplesofmisbehaviorsteps(1/2). Eachexampleshowsthedefinitionofthemisbehavior,anexcerpt
ofthetask,thelaststepoftheprefix,theerrorstepofthepolicy,andtheexplanationoftheerrorlocalizer. The
percentageistheshareoferrorstepswiththemisbehavior. Longtextiscutwithanellipsis.
| | | | | | SB/(1−d) | = | 2,370 | |
| --- | --- | --- | --- | --- | -------- | --- | ----- | --- |
TMax-4B. TheTMaxpaperreportsthetraining rollouts and environments.
settingsandcurvesforTMax-4B,butnotthetotal We estimate its tokens and GPU hours from the
number of rollouts, tokens, or GPU hours. We tokensperstep,stepsperrollout,andthroughputof
estimatethesetotalsfromthereportedsettingsand ourTMax-189run,exceptfortheoutputtokensper
curves. The number of rollouts is SBG/(1−d), step,whichcomefromFigure6intheTMaxpaper.
| where | S = 200 steps, | B = 8 environments | per | | | | | |
| ------- | -------------- | ------------------ | -------- | ----- | ---------------- | --- | --------- | ---------- |
| | | | | Total | cost. We convert | the | tokens of | the policy |
| step, G | = 32 rollouts | per environment, | and d is | | | | | |
intodollarsattheTinkerpricesinTable4,taking
| theshareofdeadgroups,whichareresampled. | | | We | | | | | |
| --------------------------------------- | --------- | -------------- | ----------- | ------------ | -------------- | --------- | -------- | ------------ |
| | | | | the uncached | share of | the input | tokens | as two di- |
| derive | d = 32.5% | from Figure 11 | in the TMax | | | | | |
| | | | | vided | by the average | number | of steps | per rollout, |
paper(Ivisonetal.,2026),whichcountsthegroups
| | | | | and add | the model API | fee. | This prices | the pol- |
| --- | --- | --- | --- | ------- | ------------- | ---- | ----------- | -------- |
inwhichallrolloutsfail;TMaxnotesthatgroups
| | | | | icy as | a service billed | per token | rather | than as a |
| --- | --- | --- | --- | ------ | ---------------- | --------- | ------ | --------- |
inwhichallrolloutspassarerareanddoesnotplot
| | | | | rented | GPU cluster, | so we report | the | GPU hours |
| ------------------------- | --- | -------------- | --- | ------ | ------------ | ------------ | --- | --------- |
| them,sowecountthemaszero. | | Thisgives75.9K | | | | | | |
separatelyanddonotaddthemtothetotalcost.
18
Killsprocesseswithoutcheckingtheiridentities 4.7% Treatsguessesasfacts 2.5%
Terminatesbyabroadnamepatternoranunverified Commitsaguessedorbadlyextractedvalueintothework
| DEFINITION | | | | | | | DEFINITION | | | | | | |
| ---------- | --- | --- | --- | --- | --- | --- | ---------- | --- | --- | --- | --- | --- | --- |
processid,withoutestablishingthatthematchedprocesses asifitwereknown,whenthehistoryshowstheevidencefor
aretheintendedtargets.Not:stoppingaprocessthatwas itismissing,garbled,orcontradictory.Not:ahypothesis
identifiedfirst. thatisthentested.Not:aconstantthetaskspecified.
TASK Youareasystemsprogrammerdebugginga TASK Iamaresearcherorganizingadatasetof
websecurityWAF(WebApplicationFirewall) academiccitations....Ihaveascreenshotof
| | service.... | | | | | | | themissing... | | | | | |
| --- | ----------- | --- | --- | --- | --- | --- | --- | ------------- | --- | --- | --- | --- | --- |
$ for pid in $(ls /proc | grep -E '^[0-9]+$'); $ sqlite3 /home/user/datasets/research_graph.db
| PREFIX | | | | | | | PREFIX | | | | | | |
| ------ | --- | -------- | --- | --- | --- | --- | ------ | ------- | ---------- | ------- | --------- | ------------ | --- |
| | do | ... done | | | | | | "SELECT | source_id, | | target_id | FROM edges;" | |
| | (no | output) | | | | | | 1|2 | 5|6 1|7 | 7|8 2|3 | | | |
“Letmetryadifferentapproach-changingto “Good,edge(2,3)isnowinthedatabase.
| STEP | | | | | | | STEP | | | | | | |
| ---- | --- | --- | --- | --- | --- | --- | ---- | --- | --- | --- | --- | --- | --- |
port5001andseeifFlaskworks,orstart NowIneedtoadd(3,4)and(4,5):P_2050
| | nativelyagain.” | | | | | | | (node3)->P_3333(node4),P_3333(node4) | | | | | |
| --------- | ------------------------------------- | --- | ------------- | ----- | --- | -------- | --- | ------------------------------------ | --------------------------------------------- | ----- | ----------- | ---------- | --- |
| | $ pkill | -9 | redis-server; | pkill | -9 | python3; | | ->P_4004(node5).” | | | | | |
| | sleep | 2 | | | | | | $ | sqlite3 /home/user/datasets/research_graph.db | | | | |
| | | | | | | | | "INSERT | INTO | edges | (source_id, | target_id) | |
| LOCALIZER | Theagent'sstatedplanistoswitchFlaskto | | | | | | | | | | | | |
VALUES (3, 4);"
port5001,yetthecommandkillsredis-server
andpython3insteadoftheunknownprocess LOCALIZER Theagentinsertededge(3,4)basedonOCR
holdingport5000,sotheportconflictwill interpretationwithoutverifyingtheambiguous
persistandrediswillbedownunnecessarily. readings(2080vs2050,3933vs3333)despite
acknowledgingtheuncertainty,whichrisks
insertinganincorrectedgeintothedatabase.
| Claimssuccesswithoutverification | | | | | | | Printsabinaryfileastext | | | | | | |
| -------------------------------- | --- | --- | --- | --- | --- | ---- | ----------------------- | --- | --- | --- | --- | --- | ---- |
| | | | | | | 1.7% | | | | | | | 1.4% |
DEFINITION Theagentdeclaresthetaskdone,orastepsuccessful, DEFINITION Sendstherawbytesofanexecutable,library,image,or
withoutrunningthecheckthatwouldshowit,oraftera archivetoatextdisplayinordertoinspectit.Not:ahex
checkthatfailed.Not:runningacheckthatmissesitstarget. dumporanextractedstring,whicharedeliberate.
Not:committingaguessedvalue. TASK Youareanetworksecurityengineer
TASK Youareabackupoperatorautomatingrestore investigatingaseriesofsuspiciouspayloads
| | tests.... | | | | | | | capturedonyourinternalnetwork.... | | | | | |
| ------ | --------- | -------- | ---- | ---- | ------- | ----------- | ------ | --------------------------------- | ------------------ | --- | --- | --- | --- |
| | $ (no | command) | | | | | | $ | ls -la /opt/certs/ | | | | |
| PREFIX | | | | | | | PREFIX | | | | | | |
| | ERROR: | no valid | bash | tool | call in | your reply. | | ... | decoy_root.pem | | | | |
... true_root.pem
| STEP | “Igotanerroraboutamissingbashtoolcall. | | | | | | | | | | | | |
| ---- | -------------------------------------- | --- | --- | --- | --- | --- | ---- | ------------------------------ | --- | --- | --- | --- | --- |
| | Letmesubmitthetasknow.” | | | | | | STEP | “Good,therearetwocertificates: | | | | | |
$ echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT decoy_root.pemandtrue_root.pem.Letme
Theagentprematurelysubmittedthetask lookatthepolicyPNGformatandextractthe
LOCALIZER
| | whilethedaemonisnotrunning(port8888is | | | | | | | fingerprintfromit.” | | | | | |
| --- | -------------------------------------- | --- | --- | --- | --- | --- | --- | ------------------- | ------------------- | --- | --- | --- | --- |
| | heldbyanunknownlistener,thedaemonnever | | | | | | | $ | cat /app/policy.png | | | | |
successfullybound),sotherestorepipelineis CatsabinaryPNGimagedirectlytothe
LOCALIZER
unverifiedandlikelybroken. terminal,whichwilldumpunreadablebinary
dataandisnotausefulwaytoinspectan
image.
Figure14: Examplesofmisbehaviorsteps(2/2). Eachexampleshowsthedefinitionofthemisbehavior,anexcerpt
ofthetask,thelaststepoftheprefix,theerrorstepofthepolicy,andtheexplanationoftheerrorlocalizer. The
percentageistheshareoferrorstepswiththemisbehavior. Longtextiscutwithanellipsis.
D Artifacts,Licenses,andUseofAI
| | | | | | | | 3.8 Flash | and | Claude | Opus | 5 through | | their com- |
| ---------- | --- | --- | --- | --- | --- | --- | -------------------------------------------- | --- | ------ | ---- | --------- | --- | ---------- |
| Assistants | | | | | | | mercialAPIsundertheproviders’termsofservice. | | | | | | |
Theselicensesallowresearchuse,andweuseeach
| Licenses. | We | use | the following | | artifacts | under | | | | | | | |
| --------------- | --- | ------- | ------------- | --- | --------- | -------- | -------- | --------- | ----------- | -------- | ----- | -------- | ---------- |
| | | | | | | | artifact | for its | intended | research | | purpose. | We re- |
| their licenses: | | the | Qwen3.5-4B | | and | Qwen3.8- | | | | | | | |
| | | | | | | | lease | our code, | checkpoint, | | data, | and | evaluation |
| 27B models | | (Apache | 2.0); | the | TMax | environ- | | | | | | | |
trajectoriesunderlicensescompatiblewiththear-
| ments | (ODC-BY) | and | the | TMax-4B | | checkpoint | | | | | | | |
| ----- | -------- | --- | --- | ------- | --- | ---------- | ------- | ---- | ----- | ---------- | --- | --- | --------- |
| | | | | | | | tifacts | they | build | on: Apache | 2.0 | for | code, the |
(Apache2.0);Terminal-BenchandOpenThoughts-
checkpoint,andthetrajectories,andODC-BYfor
| TBLite | (Apache | 2.0); | the | open-instruct | | training | | | | | | | |
| ------ | ------- | ----- | --- | ------------- | --- | -------- | ------------ | --- | ---- | -------- | ------------- | --- | --- |
| | | | | | | | data derived | | from | the TMax | environments. | | See |
code (Apache 2.0) and the mini-SWE-agent har- https://github.com/runchu-tian/MiRL.
ness(MIT)usedbyTMax;andthejudgeprompt
| of NeMo-Gym | | (Apache | 2.0). | We | access | Gemini | | | | | | | |
| ----------- | --- | ------- | ----- | --- | ------ | ------ | --- | --- | --- | --- | --- | --- | --- |
19
Model Input Cached Output
Qwen3.5-4B $0.33 $0.066 $1.005
Qwen3.8-27B ¥3.646 ¥0.729 ¥21.875
Gemini3.8Flash $0.75 $0.075 $3.75
ClaudeOpus5 $5 $0.50 $25
Table 4: Prices per million tokens at the time of our
experiments: Qwen3.5-4B(policy)onTinker,Qwen3.8-
27B (error localizer and teacher) on Alibaba Cloud,
Gemini 3.8 Flash (judges) from Google, and Claude
Opus5(annotator)fromAnthropic.
Packages. Table5liststhesoftwarethatweuse
anditssettings;Table1givestheremaininghyper-
parameters.
Package Version Use
open-instruct 7387d2f9 RLtraining(grpo_fast)
DeepSpeed 0.18.4 RLtraining
Ray 2.53.0 RLtraining
PyTorch 2.10.0 RLtraining
Transformers 5.4.0 RLtraining
vLLM 0.19.1 Inference(bf16)
Harbor 0.6.6 Evaluation
Apptainer 1.3.4 Sandbox
LiteLLM 1.75.0/1.83.14 LLMcalls(train/eval)
difflib stdlib PivotRLstepmatching
Model/prompt Revision
Qwen3.5-4B 851bf6e8 Policy
Qwen3.8-27B 1d4bf0f2 Localizer,teacher,judge
NeMo-Gym 325fb49d PivotRLjudgeprompt
Table 5: Software and model versions. open-instruct
andtheTMaxagentcomefromthesameTMaxcommit.
vLLMusesprefixcachingandtheqwen3_xmltool-call
parser;Harborallowsatmost64formaterrors;PivotRL
matchesstepswithSequenceMatcheratathresholdof
0.95;Qwen3.8-27Basthemisbehavior-ratejudgeuses
thinkingwithloweffort.
Use of AI assistants. We used Claude Code, a
codingagent,intheterminaltohelpwritePython
andshellscriptsandtoformattheLaTeXsource.
20