← Back to summary
FOCUS: Active Critic Learning for Long-Horizon Language Model Agents
Converted from PDF with markitdown. Layout may be rough.
FOCUS: ACTIVE CRITIC LEARNING FOR LONG-
HORIZON LANGUAGE MODEL AGENTS
ZhengpingJiang WilliamJurayj LiaoyaqiWang ChunshengZuo
AnqiLiu BenjaminVanDurme
JohnsHopkinsUniversity
{zjiang31, aliu.cs, vandurme}@jhu.edu
ABSTRACT
Sparseoutcomerewardsmakeitdifficultforlong-horizonlanguage-modelagents
toassigncredittoearlieractions. Learnedcriticsprovidevalueestimatesforin-
termediatestates,buttheseestimatescanbeinaccurate,whileestimatinginterme-
diate values with repetitive continuation samples from points within a trajectory
incurs substantial costs. To mitigate these issues, we introduce FOCUS, an RL
methodthatusesselectivebranchingtohelpthecriticnetworklearnitsvaluees-
timator. Therolloutcontrollerfacilitatesactivelearningforthecriticbysampling
continuationsfromstatesthatareexpectedtomostreducethecritic’sestimation
error. These refined value estimates then score the segment between consecu-
tivestateswithaTD(λ)advantage,whilethesampledevidencesuppliestraining
targetsforthecritic. FOCUSsupportslong-horizonagenticreinforcementlearn-
ingthroughamodularcontext-managementinterfacethatdecouplescontextman-
agement from rollout allocation. Our experiments show that using compaction-
inducedsegmentboundariesforvalueestimationandcreditassignment,FOCUS
outperforms GRPO and VinePPO with matched per-step generation budgets on
legalreasoning,deepsearch,andcodelocalizationbenchmarks,improvingheld-
out accuracy at Qwen3.5-9B scale by 7.2 points over the strongest baseline on
BrowseComp-Plus (55.4 vs. 48.2) and by 4.1 points on DeonticBench (56.5 vs.
52.4).
1 INTRODUCTION
Language-modelagentsareincreasinglytrainedtosolvetasksthatunfoldoverdozensorhundreds
oftoolcalls: resolvinganissueinarealrepository(Jimenezetal.,2024;CursorResearch,2026),
completing a task on a live website (Zhou et al., 2024b), or answering a question that requires
manyroundsofsearchandreading(Chenetal.,2026;KimiTeam,2026). ReinforcementLearning
with Verified Rewards (RLVR) is the standard way to improve such agents, but long trajectories
meanbothsamplingandcreditassignmentarecostly. Asingleverifiablerewardmustbedistributed
acrosseverytokenwithinarolloutevenwhenmanyhadlittleimpactonthefinaloutcome. Tolearn
efficiently,weneedproperintermediatevalueestimatesunderalimitedrolloutbudget.
Theseestimatescancomefromalearnedcriticnetwork(Konda&Tsitsiklis,1999)orthroughMonte
Carloapproximationfromagivenstate(Shaoetal.,2024).Alearnedcriticreusesinformationacross
trajectories, but its predictions can be inaccurate. Tree-structured rollouts branch at intermediate
states to refine value estimates (Kazemnejad et al., 2025; Hou et al., 2025; Li et al., 2026d; Guo
etal.,2025;Jietal.,2026). However, samplingadditionallanguagemodelcontinuationsforeach
estimateincurssubstantialcosts,raisingthequestionofhowtoallocatearolloutbudgettoimprove
valueestimationmostefficiently.
Wetreatrefiningvalueestimatesviaadditionalcontinuationbranchingasanactivelearningprob-
lem for a critic network: continuations should be sampled selectively from states where addi-
tionalevidenceisexpectedtoimprovevalueestimationthemost. WeintroduceFOCUS:Focused
OptimizationonCriticalUnderperformingSegments,arolloutcontrollerthatrunseachepisodeas
asearchtree. FOCUScombinesthecritic’spredictionswithevidencefromsampledcontinuations
and allocates further rollouts according to their estimated marginal reduction in value-estimation
1
6202
tcO
62
]GL.sc[
1v10853.0162:viXra
| | | | | | φ | Critic mean | Initial rollouts |
| --- | --- | --- | --- | -------- | --- | --------------- | ------------------- |
| | | | | | | Posterior | previous expansions |
| | | | | φ STEP 2 | | | |
| | | | | | | Possible update | current expansion |
| | | | | | 0 | 1 | |
| | | | 0 | 1 | | | |
Correct · r = 1 STEP 1
Which city houses
| the museum founded | | | State | | | | |
| ------------------ | --- | --- | --------- | --- | --- | ------ | --- |
| by the author of | | | Mira Vale | | | STEP 4 | |
Evidence
| The Glass Orchard? | search | open | D1, lines 8–12 | | | | Larkhaven |
| ------------------ | --------------- | ---------- | ------------------------- | --- | --- | ----------------- | --------- |
| | | | O p e n q u e s t i o ns | | | fi n d | submit |
| | “Glass Orchard” | D1 · L8–12 | W h i c h m u s e u m ? | | φ | | |
| | | | N e x t s t e p s | | | D2 · “ l oc ated” | |
Search founder
Scheduler loop
| STEP 1 G e t an expanded | | STEP 2 S c o re | co m p a c t i o n | 0 | 1 | | |
| ------------------------ | --- | ----------------- | ----------------------- | --- | --- | --- | ----------------- |
| tre e | | st a te s | u si ng c r i t ic φ | | | | Incorrect · r = 0 |
2 rollouts
STEP 3
r=0
C o l l e c t a d d itional C o m p u t e e x p e cted Incorrect · r = 0
| STEP 4 co n t i | n u a t io n | STEP 3 er ro r r | e d u c ti o n | | | | |
| --------------- | ------------ | ----------------- | -------------- | --- | --- | --- | --- |
Figure 1: Overview of the proposed FOCUS rollout scheduler for a deep-research-style problem.
FOCUScontinuesworkingonarollouttree. Ateachiteration,FOCUSSTEP1: takesanexpanded
tree; STEP 2: scores each compaction-induced state with the critic; STEP 3: compares against
outcomes to decide critic quality (subsection 2.2); STEP 4: sources additional continuations that
areexpectedtocorrectcriticerrorsmostefficiently(subsection2.3). Withthisactivelearningloop
forrolloutacquisition,FOCUSprovidesamoreaccuratevalueestimatewithlimitedrolloutbudgets
(subsection3.4).
error (Hayet al., 2012; Kaufmann& Koolen, 2017). These refined valuesprovide stepwise credit
forthepolicy,whiletheacquiredevidencesuppliestrainingtargetsforthecritic.
Inlong-horizontasks,contextcompaction(Cursor,2026;Harvey,2026;Sunetal.,2026;Lietal.,
2026e),wheretheagentresumesfromacompactedaccountofitspreviousactionsandobservations,
providesanaturalstructureforapplyingselectiverollouts: eachsegmentbetweencompactionsbe-
comesastep,atwhichstep-levelrewardsandvaluescanbecalculated(Figure1).However,FOCUS
doesnotdependonaparticularcontext-managementprotocol;itsmodulardesignseparatesrollout
assignment.1
allocation, context management, and credit Appendix H details the implementation,
andSection4discussesrelatedwork.
WeevaluateFOCUSonDeonticBench(Douetal.,2026a),BrowseComp-Plus(Chenetal.,2026),
and LocBench (Chen et al., 2025). FOCUS outperforms GRPO and VinePPO on all three bench-
marksunderamatchedper-stepgenerationbudget. WithQwen3.5-9B,FOCUSimprovesheld-out
accuracyoverthestrongestbaselineby7.2pointsonBrowseComp-Plus(55.4vs.48.2)(Table1).
Diagnostics show more effective reduction of value-estimation error as the refinement budget in-
creases,whileablationsdemonstratethatthegainsarenottiedtoasinglecompactionimplementa-
tion(Sections3.4and3.5).
Insummary,ourcontributionsare:
• Wecastthecombinationofcriticpredictionsandcontinuationevidenceasanactivecritic
learningproblem(Sections2.2and2.3).
• Weintroduce FOCUS,ahighly customizablerolloutcontroller, whichallocatesa limited
rollout budget according to estimated value-estimation error reduction and integrates the
resultingestimateswithcreditassignment.
• We show that FOCUS improves over critic-free baselines on long-horizon benchmarks
(Section3).
2 METHOD
2.1 PROBLEMSETUP
We consider reinforcement learning of a language-model policy π from outcome rewards. The
θ
policygeneratesresponsesoractionsfromitscurrentcontext,andacriticV ,parameterizedbyϕ,
ϕ
1Code:https://github.com/zipJiang/FOCUS.
2
estimatestheexpectedremainingreturn. Weassumethatatleastsomeoftheintermediatestatesare
easilyforkableandthatrepeatedsamplingfromthemistrivial. Anepisodeξ recordsthegenerated
turnsandrewardsfromonecompletedrun.
Givenafiniterolloutbudget,FOCUSselectsfromcandidatestatesfromwhichtosampleadditional
continuations,refinestheirvalueestimates,andconstructstrainingsamplesforboththecriticandthe
policy. Fortool-usingagents,compactionboundariesprovidenaturalcandidatestates. AppendixH
detailstheexecutionmechanicsusedinourexperiments.
2.2 CRITICREFINEMENTANDLEARNING
We first describe how collected continuations produce the learning signals for the critic and the
policy. Section2.3describeshowweallocatetherolloutbudget.
States and rollout segments. For each prompt x, we represent collected rollouts as a tree T x .
A nonterminal node denotes a state from which the policy can resume with the same context and
environmentstate; anedgecontainsthepolicy-generatedcontinuationandenvironmenttransitions
untilthenextstate. Wewritec¯forthecontextofanode(state),leavingitsassociatedenvironment
stateimplicit.
Acompletedtrajectorycontainsnonterminalnodesc¯ = x,c¯ ,...,c¯ andaterminalnodec¯ ,
| | | | | | | | 0 | 1 | M | | M+1 |
| ----------------- | --- | --- | ---- | -------------- | --- | --- | --- | --- | --- | --- | --- |
| connectedbyedgese | | | =(c¯ | →c¯),i=1,...,M | | | +1. | | | | |
| | | | i | i−1 | i | | | | | | |
Refining state values. Let Vπ(c¯) denote the expected discounted return of a continuation from c¯
underthecollectionpolicy. ThecriticV ϕ (c¯)predictsthisvaluefromthestate’scontext. However,
thispredictioncanhavehighvariance. Werefineitspredictionusingtheoutcomeevidencefromthe
n(c¯)continuationsch(c¯)thathavebeenexpandedfromit,backeduponelevelasinTreeRPO(Yang
etal.,2025):
| | | | 1 | (cid:16) | | | (cid:17) | | | | |
| --- | ------ | --- | ------ | -------- | ------ | ------ | -------- | ------------------------- | --- | --- | --- |
| | G¯(c¯) | | | (cid:88) | +γd | | V˜(c¯′) | V˜(c¯′)=0ifc¯′isterminal, | | | |
| | | = | | | R | c¯→c¯′ | , | | | | (1) |
| | | | n( c¯) | | c¯→c¯′ | | | | | | |
c¯′∈ch(c¯)
whereV˜(c¯′)isthecontinuation’sownrefinedvalue(Eq.2below). Wepreferthistoarawaverage
ofdescendantoutcomes(Lietal.,2026d)becausetheallocatorchooseswhichnodestoexpand,so
theleavesbeneathc¯arenolongeranon-policydrawfromc¯;thecontinuation’srefinedvaluecarries
itsevidencebackupinstead,onelevelatatime. Wecombinethecriticpredictionandtherecursive
backupwithafixedpseudo-observationweightκ,
(c¯)+n(c¯)G¯(c¯)
κV
| | | | | | V˜(c¯) = | ϕ | | . | | | (2) |
| --- | --- | --- | --- | --- | -------- | --- | --- | --- | --- | --- | --- |
κ+n(c¯)
| | | | V˜(c¯) | | | | | G¯; | | | |
| --- | --- | --- | ------ | --- | --- | --- | --- | --- | --- | --- | --- |
For n(c¯) = 0, we set = V ϕ (c¯) without evaluating terminal values are zero. As n(c¯)
increases,theblendplacesmoreweightonthebacked-upmean(Silveretal.,2016;Tesauroetal.,
2010).
Criticsupervisionandactoradvantages. Aftercollectionandthefinalbackup,wetrainthecritic
ontherecursivetargetG¯
ateverynon-terminalstate:
| | | | | | (cid:104) | (cid:88) | (cid:0) | | (cid:1)2 (cid:105) | | |
| --- | --- | --- | --- | ------ | --------- | -------- | ------- | ----------- | ------------------ | --- | --- |
| | | | L | (ϕ) | = E | | V | (c¯)−G¯(c¯) | . | | (3) |
| | | | | critic | | | ϕ | | | | |
c¯∈Tx:n(c¯)>0
The target is held fixed during training. This update transfers the collected continuation evidence
Withinthecurrentiteration,therefinedvaluesV˜
| intothecriticusedinsubsequenttraining. | | | | | | | | | | | supply |
| -------------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ------ |
thesegment-leveltemporal-differenceresidual2
| | | | | Aˆ | | +γdiV˜(c¯)−V˜(c¯ | | | | | |
| --- | --- | --- | --- | --- | ---- | ---------------- | --- | --- | --- | --- | --- |
| | | | | | =δ = | R | | | ). | | (4) |
| | | | | i | i | i | i | i−1 | | | |
Weapplyeachedge’sdetachedadvantagetoitspolicy-generatedtaskandcontext-managementto-
kens,denotedy ei ,maskingenvironmentobservationsandconditioningtext.Withπ old thecollection
policy,ρ thetoken-levelimportanceratio,andϵtheclippingthreshold,theactorminimizes
t
| | | | (cid:104)(cid:88) | | | | | | (cid:1)(cid:105) | π (y |y | ) |
| ----- | --- | --- | ----------------- | -------- | ---------- | ------ | ----------- | --- | ---------------- | --------- | ----- |
| | | −E | | (cid:88) | (cid:0) Aˆ | | ,1−ϵ,1+ϵ)Aˆ | | | θ t | <t |
| L | (θ) | = | | min | ρ , | clip(ρ | | | , ρ | = | . (5) |
| actor | | | | | t i | t | | | i | t π (y |y | ) |
| | | | | | | | | | | old t | <t |
| | | | i t∈yei | | | | | | | | |
AppendixF.1givesthegeneralTD(λ)extension.
2Unlessotherwisespecified,allexperimentsuseγ
=1,λ=0,andκ=1.
3
2.3 SELECTIVEROLLOUTALLOCATIONFORCRITICLEARNING
Branching continuations from candidate states supplies additional value evidence (Kazemnejad
et al., 2025; Hou et al., 2025; Li et al., 2026d; Guo et al., 2025), but uniform allocation does not
accountfordifferencesincriticerrorortheexpectedbenefitofanothersample. Givenanexplored
treeT andanadditionalrolloutbudgetB,FOCUSchooseswhichcandidatenodereceivesthenext
x
continuation. Theaimistoacquireevidencewhereitisexpectedtoreducevalueestimationerror
themost(Roy&McCallum,2001;Mussmannetal.,2022).
Outcomeevidenceforselection. Whileselectionusesthesameformulaformixingobservedcon-
tinuation outcomes and critic predictions, it does not use the continuation-refined backups of its
children as in Eq. 1. Specifically, let N(c¯) be the number of continuation outcomes and G¯ (c¯)
obs
their mean. A completed continuation contributes one outcome at its starting node and at each
newlycreatednodealongthatcontinuation. Laterbranchingfromadescendantdoesnotaddmore
observations to its strict ancestors. Thus N(c¯) counts observed continuation outcomes, whereas
n(c¯)inEq.1countsdirectchildren.
Forbinarycontinuationoutcomes, theselectionmodelusesaBetapriorofstrengthκcenteredon
v =V (c¯). Itsposteriormeanandexpectedsquaredcriticerrorare
ϕ
κv+NG¯ µ (1−µ )
µ
obs
=
κ+N
obs, e(cid:98)2 =(v−µ
obs
)2+ o
κ
bs
+N +
o
1
bs . (6)
Noticehowµ isdistinctfromthecontinuation-refinedtrainingvalueV˜. Thepredictivevariance
obs
ofthenextoutcomeisσ(cid:99)2 =µ
obs
(1−µ
obs
). Inparticular,werelyonµ
obs
toconducttheselection
duetotheobservationoutlinedinAppendixE.
Expected benefit of a continuation. We score the expected utility of a continuation using the
marginalMSEreductionofvalueestimationerroratthatcandidatestate. Forcriticsquarederrore2
andoutcomevarianceσ2,theacquisitionfunctionbecomesMSE(N)=(κ2e2+Nσ2)/(κ+N)2.
Differentiatingwithe2andσ2heldfixed,thensubstitutingtheposteriorestimatesgives
2κ2e(cid:98)2+(N −κ)σ(cid:99)2
g(c¯) = , (7)
(κ+N)3
whereN =N(c¯)countscompletedcontinuationoutcomes.
Algorithm1firstacquiresrolloutevidence(Section2.3),thenrefinesvaluesbottom-up(Section2.2).
Eachcontinuationcompletesbeforethenextallocationdecision. TheeligiblesetisF = {c¯∈ T :
x
c¯nonterminalandresumable, g(c¯) > 0}. TheincludedCONTINUE(c¯,π)helperreturnsnewnodes
andconnectingsegmentsthroughtermination.Sincegusesobservedsuffixstatisticsandfixedcritic
predictions,valuerefinementcanfollowcollection. AppendixFgivesthefulltrainingloop.
Practical considerations. Total branching width at any node is uncapped. The allocation phase
can also run asynchronously, with each completed continuation updating the acquisition statistics
while others remain in flight. In this case, we replace N in Eq. 7 by N (c¯) = N(c¯) + m(c¯),
eff
wheremcountspendingcontinuations;posteriorestimatesstillusecompletedoutcomesonly. We
limit concurrent continuations per node to C = 2 and globally to worker capacity W, restricting
eligibility to m(c¯) < C. Completed children do not consume concurrency slots. In practice, the
two loops can run concurrently, each with multiprocessing. After the launch budget is exhausted,
pendingcontinuationsfinishandafinalbottom-uprefinementensuresthatthetrainingtargetsreflect
thecompletedtree.
3 EXPERIMENTS
WefirstcompareFOCUSwithPPOandGRPOinapreliminaryevaluationonMATH(Hendrycks
etal.,2021)problemsatdifficultylevels4–5,trackingaccuracy,evaluationoutputlength,andper-
rolloutreductionovertrainingupdates. WethencompareFOCUStoGRPOandVinePPOonthree
long-horizon benchmarks at Qwen3.5-9B and Qwen3.5-27B (Qwen Team, 2026) (LoRA), assess
howMCTSrefinementimprovescriticestimates,andablatethecompactionoperatorandrefinement
rollouts.
4
Algorithm1Selectiverolloutallocationandvaluerefinement,withcontinuationhelper
| Require: | treeT x ;fixedactorπ | | θ andcriticV | ϕ ;budgetB;priorstrengthκ | | | |
| -------- | ------------------------------------------------------------ | --- | ------------ | ------------------------- | --- | --- | --- |
| Require: | environmentE;branchpredicateB;compactiontriggerGandoperatorC | | | | | π | |
| Ensure: | treeT withcritictargetsG¯andrefinedvaluesV˜ | | | | | | |
x
| initializeN,G¯ | | fromcompletedsuffixes;refreshF,g | | | | | |
| -------------- | --- | -------------------------------- | --- | --- | --- | --- | --- |
obs
§2.3:Selectiverolloutallocation
| whileB | >0 ∧ | F ̸=∅do | | | | | |
| ------ | --------- | -------------------- | --- | --- | --- | --- | --- |
| | c¯←argmax | g(u);ξ←CONTINUE(c¯,π | | | ) | | |
| | | u∈F | | | θ | | |
ifξ=⊥then
| | B ←B−1 | | | | | | ▷executionfailure |
| --- | ------ | --- | --- | --- | --- | --- | ----------------- |
else
| | T ←ATTACH(T | | ,c¯,ξ) | | | | |
| --- | ----------- | --- | ------ | --- | --- | --- | ------------------------ |
| | x | | x | | | | |
| | B ←B−d(ξ) | | | | | | ▷d(ξ):newtasktransitions |
N(u),G¯
| | | (u)←0foru∈New(ξ) | | | | | ▷newnodes |
| --- | --- | ---------------- | --- | --- | --- | --- | --------- |
obs
| | SCORE(T | ,V ) | | | | ▷cacheunscorednonterminalvalues | |
| --- | ------------------ | ---- | ------------ | --- | ------------ | ------------------------------- | --- |
| | | x ϕ | | | | | |
| | foru∈{c¯}∪New(ξ)do | | | | | ▷Gξ(u):observedsuffixreturn | |
| | | | N(u)G¯ (u)+G | (u) | | | |
| | G¯ | (u)← | obs | ξ | ;N(u)←N(u)+1 | | |
| | obs | | N(u)+1 | | | | |
endfor
endif
refreshF,g
endwhile
§2.2:Valuerefinementandcritictargets
V˜(u)←0forterminalnodes;V˜(u)←V
ϕ (u)fornonterminalleaves
| foru∈T | x withn(u)>0,inreversetopologicalorderdo | | | | | | |
| ------ | ---------------------------------------- | --- | --- | --- | --- | --- | --- |
1
| | G¯(u)← | (cid:88) | (cid:0) +γdu→vV˜(v) | | (cid:1) | | |
| --- | ------ | -------- | ------------------- | --- | ------- | --- | --- |
R u→v
n(u)
v∈ch(u)
κV (u)+n(u)G¯(u)
| | V˜(u)← | ϕ | | | | | |
| --- | ------ | --- | --- | --- | --- | --- | --- |
κ+n(u)
endfor
returnT
x
functionCONTINUE(c¯,π)
| | (c,s)←FORK(c¯);ξ←[];ζ | | ←[] | | | | ▷ζ:currentsegment |
| --- | --------------------- | --- | --- | --- | --- | --- | ----------------- |
while¬TERMINAL(s)do
Modularcontextmanagement
ifG(c)=1then
| | c←C | (c) | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- |
π
else
τ ∼π(·|c);(s,o,r)∼E(s,a(τ));c←c∥τ∥o
endif
| | η←TRACE();ζ | | ←ζ∥η | | | ▷task/compactionturns,rewards,transitions | |
| --- | ------------- | --- | ---- | --- | --- | ----------------------------------------- | --- |
| | s←UPDATE(s,η) | | | | | ▷countersandterminationflags | |
ifFAILED(s)thenreturn⊥
| | ifTERMINAL(s) | | ∨ B(c,s)=1then | | | | ▷BindependentofG |
| --- | ---------------------------------- | --- | -------------- | --- | --- | --- | -------------------- |
| | c¯′ ←c;ξ←ξ∥[(c¯′,SNAPSHOT(s),ζ)];ζ | | | | ←[] | | ▷saveresumptionstate |
endif
endwhile
returnξ
endfunction
3.1 PRELIMINARYEVALUATIONONMATHEMATICALREASONING
WecompareFOCUS,PPO,andGRPOwithQwen3.5-4B(QwenTeam,2026)onMATHLevel4–
5 problems for 32 completed updates, evaluating the base checkpoint and checkpoints at updates
8, 16, 24, and 32. We use 512 training questions from the official MATH training split and 256
5
validationquestionsfromitsofficialtestsplit.3 Thetrainingsetcontains226Level4and286Level
5questions;validationcontains123Level4and133Level5questions.
All methods consume the fixed 512-question order for 32 training steps. PPO uses 64 questions
pertrainingstep; GRPOandFOCUSeachuse16questionspertrainingstep. Tokeepthetraining
computeroughlymatched, GRPOusesgroupsize6, andFOCUSusesaper-questionedgebudget
of64,witheachedgebeingamaximumof256tokens.
90
70
50
0 8 16 24 32
Completed updates
)%(
ycaruccA
12k
8k
PPO 4k
GRPO FOCUS
0
0 8 16 24 32
Completed updates
(a)Accuracy.
snekoT
0.06
0.04
0.02
0.00
0 8 16 24 32
Completed updates
(b)Tokens.
noitcuder
egarevA
tuollor
rep
(c)Per-rolloutreduction.
Figure2: PreliminaryevaluationwithQwen3.5-4BonMATHLevel4–5. Accuracyandthemean
numberofgeneratedtokensperevaluationquestionareshownatthebasecheckpoint(update0)and
after8, 16, 24, and32completedupdates. Markersdenoterecordedcheckpoints. Panel(c)shows
FOCUS’s mean g(c) over all pre-allocation score evaluations within each update from 1 to 32;
shadingdenotes±1standarddeviationofthosescores,computedasthesquarerootofthereported
variance.
FOCUShasthehighestaccuracyatupdates8,16,and24(Figure2),andisonlyonecorrectanswer
behindGRPOatupdate32.FOCUSimprovesaccuracyearlierbyupdatecount,hintingatatraining-
computeadvantage. Alsoatupdate32,FOCUSgenerates2,782tokensperevaluationproblemon
average, compared with 5,747 for GRPO and 7,207 for PPO. We hypothesize this results from
FOCUS’sabilitytorefinevalueestimatesmoreeffectively.
Figure2cshowsthatthroughouttrainingg(c)decreasesbyabout50%fromupdate1toupdate32,
showingthatasFOCUSgraduallytunesitscritic, theexpectedreductioninvalueestimationerror
decreases,suggestingthatFOCUSgraduallylearnstopredicttheexpectedoutcomemoreaccurately.
3.2 LONG-HORIZONDATASETS
DeonticBench (Dou et al., 2026a) DeonticBench evaluates models on context-grounded deontic
reasoning: deriving obligations, permissions, and prohibitions from real statutes and case facts. It
comprises 6,232 tasks across U.S. federal tax, airline baggage policy, immigration administration,
andstatehousinglaw.Applyingalongstatutetoaspecificcaseisamulti-stepprocess,whichmakes
thebenchmarkasettingforlong-horizonagenticreasoninginwhichthemodelconsultsthestatutes
ondemand(Douetal.,2026b)andanaturaltestbedforstepwiserollout. WefocusontheAIRLINE,
SARA-NUMERIC,and SARA-BINARY subsets,whosestatutesarereasonablylongandinvolvenon-
trivial long-horizon tasks. The actor harness exposes list, grep, read, and python. We
stratifythedataintoequallysizedtrainingandheld-outsplits.
BrowseComp-Plus (Chen et al., 2026) BrowseComp-Plus evaluates deep search agents on com-
plex, multi-hop information-seeking queries over a fixed, human-verified corpus rather than a live
webAPI.Theresultingtrajectoriesarelong-horizon: iterativequeryformulation,retrieval,andre-
flection,whichmakesthebenchmarkanaturaltestbedforstepwiserollout. FollowingTRACE(Tao
et al., 2026), the actor harness exposes search, open, and find over the fixed corpus. We
randomlysplitthe830testtasksinto498train,83validation,and249held-outtasks.
LocBench(Chenetal.,2025)LocBenchisacode-localizationbenchmarkof560realGitHubis-
sues (242 bug reports, 150 feature requests, 139 performance issues, and 29 security issues) from
repositories such as transformers, scikit-learn, vllm, and ultralytics; in LocA-
gent’sconstruction,bugreportsareissuescreatedrelativelyrecently,andsecurityandperformance
instances are collected via GitHub keyword search. Given only the issue, the agent must locate
3https://huggingface.co/datasets/EleutherAI/hendrycks_math
6
the files and functions touched by the gold patch. We instantiate it using the harness described in
AppendixH;theactorexposes list, grep,and read forrepositorynavigation. Wescoresub-
missionsasinLocAgent:file-levelandfunction-levelAcc@kis1wheneverygoldlocationappears
in the top k (added functions are excluded); we report Acc@{1,5} at the file level and train on
file-levelrecallatk=5. Wesplittheinstancesintoequaltrainingandheld-outhalves,stratifiedby
repositoryandissuecategory.
3.3 MAINRESULTS
We compare GRPO, VinePPO, and FOCUS using Qwen3.5-9B on DeonticBench, BrowseComp-
Plus,andLocBench,andadditionallywithQwen3.5-27B(LoRA)onDeonticBench.Table1reports
held-out accuracy alongside untrained reference models. All methods train for one epoch under a
matchedper-stepgenerationbudgetwithcompactiononeveryrollout; protocol, optimization, and
computedetailsareinAppendixA.
Table 1: Evaluation results for FOCUS vs critic-free methods. Entries are accuracy (%) on the
held-outsplitofeachbenchmark. DeonticBenchHardaggregatesdataset-markedhardsubsetsfrom
Airline, SARA-N, and SARA-B. Reference rows are untrained. ∆ is FOCUS minus GRPO in
percentagepoints;subscriptsandsuperscriptsgivethelowerandupperendpointsofmarginal90%
paired bootstrap CIs, respectively. Bootstrap and checkpoint details are in Appendix B. For cost
reasons,weonlyreportDeonticBenchatQwen3.5-27B.
| | | | | DeonticBench | | | BrowseComp | LocBench | |
| ----- | ------ | ------- | ------ | ------------ | ---- | ------- | ---------- | -------- | ------ |
| Model | Method | | | | | | -Plus | | |
| | | Airline | SARA-N | SARA-B | Hard | Overall | | File@1 | File@5 |
Reference(noRL)
| gpt-oss-20b(medium) | | 2.7 | 8.0 | 76.8 | 14.1 | 33.7 | 18.6 | 54.0 | 57.5 |
| ------------------- | ------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| Nemotron3Nano30BA3B | | 2.7 | 12.0 | 89.9 | 18.3 | 36.1 | 19.3 | 57.9 | 66.6 |
| Qwen3.5-9B | | 5.3 | 30.0 | 89.9 | 19.7 | 45.9 | 29.7 | 62.1 | 67.8 |
| Qwen3.5-27B | | 58.7 | 60.0 | 94.2 | 43.7 | 73.4 | 50.6 | 66.7 | 79.3 |
| | GRPO | 14.0 | 56.0 | 92.8 | 28.2 | 52.4 | 48.2 | 70.1 | 80.5 |
| Qwen3.5-9B | VinePPO | 10.0 | 50.0 | 92.0 | 24.0 | 49.4 | 46.6 | 67.8 | 79.3 |
| | FOCUS | 21.3 | 52.0 | 96.4 | 26.8 | 56.5 | 55.4 | 72.4 | 81.6 |
∆vs.GRPO +7.3+14.00 −4.0+8.00 +3.6+7.25 −1.4+5.63 +4.1+7.69 +7.2+12.05 +2.3+5.7 +1.1+3.4
| | | +0.67 | −16.00 | +0.00 | −8.45 | +0.59 | +2.41 | −1.1 | +0.0 |
| --- | ---- | ----- | ------ | ----- | ----- | ----- | ----- | ---- | ---- |
| | GRPO | 60.7 | 60.0 | 94.2 | 43.7 | 74.3 | | | |
Qwen3.5-27B VinePPO 62.7 58.0 94.2 36.6 74.9 outsidethe27Bstudy
| | FOCUS | 62.7 | 62.0 | 96.4 | 43.7 | 76.3 | | | |
| --- | -------- | --------- | ---------- | --------- | --------- | --------- | --- | --- | --- |
| | | +2.0+8.67 | +2.0+10.00 | +2.2+4.35 | +0.0+7.04 | +2.0+5.33 | | | |
| | ∆vs.GRPO | −4.67 | −6.00 | +0.72 | −7.04 | −1.18 | | | |
FOCUSranksfirstamongthetrainedmethodsoneverybenchmarkinTable1. AtQwen3.5-9B,it
leadsonDeonticBenchoverall(56.5vs.52.4forGRPOand49.4forVinePPO),reportsthestrongest
BrowseComp-Plus result (55.4 vs. 48.2 for GRPO and 46.6 for VinePPO, each the best held-out
checkpointovertraining),andlocalizesbestonLocBench(72.4/81.6File@1/@5,aheadof70.1/
80.5and67.8/79.3). FOCUSalsoleadsacrossscale: Qwen3.5-27BwithLoRAonDeonticBench
stillgainsfromFOCUSfine-tuning(76.3overallvs.74.3and74.9),butthemarginoverthebaselines
shrinks. This could partially be because the untuned 27B model already starts much higher (73.4
overall).
Training dynamics. FOCUS attains the highest on-policy reward throughout training on Deon-
ticBench and BrowseComp-Plus (Figure 3), and the highest held-out accuracy on DeonticBench.
On DeonticBench and LocBench, the three methods track closely early, then FOCUS pulls ahead
and keeps improving or holds a higher plateau, whereas GRPO on DeonticBench peaks mid-run
and then regresses. Late in training, GRPO’s mean tool calls collapse on all three benchmarks
whileVinePPOandFOCUSstayneartheirinitiallevels: GRPOincreasingly submitsrightafter
compaction even when the statute or repository still needs reading (Appendix G). We hypothe-
sizethattrajectory-levelgroupcreditmakessuchprematuresubmissionseasytoreinforce,whereas
VinePPO’ssuffixbranchingandFOCUS’srefinedcheckpointvaluespreservelongerhorizons.
7
BrowseComp-Plus
| | DeonticBench | | | | | | | | LocBench | |
| ------------------------ | ------------ | ------- | ----- | --------------------------- | --- | ------------- | ------------------------ | ---- | -------- | ----- |
| | | | | GRPO | | VinePPO FOCUS | | | | |
| GRPO | | VinePPO | FOCUS | | | | | GRPO | VinePPO | FOCUS |
| )%( drawer lanimret naeM | | | | )%( drawer lanimret naeM 80 | | | 100 | | | |
| 80 | | | | | | | )%( drawer lanimret naeM | | | |
90
70
| | | | | 60 | | | 80 | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
60
70
| 50 | | | | 40 | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
60
40
| | | | | 20 | | | 50 | | | |
| --- | ----- | ----- | ----- | --- | -------- | ----------- | --- | ------- | ----- | ----- |
| 30 | | | | 0 | 12 24 36 | 48 60 72 84 | | | | |
| 0 | 12 24 | 36 48 | 60 72 | | | | | 0 12 24 | 36 48 | 60 71 |
Actor update
| | Actor update | | | | | | | Actor update | | |
| ---- | ------------ | ------- | ----- | ---- | --------------- | ------------- | --- | ------------ | ------- | ----- |
| | DeonticBench | | | | BrowseComp-Plus | | | LocBench | | |
| | | | | GRPO | | VinePPO FOCUS | | | | |
| GRPO | | VinePPO | FOCUS | | | | | GRPO | VinePPO | FOCUS |
| 60 | | | | | | | 85 | | | |
55
| )%( ycaruccA 55 | | | | )%( ycaruccA | | | 80 | | | |
| --------------- | --- | --- | --- | ------------ | --- | --- | ---------- | --- | --- | --- |
| | | | | 50 | | | )%( 5@eliF | | | |
| 50 | | | | | | | 75 | | | |
45
45
70
40
| 40 | | | | | | | 65 | | | |
| ---- | ------------ | ------- | ----- | ---- | --------------- | ------------- | --- | ------------ | ------- | ----- |
| 0 | 12 24 | 36 48 | 60 72 | 0 | 12 24 36 | 48 60 72 84 | | 0 17 | 35 47 | 59 71 |
| | Actor update | | | | Actor update | | | Actor update | | |
| | DeonticBench | | | | BrowseComp-Plus | | | LocBench | | |
| GRPO | | VinePPO | FOCUS | GRPO | | VinePPO FOCUS | | GRPO | VinePPO | FOCUS |
| 35 | | | | 65 | | | | | | |
35
| sllac loot naeM | | | | sllac loot naeM 60 | | | sllac loot naeM | | | |
| --------------- | --- | --- | --- | ------------------ | --- | --- | --------------- | --- | --- | --- |
30
| | | | | 55 | | | 30 | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 25 | | | | 50 | | | 25 | | | |
20
| 20 | | | | 45 | | | | | | |
| --- | ----- | ----- | ----- | --- | -------- | ----------- | --- | ---- | ----- | ----- |
| 0 | 12 24 | 36 48 | 60 72 | 0 | 12 24 36 | 48 60 72 84 | | 0 17 | 35 47 | 59 71 |
Actor update
| | Actor update | | | | | | | Actor update | | |
| --- | ---------------- | --- | --- | ------------------- | --- | --- | --- | ------------ | --- | --- |
| | (a)DeonticBench. | | | (b)BrowseComp-Plus. | | | | (c)LocBench. | | |
Figure 3: Training dynamics. Each column is one benchmark, showing training reward against
optimizer step (top), held-out evaluation (middle), and mean numbers of tool calls per episode on
theheld-outsplit(bottom),forGRPO,VinePPO,andFOCUSunderamatchedper-stepgeneration
budget.Toppanelsplotmeanterminalrewardfromon-policycollection;LocBenchheld-outreward
isFile@5. BrowseComp-PlusresultsaretheequivalencejudgementofQwen3-32B(QwenTeam,
2025).
3.4 VALUEREFINEMENT
ToassesshowefficienttheFOCUScorrectionprocessisinreducingthecriticerrorforlong-horizon
tasks, as well as whether the critic can improve throughout training, we conduct two diagnostic
analyses.
CriticErrorReductionWestartbyassumingtwotypesofimperfectcritics: aconstantcriticinit
and a random critic init, and we compare how different priority schedules affect the critic error
reduction against a ground truth determined by the mean outcome of n=16 MCTS continuations.
Figure 4a shows that under both types of critics, FOCUS’s correction process is noticeably more
efficient than random selection. Also, using the raw critic value residual performs poorly. This
showsthatFOCUS-styleschedulingnotonlyworkswellunderdegeneratecriticsbutalsoreduces
varianceevenwhenthecriticprovidesalmostnoinformation.
Critic Improvement We also test whether the critic improves over training. At each LocBench
evaluationcheckpoint,wesamplen=16MCTScontinuationsfromasetofm=8completedtraces.
We then compute the Spearman correlation between the critic value and the MCTS ground truth.
Figure4bshowsthattheFOCUScriticimprovessteadily.
8
0.08
0.06
0 10 20 30 40 50
Additional rollouts per actor update
ESM
naeM
0.6
0.4
FOCUS
Random
Residual 0.2
0 17 35 47 59 71
Actor update
(a)CriticMSEvs.budget(DeonticBench).
½
namraepS
(b)CriticSpearmanρ(LocBench).
Figure 4: Value refinement for FOCUS. (a) MSE on fixed audit trees as MCTS continuations are
allocated by each rule (solid: constant critic init; dashed: random critic init). (b) Rank agreement
betweencheckpointcriticvaluesandanoversampled,critic-blindMCTStargetonfixedauditstates.
3.5 ABLATIONSTUDIES
WeablatethecompactionoperatorandtherefinementrolloutstotestthatFOCUSisnottiedtoone
compactorandthatitsgainscomefromactiveerrorcorrection,notthecriticalone.
Compaction operator. On DeonticBench, we replace the in-context compaction operator, which
writeseverythingintothenewcontextwindow,withanin-memoryworkspacethatprovidesitsown
put, get, scan and delete.Thetwoperformsimilarly(Figure5a),soFOCUS’seffectiveness
extendstodifferentcompactionoperators.
Refinementrollouts. OnBrowseComp-Plus,weremovetherefinementrolloutsofsubsection2.3,
which amounts to collecting independent continuations from the question prompt and estimating
step-wiseadvantagesfromthecriticalone. Figure5bshowsthatrefinementrolloutsarecrucialfor
FOCUSonBrowseComp-Plus;withoutthem,FOCUStrainsmoreslowly,especiallyearlyon.
Structuralcompaction. Wealsocompareafree-formcompactionpromptwithastructuralprompt
thatinducesamoreconstrainedcompactiononLocBench(Figure5c;bothpromptsarereproduced
inAppendixD).Thefree-formpromptsuggestsheadingsandkeepsthereplyverbatim. Thestruc-
turalpromptrequiresasinglesummarywithfoursections(State, Evidence, Openquestions, Next
steps)andrejectsrepliesthatdonotparse. Thefree-formpromptreachesahigherpeak,whilethe
structuralpromptimprovesmoresteadily.
In-context In-memory
55
50
45
0 12 24 36 48 60 72
Actor update
)%(
ycaruccA
FOCUS FOCUS w/o refinement
55
50
45
40
0 12 24 36 48 60 72 84
Actor update
(a)Compactionoperator
(DeonticBench).
)%(
ycaruccA
free-form structural
80
75
0 17 35 47 59 71
Actor update
(b)Refinementrollouts
(BrowseComp-Plus).
)%(
5@ccA
eliF
(c)Structuralcompaction
(LocBench).
Figure 5: Ablations at evaluation checkpoints. (a) Varying the compaction operator on Deon-
ticBenchunderamatchedbudget;(b)FOCUSwithfullrefinementvs.FOCUSwithoutrefinement
rolloutsonBrowseComp-Plus;(c)free-formvs.structuralcompactiononLocBench(FileAcc@5).
4 RELATED WORK
StructuredrolloutsandrolloutallocationSearchoverintermediatestatesbeganasaninference-
timetechnique:LLMtreesearch(Yaoetal.,2023;Haoetal.,2023;Zhouetal.,2024a)andverifier-
guided test-time scaling (Lightman et al., 2024; Snell et al., 2025) improved model responses. It
was laterused offlineto generatetraining supervision(Tian etal., 2024;Zhang et al.,2024; Guan
etal.,2025)andthenmovedintotheRLloop. VinePPO(Kazemnejadetal.,2025),SPO(Guoetal.,
9
2025), and TEMPO (Tran et al., 2025) re-sample continuations from intermediate reasoning steps
to replace the value network with Monte Carlo estimates; TreeRL (Hou et al., 2025), TreePO (Li
etal.,2026d),andDeepSearch(Wuetal.,2026)runthesearchinsideRLVRtrainingsothatshared
prefixes cut generation cost and the tree yields process-level signal. Other work proposes where
to branch: at high-entropy tokens (Zheng et al., 2025; Cao et al., 2026), at attention peaks (Liu
et al., 2026), where the model’s belief shifts (Lei et al., 2026), or, for tool-using agents, at the
turn boundaries after an observation (Dong et al., 2026; Zong et al., 2026). Tree-GRPO (Ji et al.,
2026)makeseachagentstepanodeandshowsthatintra-treegroupobjectivesamounttostep-level
preferencelearning, whilePaTR(Lietal.,2026c)routesthebranchingbudgetbyaprocessscore,
andIGRPO(Zhangetal.,2026b)andSIGHT(Zhongetal.,2026)byinformationgain.T2PO(Wang
etal.,2026a)insteadresamplesturnsthatitstoken-andturn-leveluncertaintysignalsflagasmaking
no progress. TRACE (Zou et al., 2026) unifies prompt- and turn-level allocation with a learned
predictor of conditional success, directing extra rollouts to the prefixes most likely to yield mixed
terminalrewards. FOCUSallocatesadditionalcontinuationsaccordingtotheirestimatedreduction
invalue-estimationerror,usingtheacquiredevidencetorefinestatevaluesandtrainareusablecritic.
ActivelearningandexpectederrorreductionExpected-error-reductionmethodsselectobserva-
tionsaccordingtotheirpredictedeffectonthelearner’serror.Roy&McCallum(2001)approximate
thisobjectivethroughsampling,whileMussmannetal.(2022)useBayesianparametersamplesto
estimatethereductioninpredictivelossoveranunlabeledvalidationsetwithoutretrainingforevery
candidate query. A related approach selects data to reduce predictive variance, connecting vari-
ancereductiontosquared-errorminimizationunderanapproximatelyunbiasedlearner(Cohnetal.,
1996). Influence-functionapproachesestimateacquisitionutilitybyapproximatinghowadditional
training data affect a fitted model’s evaluation loss, without retraining for each candidate (Koh &
Liang, 2017; Xu & Kazantsev, 2019). FOCUS applies the error-reduction principle to allocating
continuationsamplesamongintermediatestates.
CriticlearningandcreditassignmentTrajectory-levelrewardisknowntofailformulti-turnagents
asitcannottellausefulturnfromaredundantone,andtrainingonitaloneisunstable(Wangetal.,
2025;Weietal.,2025;Tanetal.,2026). Oneresponseistoplacetherewardandcriticatthelevel
ofaturnorstepratherthanatoken(toofine-grained)orawholetrajectory(toocoarse)(Zhouetal.,
2024c;2025;Lietal.,2026a;Zhangetal.,2026a;Zhenetal.,2026). Aturnisthenaturalunitof
decision for an agent; the horizon measured in turns is short enough for a critic to bootstrap over,
andaper-turnvaluegiveseveryturnitsownsignalfromasinglerollout. Large-scaletrainingisalso
shifting from structural comparison to a learned value: SAO (Hou et al., 2026) replaces GRPO’s
group sampling with one rollout per prompt scored by a value model, as full groups are slow and
off-policyunderasynchronoustraining.Someprocessrewardmodelsserveasfixedvalueestimators
duringpolicyoptimization(Wangetal.,2024;Luoetal.,2024;Li&Li,2025;Choudhury,2025;
Setluretal.,2025;Taoetal.,2026). Morerecently, LeCritique(Venkatramanetal.,2026)mixes
a learned baseline with a training-free group-relative one according to how accurate the critic is.
FOCUSusesrefinedstatevaluestoconstructsegment-levelactoradvantagesandrecursivebackups
to supervise the critic, coupling credit assignment in the current iteration with critic learning for
subsequentiterations.
Context management for long-horizon agents Compaction is becoming standard for agents that
outruntheircontextwindow(Packeretal.,2023;Yuetal.,2026;Sunetal.,2026;Lietal.,2026b),
and thereis a recent pushto train it jointlywith the task ratherthan treat itas a frozen preproces-
sor(Cursor,2026;Lietal.,2026e). Staterestorationisalsocentraltoidentifyingstep-levelcredit
(Zhang,2026). Inourlong-horizonexperiments, compactionboundariesprovideresumablestates
for acquiring continuation evidence. The rollout controller operates on these states independently
ofthecompactionoperator;itsobjectiveistorefinevalueestimatesandconstructcriticsupervision
throughselectiverollouts.
5 CONCLUSION
WepresentFOCUS,arolloutcontrollerforlanguage-modelreinforcementlearningthatrefinesin-
termediate value estimates through active critic learning and selective rollouts. FOCUS naturally
supportslong-horizonagenticRLthroughflexibleandmodularcontext-managementprotocolssuch
as compaction. Across DeonticBench, BrowseComp-Plus, and LocBench, FOCUS improves over
10
GRPO and VinePPO under a matched per-step generation budget, with the largest held-out gains
ondeepsearchanddeonticreasoningandconsistentbenefitsoncodelocalization. Trainingcurves
show that these improvements come with more stable exploration: FOCUS maintains higher col-
lection reward and avoids the late-training collapse in tool use seen under trajectory-level group
credit. Ablations indicate that gains are not tied to a single compaction implementation and that
refinementrolloutsarebeneficialonBrowseComp-Plus. Wereleaseourmodularrolloutcontroller
with interfaces for alternative harnesses, search policies, and credit estimators. Future work may
scaleFOCUStolargermodelsandlongerbudgets,tightenthelinkbetweencompactionqualityand
trainingstability,andadoptstrongeruncertaintyquantificationforvaluerefinement.
ACKNOWLEDGMENTS
ThisworkwassupportedinpartbytheDefenseAdvancedResearchProjectsAgency(DARPA)Sci-
entificFeasibility(SciFy)programundercontractHR001125C0304;bySchmidtSciencesthrough
the“SafelyConfidentReasoning”projectinitsScienceofTrustworthyAIprogram;andbytheOf-
ficeofNavalResearch(ONR)undergrantN00014-24-1-2089. Anyopinions,findings,andconclu-
sionsorrecommendationsexpressedinthismaterialarethoseoftheauthorsanddonotnecessarily
reflecttheviewsofDARPA,theOfficeofNavalResearch,orSchmidtSciences. Weacknowledge
theuseofcomputationalresourcesfromtheJohnsHopkinsDSAIcluster.
REFERENCES
Lang Cao, Hui Ruan, Yongqian Li, Peng Chao, Wu Ning, Haonan Song, Renhong Chen, and Yi-
tongLi. TreeAdv: Tree-structuredadvantageredistributionforgroup-basedRL. arXivpreprint
arXiv:2601.03703,2026.
ZhaolingChen,XiangruTang,GangdaDeng,FangWu,JialongWu,ZhiweiJiang,ViktorPrasanna,
ArmanCohan,andXingyaoWang. LocAgent: Graph-guidedLLMagentsforcodelocalization.
In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics,
2025. arXiv:2503.09089.
ZijianChen,XueguangMa,ShengyaoZhuang,PingNie,KaiZou,SahelSharifymoghaddam,An-
drewLiu,JoshuaGreen,KshamaPatel,RuoxiMeng,MingyiSu,YanxiLi,HaoranHong,Xinyu
Shi,XuyeLiu,HosnaOyarhoseini,NandanThakur,CrystinaZhang,LuyuGao,WenhuChen,and
Jimmy Lin. BrowseComp-plus: A fair and disentangled evaluation benchmark for deep search
agents. In Proceedings of the 64th Annual Meeting of the Association for Computational Lin-
guistics(Volume1: LongPapers), pp.22349–22370, SanDiego, California, UnitedStates, July
2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.acl-long.1023. URL
https://aclanthology.org/2026.acl-long.1023/.
SanjibanChoudhury. ProcessrewardmodelsforLLMagents: Practicalframeworkanddirections.
arXivpreprintarXiv:2502.10325,2025.
DavidA.Cohn,ZoubinGhahramani,andMichaelI.Jordan. Activelearningwithstatisticalmodels.
Journal of Artificial Intelligence Research, 4:129–145, 1996. URL https://arxiv.org/
abs/cs/9603104.
Cursor. Training Composer for longer horizons. https://cursor.com/blog/
self-summarization,2026. Blogpost,March2026.
CursorResearch. Composer2technicalreport. arXivpreprintarXiv:2603.24477,2026.
Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia
Chen,JiazhenDu,HuiyangWang,FuzhengZhang,GuoruiZhou,YutaoZhu,Ji-RongWen,and
ZhichengDou. Agenticreinforcedpolicyoptimization. InInternationalConferenceonLearning
Representations,2026. arXiv:2507.19849.
Guangyao Dou, Luis Brena, Akhil Deo, William Jurayj, Jingyu Zhang, Nils Holzenberger, and
Benjamin Van Durme. DeonticBench: A benchmark for reasoning over rules. arXiv preprint
arXiv:2604.04443,2026a.
11
GuangyaoDou,WilliamJurayj,NilsHolzenberger,andBenjaminVanDurme. DAR:Deonticrea-
soningwithagenticharnesses. arXivpreprintarXiv:2606.05009,2026b.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao
Yang. rStar-Math: SmallLLMscanmastermathreasoningwithself-evolveddeepthinking. In
InternationalConferenceonMachineLearning,2025. arXiv:2501.04519.
Yiran Guo, Lijie Xu, Jie Liu, Dan Ye, and Shuang Qiu. Segment policy optimization: Effective
segment-levelcreditassignmentinRLforlargelanguagemodels. InAdvancesinNeuralInfor-
mationProcessingSystems,2025. arXiv:2505.23564.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting
Hu. Reasoningwithlanguagemodelisplanningwithworldmodel. InProceedingsofthe2023
ConferenceonEmpiricalMethodsinNaturalLanguageProcessing,2023.
Harvey. Harvey Tenet research preview. https://www.harvey.ai/blog/
post-training-update-harvey-tenet,2026. Blogpost,August2026.
NicholasHay, StuartRussell,DavidTolpin, andSolomonEyalShimony. Selectingcomputations:
Theoryandapplications. InConferenceonUncertaintyinArtificialIntelligence,2012.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn
Song, and Jacob Steinhardt. Measuringmathematical problem solving with the MATH dataset.
NeurIPS,2021. URLhttps://arxiv.org/abs/2103.03874.
ZhenyuHou,ZiniuHu,YujiangLi,RuiLu,JieTang,andYuxiaoDong. TreeRL:LLMreinforce-
ment learning with on-policy tree search. In Proceedings of the 63rd Annual Meeting of the
AssociationforComputationalLinguistics,2025. arXiv:2506.11902.
ZhenyuHou,YujiangLi,JieTang,andYuxiaoDong. Single-rolloutasynchronousoptimizationfor
agenticreinforcementlearning. arXivpreprintarXiv:2607.07508,2026.
YuxiangJi,ZiyuMa,YongWang,GuanhuaChen,XiangxiangChu,andLiaoniWu. Treesearchfor
LLM agent reinforcement learning. In International Conference on Learning Representations,
2026. arXiv:2509.21240.
CarlosE.Jimenez, JohnYang, AlexanderWettig, ShunyuYao, KexinPei, OfirPress, andKarthik
Narasimhan. SWE-bench: Canlanguagemodelsresolvereal-worldGitHubissues? InInterna-
tionalConferenceonLearningRepresentations,2024. arXiv:2310.06770.
BowenJin,HansiZeng,ZhenruiYue,JinsungYoon,SercanArik,DongWang,HamedZamani,and
JiaweiHan.Search-R1:TrainingLLMstoreasonandleveragesearchengineswithreinforcement
learning. InConferenceonLanguageModeling,2025. arXiv:2503.09516.
Emilie Kaufmann and Wouter M. Koolen. Monte-Carlo tree search by best arm identification. In
AdvancesinNeuralInformationProcessingSystems,2017.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy,
Aaron Courville, and Nicolas Le Roux. VinePPO: Refining credit assignment in RL training
ofLLMs. InInternationalConferenceonMachineLearning,2025.
KimiTeam. KimiK3: Openfrontierintelligence. arXivpreprintarXiv:2607.24653,2026.
DiederikP.KingmaandJimmyBa. Adam: Amethodforstochasticoptimization. InInternational
ConferenceonLearningRepresentations,2015.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions.
In Proceedings of the 34th International Conference on Machine Learning, volume 70 of
Proceedings of Machine Learning Research, pp. 1885–1894. PMLR, 2017. URL https:
//proceedings.mlr.press/v70/koh17a.html.
VijayKondaandJohnTsitsiklis. Actor-criticalgorithms. InAdvancesinNeuralInformationPro-
cessingSystems,volume12,1999.
12
Bin Lei, Yu Li, Prafulla Kumar Choubey, Jiaxin Zhang, Becky Xiangyu Peng, Qinyuan Ye, Kar-
tik Narayan, Caiwen Ding, Silvio Savarese, and Chien-Sheng Wu. Fork where the model
changesitsmind:Belief-shiftbranchingfortree-structuredreinforcementlearning.arXivpreprint
arXiv:2609.11061,2026.
Junbo Li, Peng Zhou, Rui Meng, Meet P. Vadera, Lihong Li, and Yang Li. Turn-PPO: Turn-level
advantageestimationwithPPOforimprovedmulti-turnRLinagenticLLMs. InProceedingsof
the19thConferenceoftheEuropeanChapteroftheAssociationforComputationalLinguistics,
2026a. arXiv:2512.17008.
Tianjian Li, Jingyu Zhang, William Jurayj, Xi Wang, Chuanyang Jin, Mehrdad Farajtabar, Eric
Nalisnick, and Daniel Khashabi. Self-compacting language model agents. arXiv preprint
arXiv:2606.23525,2026b.
WendiLiandYixuanLi. ProcessrewardmodelwithQ-valuerankings. InInternationalConference
onLearningRepresentations,2025. arXiv:2410.11287.
XintongLi,ShaLi,YuweiZhang,ChanglongYu,RongmeiLin,HongyeJin,ShuyiGuan,XinLiu,
Linwei Li, Qingyu Yin, and Jingbo Shang. Process reward informed tree rollout for effective
multi-turnRL. arXivpreprintarXiv:2607.15610,2026c.
YizhiLi, QingshuiGu, ZhoufutuWen, ZiniuLi, TianshunXing, ShuyueGuo, TianyuZheng, Xin
Zhou,XingweiQu,WangchunshuZhou,ZhengZhang,WeiShen,QianLiu,ChenghuaLin,Jian
Yang,GeZhang,andWenhaoHuang.TreePO:Enhancingpolicyefficacyandinferenceefficiency
withtreemodeling.InInternationalConferenceonMachineLearning,2026d.arXiv:2508.17445.
Yujiang Li, Zhenyu Hou, Yi Jing, Jie Tang, and Yuxiao Dong. CompactionRL: Reinforcement
learning with context compaction for long-horizon agents. arXiv preprint arXiv:2607.05378,
2026e.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan
Leike,JohnSchulman,IlyaSutskever,andKarlCobbe. Let’sverifystepbystep. InInternational
ConferenceonLearningRepresentations,2024.
RunzeLiu,JiakangWang,YulingShi,ZhihuiXie,ChenxinAn,KaiyanZhang,JianZhao,Xiaodong
Gu, Lei Lin, Wenping Hu, Xiu Li, Fuzheng Zhang, Guorui Zhou, and Kun Gai. Attention as a
compass: Efficientexplorationforprocess-supervisedRLinreasoningmodels. InInternational
ConferenceonLearningRepresentations,2026. arXiv:2509.26628.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li,
LeiShu,YunZhu,LeiMeng,JiaoSun,andAbhinavRastogi. Improvemathematicalreasoning
inlanguagemodelsbyautomatedprocesssupervision. arXivpreprintarXiv:2406.06592,2024.
StephenMussmann,JuliaReisler,DanielTsai,EhsanMousavi,ShayneO’Brien,andMoisesGold-
szmidt. Activelearningwithexpectederrorreduction. arXivpreprintarXiv:2211.09283, 2022.
URLhttps://arxiv.org/abs/2211.09283.
CharlesPacker,SarahWooders,KevinLin,VivianFang,ShishirG.Patil,IonStoica,andJosephE.
Gonzalez. MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560,
2023.
QwenTeam. Qwen3technicalreport,2025. URLhttps://arxiv.org/abs/2505.09388.
QwenTeam.Qwen3.5:Towardsnativemultimodalagents,February2026.URLhttps://qwen.
ai/blog?id=qwen3.5.
NicholasRoyandAndrewMcCallum. Towardoptimalactivelearningthroughsamplingestimation
oferrorreduction. InProceedingsoftheEighteenthInternationalConferenceonMachineLearn-
ing, pp. 441–448. Morgan Kaufmann, 2001. URL https://groups.csail.mit.edu/
rrg/papers/icml01.pdf.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-
dimensional continuous control using generalized advantage estimation. In International Con-
ferenceonLearningRepresentations,2016.
13
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal,
Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated
processverifiersforLLMreasoning. InInternationalConferenceonLearningRepresentations,
2025. arXiv:2410.08146.
Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li, Minheng Wang, Hamish Ivison,
Radha Poovendran, Nathan Lambert, Teng Xiao, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer,
and Pang Wei Koh. Context language models. arXiv preprint arXiv:2609.37725, 2026. URL
https://arxiv.org/abs/2609.37725.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang,
MingchuanZhang,Y.K.Li,Y.Wu,andDayaGuo. DeepSeekMath: Pushingthelimitsofmath-
ematicalreasoninginopenlanguagemodels. arXivpreprintarXiv:2402.03300,2024.
DavidSilver,AjaHuang,ChrisJ.Maddison,ArthurGuez,LaurentSifre,GeorgevandenDriessche,
JulianSchrittwieser,IoannisAntonoglou,VedaPanneershelvam,MarcLanctot,etal. Mastering
thegameofGowithdeepneuralnetworksandtreesearch. Nature,529(7587):484–489,2016.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute opti-
mallycanbemoreeffectivethanscalingparametersforreasoning. InInternationalConference
onLearningRepresentations,2025. arXiv:2408.03314.
WeiweiSun,MiaoLu,ZhanLing,KangLiu,XuesongYao,YimingYang,andJiecaoChen.Scaling
long-horizonagentviacontextfolding. InInternationalConferenceonMachineLearning,2026.
arXiv:2510.11967.
Hui-ZeTan,Xiao-WenYang,HaoChen,Jie-JingShao,YiWen,YutengShen,WeihongLuo,Xiku
Du, Lan-Zhe Guo, andYu-FengLi. Hindsightcreditassignment forlong-horizonLLM agents.
arXivpreprintarXiv:2603.08754,2026.
LeitianTao, BaolinPeng, WenlinYao, TaoGe, HaoCheng, MikeHangWang, JianfengGao, and
SharonLi. TRACE:Turn-levelrewardassignmentviacreditestimationforlong-horizonagents.
arXivpreprintarXiv:2607.13988,2026.
GeraldTesauro,V.T.Rajan,andRichardSegal. BayesianinferenceinMonte-Carlotreesearch. In
ConferenceonUncertaintyinArtificialIntelligence,2010.
ThinkingMachinesLab. Tinker,2026. URLhttps://thinkingmachines.ai/tinker/.
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Lei Han, Haitao Mi, and Dong Yu.
Toward self-improvement of LLMs via imagination, searching, and criticizing. In Advances in
NeuralInformationProcessingSystems,2024.
HieuTran,ZonghaiYao,andHongYu.ExploitingtreestructureforcreditassignmentinRLtraining
ofLLMs. arXivpreprintarXiv:2509.18314,2025.
SiddarthVenkatraman,MatthieuDinot,andLaurenceAitchison. Lecritique:Privilegedvaluefunc-
tionsforLLMreinforcementlearning. arXivpreprintarXiv:2608.16739,2026.
Haixin Wang, Hejie Cui, Chenwei Zhang, Xin Liu, Shuowei Jin, Shijie Geng, Xinyang Zhang,
Nasser Zalmout, Zhenyu Shi, and Yizhou Sun. T2PO: Uncertainty-guided exploration control
for stable multi-turn agentic reinforcement learning. In International Conference on Machine
Learning,2026a. arXiv:2605.02178.
PeiyiWang,LeiLi,ZhihongShao,R.X.Xu,DamaiDai,YifeiLi,DeliChen,Y.Wu,andZhifang
Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In
Proceedingsofthe62ndAnnualMeetingoftheAssociationforComputationalLinguistics,2024.
arXiv:2312.08935.
ZhentingWang,HuanchengChen,JiayunWang,andWeiWei. Memex(RL):Scalinglong-horizon
LLM agents via indexed experience memory. arXiv preprint arXiv:2603.04257, 2026b. URL
https://arxiv.org/abs/2603.04257.
14
ZihanWang,KangruiWang,QinengWang,PingyueZhang,LinjieLi,ZhengyuanYang,XingJin,
KefanYu,MinhNhatNguyen,LichengLiu,EliGottlieb,YipingLu,KyunghyunCho,JiajunWu,
LiFei-Fei,LijuanWang,YejinChoi,andManlingLi. RAGEN:Understandingself-evolutionin
LLMagentsviamulti-turnreinforcementlearning. arXivpreprintarXiv:2504.20073,2025.
QuanWei,SiliangZeng,ChenliangLi,ZhongruoWang,WilliamBrown,OanaFrunza,WeiDeng,
Anderson Schneider, Yuriy Nevmyvaka, Yang Katie Zhao, Alfredo Garcia, and Mingyi Hong.
Reinforcingmulti-turnreasoninginLLMagentsviafine-grainedrewardstructureandcreditas-
| signment. arXivpreprintarXiv:2505.11821,2025. | | | | |
| --------------------------------------------- | --- | --- | --- | --- |
FangWu,WeihaoXuan,HeliQi,AaronTu,XimingLu,LiErranLi,andYejinChoi. DeepSearch:
Overcomethebottleneckofreinforcementlearningwithverifiablerewardsviatree-basedsearch.
| InInternationalConferenceonLearningRepresentations,2026. | | | arXiv:2509.25454. | |
| -------------------------------------------------------- | --- | --- | ----------------- | --- |
MinjieXuandGaryKazantsev.Understandinggoal-orientedactivelearningviainfluencefunctions.
arXivpreprintarXiv:1905.13183,2019. URLhttps://arxiv.org/abs/1905.13183.
ZhichengYang,ZhijiangGuo,YinyaHuang,XiaodanLiang,YiweiWang,andJingTang.TreeRPO:
| Treerelativepolicyoptimization. | | arXivpreprintarXiv:2506.05183,2025. | | |
| ------------------------------- | --- | ----------------------------------- | --- | --- |
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik
Narasimhan. Treeofthoughts: Deliberateproblemsolvingwithlargelanguagemodels. InAd-
vancesinNeuralInformationProcessingSystems,2023.
HongliYu,TinghongChen,JiangtaoFeng,JiangjieChen,WeinanDai,QiyingYu,Ya-QinZhang,
Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, and Hao Zhou. MemAgent: Reshaping long-
contextLLMwithmulti-convRL-basedmemoryagent. InInternationalConferenceonLearning
| Representations,2026. | arXiv:2507.02259. | | | |
| --------------------- | ----------------- | --- | --- | --- |
Alex L. Zhang, Tim Kraska, and Omar Khattab. Recursive language models. arXiv preprint
| arXiv:2512.24601,2025. | URLhttps://arxiv.org/abs/2512.24601. | | | |
| ---------------------- | ------------------------------------ | --- | --- | --- |
ChenchenZhang. Fromreasoningtoagentic: Creditassignmentinreinforcementlearningforlarge
| languagemodels. | arXivpreprintarXiv:2604.09459,2026. | | | |
| --------------- | ----------------------------------- | --- | --- | --- |
DanZhang, SiningZhoubian, ZiniuHu, YisongYue, YuxiaoDong, andJieTang. ReST-MCTS*:
LLM self-training via process reward guided tree search. In Advances in Neural Information
ProcessingSystems,2024.
WenxuanZhang,YuhuiWang,DonggangJia,XiaoqianShen,JianDing,IvanViola,Ju¨rgenSchmid-
huber,andMohamedElhoseiny.HybridadvantageestimationwithunifiedcriticforVLMagentic
reinforcementlearning. InEuropeanConferenceonComputerVision,2026a. arXiv:2607.23605.
Yijun Zhang, Fan Xu, Jiaxin Ding, Yule Xie, Shiqing Gao, Xin Ding, Haoxiang Zhang, Luoyi
Fu, and Xinbing Wang. Information gain-based rollout policy optimization: An adaptive tree-
structured rollout approach for multi-turn LLM agents. arXiv preprint arXiv:2607.06223v1,
2026b. URLhttps://arxiv.org/abs/2607.06223v1.
Shuai Zhen, Yanhua Yu, Ruopei Guo, Nan Cheng, and Yang Deng. Hierarchical reinforcement
learningwithaugmentedstep-leveltransitionsforLLMagents.InProceedingsofthe64thAnnual
| MeetingoftheAssociationforComputationalLinguistics,2026. | | | arXiv:2604.05808. | |
| -------------------------------------------------------- | --- | --- | ----------------- | --- |
TianyuZheng,TianshunXing,QingshuiGu,TaoranLiang,XingweiQu,XinZhou,YizhiLi,Zhou-
futu Wen, Chenghua Lin, Wenhao Huang, Qian Liu, Ge Zhang, and Zejun Ma. First return,
| entropy-elicitingexplore. | arXivpreprintarXiv:2507.07017,2025. | | | |
| ------------------------- | ----------------------------------- | --- | --- | --- |
Wenlin Zhong, Jinluan Yang, Yiquan Wu, Yi Liu, Jianhang Yao, and Kun Kuang. SIGHT: Rein-
forcement learning with self-evidence and information-gain diverse branching for search agent.
arXivpreprintarXiv:2602.11551,2026.
AndyZhou,KaiYan,MichalShlapentokh-Rothman,HaohanWang,andYu-XiongWang.Language
International
| agent tree search | unifies reasoning, | acting, and planning | in language | models. In |
| ---------------------------------- | ------------------ | -------------------- | ----------- | ---------- |
| ConferenceonMachineLearning,2024a. | | arXiv:2310.04406. | | |
15
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng,
Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realis-
tic webenvironment for building autonomousagents. In International Conference on Learning
Representations,2024b. arXiv:2307.13854.
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. ArCHer: Training lan-
guage model agents via hierarchical multi-turn RL. In International Conference on Machine
Learning,2024c.
YifeiZhou,SongJiang,YuandongTian,JasonWeston,SergeyLevine,SainbayarSukhbaatar,and
Xian Li. SWEET-RL: Training multi-turn LLM agents on collaborative reasoning tasks. arXiv
preprintarXiv:2503.15478,2025.
ZilinZhu,ChengxingXie,XinLv,andslimeContributors.slime:AnLLMpost-trainingframework
forRLscaling,2025. URLhttps://github.com/THUDM/slime.
ZefangZong, DingweiChen, YangLi, QiYi, BoZhou, ChengmingLi, BoQian, PengChen, and
JieJiang. AT2PO:Agenticturn-basedpolicyoptimizationviatreesearch. InProceedingsofthe
64thAnnualMeetingoftheAssociationforComputationalLinguistics(Volume1: LongPapers),
2026. arXiv:2601.04767.
HemingZou, QiWang, YunQu, YuhangJiang, LizhouCai, YixiuMao, RuPeng, XinXu, Weijie
Liu, Kai Yang, Saiyong Yang, and Xiangyang Ji. TRACE: A unified rollout budget allocation
framework for efficient agentic reinforcement learning. In Proceedings of the 2026 Conference
onEmpiricalMethodsinNaturalLanguageProcessing,2026. arXiv:2606.11119.
16
A TRAINING
SETUP
Tables2–13givethetrainingsetupofeachmethodoneachbenchmark,onetableperrunconfigu-
ration. AdamisfromKingma&Ba(2015).
Shared FOCUS settings. Across all FOCUS experiments, we use γ = 1, λ = 0, and κ = 1.
Thus rewards are undiscounted, actor advantages use the local segment TD residual in Eq. 4, and
the critic contributes one pseudo-observation to both the recursive value blend and the selection
posterior. Total branching width is uncapped, with at most C = 2 concurrent continuations per
node; completed children do not count against this concurrency limit. These settings are shared
acrosstasksandmodelsizes. TheselectionestimatorsaredefinedinEq.6andAppendixE;thefull
trainingloopandbudgetsemanticsaregiveninAppendixF.
Table 2: FOCUS on BrowseComp-Plus, Table 3: GRPO on BrowseComp-Plus,
| Qwen3.5-9B. | | | Qwen3.5-9B. | | |
| ------------------ | --- | --------------- | ------------------ | --- | ------------------ |
| Setting | | Value | Setting | | Value |
| Protocol | | | Protocol | | |
| Actormodel | | Qwen3.5-9B | Actormodel | | Qwen3.5-9B |
| Criticmodel | | 9B | Traininglength | | 1epoch |
| Traininglength | | 1epoch | Training-dataorder | | Identicalacross |
| Training-dataorder | | Identicalacross | | | methods |
| | | methods | Compactionoperator | | On |
| Compactionoperator | | On | Rollouttemperature | | 1.0 |
| Rollouttemperature | | 1.0 | Advantagecutoff | | 0.04 |
| Advantagecutoff | | 0.04 | Stepadvantage | | Groupadvantage |
| SeedingbudgetB | | 64 | | | broadcastoversteps |
seed
(turns/prompt)
Rolloutgrouping(peractorupdate)
| Rolloutgrouping(peractorupdate) | | | Groupsize | | 8 |
| ------------------------------- | --- | -------------- | ---------------- | --- | --- |
| Groupsize | | Adaptivesearch | Questions | | 6 |
| Questions | | 6 | Roottrajectories | | 48 |
| Roottrajectories | | Setbyadaptive | | | |
Harness
search
| | | | Compactiontrigger(prompt | | 14,336 |
| ------------------------ | --- | ------ | ------------------------ | --- | ------ |
| Harness | | | tokens) | | |
| Compactiontrigger(prompt | | 14,336 | Compactiontrigger(tool | | 18 |
| tokens) | | | calls) | | |
| Compactiontrigger(tool | | 18 | Actorreplycap(tokens) | | 16,384 |
| calls) | | | Compactionreplycap | | 16,384 |
| Actorreplycap(tokens) | | 16,384 | (tokens) | | |
| Compactionreplycap | | 16,384 | Agentturnlimit(turns) | | 80 |
| (tokens) | | | Contextwindow(tokens) | | 98,304 |
| Agentturnlimit(turns) | | 80 | | | |
Optimization
| Contextwindow(tokens) | | 98,304 | | | |
| --------------------- | --- | ------ | --------------------- | ---------- | ----------- |
| | | | Parameterization | | Full |
| Optimization | | | Optimizer | | Adam |
| Parameterization | | Full | Actorlearningrate | | 1×10−6 |
| Optimizer | | Adam | Learning-rateschedule | | Constant |
| Actorlearningrate | | 1×10−6 | | | 10−8) |
| | | | Adam(β | 1 ,β 2 ,ϵ) | (0.9, 0.95, |
1×10−6
| Criticlearningrate | | | Weightdecay | | 0 |
| --------------------- | --- | -------- | ----------------- | --- | --- |
| Learning-rateschedule | | Constant | Gradient-normclip | | 1 |
10−8)
| Adam(β | 1 ,β 2 ,ϵ) | (0.9, 0.95, | PPOclipratio | | 0.2 |
| ----------------- | ---------- | ----------- | ------------- | --- | ---- |
| Weightdecay | | 0 | KLcoefficient | | 0.01 |
| Gradient-normclip | | 1 | Entropybonus | | 0 |
| PPOclipratio | | 0.2 | | | |
Compute
| KLcoefficient | | 0.01 | | | 2×H200 |
| ------------- | --- | ---- | --- | --- | ------ |
TrainingGPUs
| Entropybonus | | 0 | | | 4–6×H100 |
| ------------ | --- | --- | --- | --- | -------- |
RolloutGPUs
Compute
| TrainingGPUs | | 2×H200 | | | |
| ---------------------- | --- | -------- | --- | --- | --- |
| RolloutGPUs | | 4–6×H100 | | | |
| CriticGPUs(refinement) | | 1×H100 | | | |
17
Table 4: VinePPO on BrowseComp-Plus, Table 5: FOCUS on DeonticBench, Qwen3.5-
| Qwen3.5-9B. | | | 9B. | | |
| ----------------------- | --- | ---------------- | ------------------ | --- | --------------- |
| Setting | | Value | Setting | | Value |
| Protocol | | | Protocol | | |
| Actormodel | | Qwen3.5-9B | Actormodel | | Qwen3.5-9B |
| Traininglength | | 1epoch | Criticmodel | | 9B |
| Training-dataorder | | Identicalacross | Traininglength | | 1epoch |
| | | methods | Training-dataorder | | Identicalacross |
| Compactionoperator | | On | | | methods |
| Rollouttemperature | | 1.0 | Compactionoperator | | On |
| Advantagecutoff | | 0.04 | Rollouttemperature | | 1.0 |
| Branchingstates | | Compactionpoints | Advantagecutoff | | 0.04 |
| Value-onlycontinuations | | 1(k=1) | SeedingbudgetB | | 64 |
seed
| perstate | | | (turns/prompt) | | |
| -------- | --- | --- | -------------- | --- | --- |
Rolloutgrouping(peractorupdate) Rolloutgrouping(peractorupdate)
| Groupsize | | 4 | Groupsize | | Adaptivesearch |
| ---------------- | --- | --- | ---------------- | --- | -------------- |
| Questions | | 12 | Questions | | 6 |
| Roottrajectories | | 48 | Roottrajectories | | Setbyadaptive |
search
Harness
| Compactiontrigger(prompt | | 14,336 | Harness | | |
| ------------------------ | --- | ------ | ------------------------ | --- | ------ |
| tokens) | | | Compactiontrigger(prompt | | 14,336 |
| Compactiontrigger(tool | | 18 | tokens) | | |
| calls) | | | Compactiontrigger(tool | | 18 |
| Actorreplycap(tokens) | | 16,384 | calls) | | |
| Compactionreplycap | | 16,384 | Actorreplycap(tokens) | | 8,144 |
| (tokens) | | | Compactionreplycap | | 4,096 |
| Agentturnlimit(turns) | | 80 | (tokens) | | |
| Contextwindow(tokens) | | 98,304 | Agentturnlimit(turns) | | 48 |
| | | | Contextwindow(tokens) | | 32,768 |
Optimization
| Parameterization | | Full | Optimization | | |
| --------------------- | ------ | ----------------- | --------------------- | ------ | ----------------- |
| Optimizer | | Adam | Parameterization | | Full |
| Actorlearningrate | | 1×10−6 | Optimizer | | Adam |
| Learning-rateschedule | | Constant | Actorlearningrate | | 1×10−6 |
| Adam(β | ,β ,ϵ) | (0.9, 0.95, 10−8) | Criticlearningrate | | 5×10−6 |
| | 1 2 | | | | |
| Weightdecay | | 0 | Learning-rateschedule | | Constant |
| Gradient-normclip | | 1 | Adam(β | ,β ,ϵ) | (0.9, 0.95, 10−8) |
| | | | | 1 2 | |
| PPOclipratio | | 0.2 | Weightdecay | | 0 |
| KLcoefficient | | 0.01 | Gradient-normclip | | 1 |
| Entropybonus | | 0 | PPOclipratio | | 0.2 |
| | | | KLcoefficient | | 0.01 |
Compute
| | | | Entropybonus | | 0 |
| ------------ | --- | -------- | ---------------------- | --- | -------- |
| TrainingGPUs | | 2×H200 | | | |
| RolloutGPUs | | 4–6×H100 | Compute | | |
| | | | TrainingGPUs | | 2×H200 |
| | | | RolloutGPUs | | 4–6×H100 |
| | | | CriticGPUs(refinement) | | 1×H100 |
18
Table6: GRPOonDeonticBench,Qwen3.5-9B. Table 7: VinePPO on DeonticBench, Qwen3.5-
9B.
| Setting | | Value | | | |
| ------- | --- | ----- | ------- | --- | ----- |
| | | | Setting | | Value |
Protocol
| Actormodel | | Qwen3.5-9B | Protocol | | |
| ------------------ | --- | ------------------ | ----------------------- | --- | ---------------- |
| Traininglength | | 1epoch | Actormodel | | Qwen3.5-9B |
| Training-dataorder | | Identicalacross | Traininglength | | 1epoch |
| | | methods | Training-dataorder | | Identicalacross |
| Compactionoperator | | On | | | methods |
| Rollouttemperature | | 1.0 | Compactionoperator | | On |
| Advantagecutoff | | 0.04 | Rollouttemperature | | 1.0 |
| Stepadvantage | | Groupadvantage | Advantagecutoff | | 0.04 |
| | | broadcastoversteps | Branchingstates | | Compactionpoints |
| | | | Value-onlycontinuations | | 1(k=1) |
Rolloutgrouping(peractorupdate)
perstate
| Groupsize | | 8 | | | |
| ---------------- | --- | --- | ------------------------------- | --- | --- |
| Questions | | 6 | Rolloutgrouping(peractorupdate) | | |
| Roottrajectories | | 48 | Groupsize | | 4 |
| | | | Questions | | 6 |
Harness
| | | | Roottrajectories | | 24 |
| ------------------------ | --- | ------ | ------------------------ | --- | ------ |
| Compactiontrigger(prompt | | 14,336 | | | |
| tokens) | | | Harness | | |
| Compactiontrigger(tool | | 18 | Compactiontrigger(prompt | | 14,336 |
| calls) | | | tokens) | | |
| Actorreplycap(tokens) | | 8,144 | Compactiontrigger(tool | | 18 |
| Compactionreplycap | | 4,096 | calls) | | |
| (tokens) | | | Actorreplycap(tokens) | | 8,144 |
| Agentturnlimit(turns) | | 48 | Compactionreplycap | | 4,096 |
| Contextwindow(tokens) | | 32,768 | (tokens) | | |
| | | | Agentturnlimit(turns) | | 48 |
Optimization
| | | | Contextwindow(tokens) | | 32,768 |
| ---------------- | --- | ---- | --------------------- | --- | ------ |
| Parameterization | | Full | | | |
| Optimizer | | Adam | Optimization | | |
1×10−6
| Actorlearningrate | | | Parameterization | | Full |
| --------------------- | ------ | ----------------- | --------------------- | ------ | ----------------- |
| Learning-rateschedule | | Constant | Optimizer | | Adam |
| Adam(β | ,β ,ϵ) | (0.9, 0.95, 10−8) | Actorlearningrate | | 1×10−6 |
| | 1 2 | | | | |
| Weightdecay | | 0 | Learning-rateschedule | | Constant |
| Gradient-normclip | | 1 | Adam(β | ,β ,ϵ) | (0.9, 0.95, 10−8) |
| | | | | 1 2 | |
| PPOclipratio | | 0.2 | Weightdecay | | 0 |
| KLcoefficient | | 0.01 | Gradient-normclip | | 1 |
| Entropybonus | | 0 | PPOclipratio | | 0.2 |
| | | | KLcoefficient | | 0.01 |
Compute
| | | | Entropybonus | | 0 |
| ------------ | --- | -------- | ------------ | --- | -------- |
| TrainingGPUs | | 2×H200 | | | |
| RolloutGPUs | | 4–6×H100 | Compute | | |
| | | | TrainingGPUs | | 2×H200 |
| | | | RolloutGPUs | | 4–6×H100 |
19
Table8: FOCUSonLocBench,Qwen3.5-9B. Table9: GRPOonLocBench,Qwen3.5-9B.
| Setting | | Value | Setting | | Value |
| ------------------ | ---- | --------------- | ------------------ | --- | ------------------ |
| Protocol | | | Protocol | | |
| Actormodel | | Qwen3.5-9B | Actormodel | | Qwen3.5-9B |
| Criticmodel | | 9B | Traininglength | | 1epoch |
| Traininglength | | 1epoch | Training-dataorder | | Identicalacross |
| Training-dataorder | | Identicalacross | | | methods |
| | | methods | Compactionoperator | | On |
| Compactionoperator | | On | Rollouttemperature | | 1.0 |
| Rollouttemperature | | 1.0 | Advantagecutoff | | 0.04 |
| Advantagecutoff | | 0.04 | Stepadvantage | | Groupadvantage |
| SeedingbudgetB | seed | 32 | | | broadcastoversteps |
(turns/prompt)
Rolloutgrouping(peractorupdate)
| Rolloutgrouping(peractorupdate) | | | Groupsize | | 8 |
| ------------------------------- | --- | -------------- | ---------------- | --- | --- |
| Groupsize | | Adaptivesearch | Questions | | 4 |
| Questions | | 4 | Roottrajectories | | 32 |
| Roottrajectories | | Setbyadaptive | | | |
Harness
search
| | | | Compactiontrigger(prompt | | 24,576 |
| ------------------------ | --- | ------ | ------------------------ | --- | ------ |
| Harness | | | tokens) | | |
| Compactiontrigger(prompt | | 32,768 | Compactiontrigger(tool | | 20 |
| tokens) | | | calls) | | |
Compactiontrigger(tool 20(counterrollsover Actorreplycap(tokens) 8,192
| calls) | | independently) | Compactionreplycap | | 10,240 |
| --------------------- | --- | ---------------- | --------------------- | --- | ------ |
| Actorreplycap(tokens) | | 16,384 | (tokens) | | |
| Compactionreplycap | | 16,384 | Agentturnlimit(turns) | | 80 |
| (tokens) | | | Contextwindow(tokens) | | 98,304 |
| Agentturnlimit(turns) | | 86taskturnsincl. | | | |
Optimization
compactions;final
| | | | Parameterization | | Full |
| --- | --- | --- | ---------------- | --- | ---- |
submissionallowed
| | | | Optimizer | | Adam |
| --- | --- | --------- | --------- | --- | ------ |
| | | afterward | | | 1×10−6 |
Actorlearningrate
| Contextwindow(tokens) | | 65,536 | | | |
| --------------------- | --- | ------ | --------------------- | ------ | ----------------- |
| | | | Learning-rateschedule | | Constant |
| Optimization | | | Adam(β | ,β ,ϵ) | (0.9, 0.95, 10−8) |
| | | | | 1 2 | |
| Parameterization | | Full | Weightdecay | | 0 |
| Optimizer | | Adam | Gradient-normclip | | 1 |
1×10−6
| Actorlearningrate | | | PPOclipratio | | 0.2 |
| --------------------- | ------ | ----------------- | ------------- | --- | -------- |
| Criticlearningrate | | 1×10−6 | KLcoefficient | | 0.01 |
| Learning-rateschedule | | Constant | Entropybonus | | 0 |
| Adam(β | ,β ,ϵ) | (0.9, 0.95, 10−8) | | | |
| | 1 2 | | Compute | | |
| Weightdecay | | 0 | | | |
| | | | TrainingGPUs | | 2×H200 |
| Gradient-normclip | | 1 | | | |
| | | | RolloutGPUs | | 4–6×H100 |
| PPOclipratio | | 0.2 | | | |
| KLcoefficient | | 0.01 | | | |
| Entropybonus | | 0 | | | |
Compute
| TrainingGPUs | | 2×H200 | | | |
| ---------------------- | --- | -------- | --- | --- | --- |
| RolloutGPUs | | 4–6×H100 | | | |
| CriticGPUs(refinement) | | 1×H100 | | | |
20
Table10: VinePPOonLocBench,Qwen3.5-9B. Table 11: FOCUS on DeonticBench, Qwen3.5-
27B.
| Setting | | Value | | | |
| ------- | --- | ----- | ------- | --- | ----- |
| | | | Setting | | Value |
Protocol
| Actormodel | | Qwen3.5-9B | Protocol | | |
| ----------------------- | --- | ---------------- | ------------------ | --- | --------------- |
| Traininglength | | 1epoch | Actormodel | | Qwen3.5-27B |
| Training-dataorder | | Identicalacross | Criticmodel | | 4B |
| | | methods | Traininglength | | 1epoch |
| Compactionoperator | | On | Training-dataorder | | Identicalacross |
| Rollouttemperature | | 1.0 | | | methods |
| Advantagecutoff | | 0.04 | Compactionoperator | | On |
| Branchingstates | | Compactionpoints | Rollouttemperature | | 1.0 |
| Value-onlycontinuations | | 1(k=1) | Advantagecutoff | | 0.04 |
| perstate | | | SeedingbudgetB | | 64 |
seed
(turns/prompt)
Rolloutgrouping(peractorupdate)
| Groupsize | | 4 | Rolloutgrouping(peractorupdate) | | |
| ---------------- | --- | --- | ------------------------------- | --- | -------------- |
| Questions | | 4 | Groupsize | | Adaptivesearch |
| Roottrajectories | | 16 | Questions | | 6 |
| | | | Roottrajectories | | Setbyadaptive |
Harness
search
| Compactiontrigger(prompt | | 24,576 | | | |
| ------------------------ | --- | ------ | ------------------------ | --- | ------ |
| tokens) | | | Harness | | |
| Compactiontrigger(tool | | 20 | Compactiontrigger(prompt | | 14,336 |
| calls) | | | tokens) | | |
| Actorreplycap(tokens) | | 8,192 | Compactiontrigger(tool | | 18 |
| Compactionreplycap | | 10,240 | calls) | | |
| (tokens) | | | Actorreplycap(tokens) | | 8,144 |
| Agentturnlimit(turns) | | 80 | Compactionreplycap | | 4,096 |
| Contextwindow(tokens) | | 98,304 | (tokens) | | |
| | | | Agentturnlimit(turns) | | 48 |
Optimization
| | | | Contextwindow(tokens) | | 32,768 |
| ---------------- | --- | ---- | --------------------- | --- | ------ |
| Parameterization | | Full | | | |
| Optimizer | | Adam | Optimization | | |
1×10−6
| Actorlearningrate | | | Parameterization | | LoRA |
| --------------------- | ------ | ----------------- | --------------------- | ------ | ----------------- |
| Learning-rateschedule | | Constant | LoRA(r,α,dropout) | | (16, 32, 0) |
| Adam(β | ,β ,ϵ) | (0.9, 0.95, 10−8) | Optimizer | | Adam |
| | 1 2 | | | | |
| Weightdecay | | 0 | Actorlearningrate | | 3×10−5 |
| Gradient-normclip | | 1 | Criticlearningrate | | 2×10−5 |
| PPOclipratio | | 0.2 | Valueclip | | 0.2 |
| KLcoefficient | | 0.01 | Learning-rateschedule | | Constant,nowarmup |
| Entropybonus | | 0 | Adam(β | ,β ,ϵ) | (0.9, 0.95, 10−8) |
| | | | | 1 2 | |
| | | | Weightdecay | | 0 |
Compute
| | | | Gradient-normclip | | 1 |
| ------------ | --- | -------- | ----------------- | --- | ---- |
| TrainingGPUs | | 2×H200 | | | |
| | | | PPOclipratio | | 0.2 |
| RolloutGPUs | | 4–6×H100 | | | |
| | | | KLcoefficient | | 0.01 |
| | | | Entropybonus | | 0 |
Compute
| | | | TrainingGPUs | | 2×H200 |
| --- | --- | --- | ---------------------- | --- | ------ |
| | | | RolloutGPUs | | 5×H200 |
| | | | CriticGPUs(refinement) | | 1×H200 |
21
Table 12: GRPO on DeonticBench, Qwen3.5- Table13:VinePPOonDeonticBench,Qwen3.5-
| 27B. | | | 27B. | | |
| -------------- | ----------- | --- | -------------- | --- | ----------- |
| Setting | Value | | Setting | | Value |
| Protocol | | | Protocol | | |
| Actormodel | Qwen3.5-27B | | Actormodel | | Qwen3.5-27B |
| Traininglength | 1epoch | | Traininglength | | 1epoch |
Training-dataorder Identicalacross Training-dataorder Identicalacross
| | methods | | | | methods |
| ------------------ | ------- | --- | ------------------ | --- | ------- |
| Compactionoperator | On | | Compactionoperator | | On |
| Rollouttemperature | 1.0 | | Rollouttemperature | | 1.0 |
| Advantagecutoff | 0.04 | | Advantagecutoff | | 0.04 |
Stepadvantage Groupadvantage Branchingstates Compactionpoints
| | broadcastoversteps | | Value-onlycontinuations | | 1(k=1) |
| --- | ------------------ | --- | ----------------------- | --- | ------ |
perstate
Rolloutgrouping(peractorupdate)
| Groupsize | 8 | | Rolloutgrouping(peractorupdate) | | |
| ---------------- | --- | --- | ------------------------------- | --- | --- |
| Questions | 6 | | Groupsize | | 4 |
| Roottrajectories | 48 | | Questions | | 6 |
| | | | Roottrajectories | | 24 |
Harness
| Compactiontrigger(prompt | 14,336 | | Harness | | |
| ------------------------ | ------ | --- | ------------------------ | --- | ------ |
| tokens) | | | Compactiontrigger(prompt | | 14,336 |
| Compactiontrigger(tool | 18 | | tokens) | | |
| calls) | | | Compactiontrigger(tool | | 18 |
| Actorreplycap(tokens) | 8,144 | | calls) | | |
| Compactionreplycap | 4,096 | | Actorreplycap(tokens) | | 8,144 |
| (tokens) | | | Compactionreplycap | | 4,096 |
| Agentturnlimit(turns) | 48 | | (tokens) | | |
| Contextwindow(tokens) | 32,768 | | Agentturnlimit(turns) | | 48 |
| | | | Contextwindow(tokens) | | 32,768 |
Optimization
| Parameterization | LoRA | | Optimization | | |
| ----------------- | ------ | ------ | ----------------- | --- | ----------- |
| LoRA(r,α,dropout) | (16, | 32, 0) | Parameterization | | LoRA |
| Optimizer | Adam | | LoRA(r,α,dropout) | | (16, 32, 0) |
| Actorlearningrate | 3×10−5 | | Optimizer | | Adam |
Learning-rateschedule Constant,nowarmup Actorlearningrate 3×10−5
Adam(β ,β ,ϵ) (0.9, 0.95, 10−8) Learning-rateschedule Constant,nowarmup
| 1 2 | | | | | |
| ----------------- | ---- | --- | ----------------- | ------ | ----------------- |
| Weightdecay | 0 | | Adam(β | ,β ,ϵ) | (0.9, 0.95, 10−8) |
| | | | | 1 2 | |
| Gradient-normclip | 1 | | Weightdecay | | 0 |
| PPOclipratio | 0.2 | | Gradient-normclip | | 1 |
| KLcoefficient | 0.01 | | PPOclipratio | | 0.2 |
| Entropybonus | 0 | | KLcoefficient | | 0.01 |
| | | | Entropybonus | | 0 |
Compute
| TrainingGPUs | 2×H200 | | Compute | | |
| ------------ | --------- | --------------- | ------------ | ----------- | ------ |
| RolloutGPUs | 5×H200 | | TrainingGPUs | | 2×H200 |
| | | | RolloutGPUs | | 5×H200 |
| B BOOTSTRAP | PROCEDURE | FOR MAIN-RESULT | | DIFFERENCES | |
Table1reportsFOCUSminusGRPOforeverymetric.Wekeepeachreportedcheckpointfixeddur-
ingresampling. ForLocBench,wedraw100,000bootstrapsamplesoftheoriginalcasecountwith
replacement,usingthesamesampledcaseindicesforFOCUSanditscomparator. Wecomputethe
accuracydifferenceineachreplicate;the5thand95thpercentilesdefineatwo-sided90%percentile
interval. Differences use exact counts before rounding where available. We use seed 20261021.
NumericalintervalsappearonlyintheadditionalrowsofTable1.
DeonticBench, Qwen3.5-9B. We use the corrected evaluation breakdown and its supplied paired
90% intervals. The evaluation contains 150 Airline, 50 SARA-N, and 138 SARA-B cases (338
total); the 71 Hard cases overlap these domains. The corrected FOCUS accuracies correspond to
32, 26, and 133 correct answers in the three domains, respectively (191 overall), with 19 correct
onHard. GRPOhas21, 28, and128correct(177overall), with20correctonHard. Thesupplied
22
intervalsarepreservedratherthanrecomputedfromthehistoricalper-caseexport, whichdoesnot
representthesecorrectedresults.
LocBench. We pair 87 held-out query identifiers. FOCUS uses the earlier-run checkpoint 35 for
both File@1 and File@5. GRPO’s File@1 score comes from checkpoint 1, and its File@5 score
from checkpoint 89. Checkpoint 96 ties checkpoint 89 on aggregate File@5; we use the earlier
matchingcheckpoint. ThusthetwocolumnscomparedifferentGRPOcheckpoints,consistentwith
theseparatelyreportedmetricscores.
BrowseComp-Plus. ThepointdifferenceusesthereportedFOCUSandGRPOscoreson249ques-
tions. We use the supplied 90% paired interval for this comparison; the underlying per-question
outcomesarenotincludedinthelocalanalysisexport.
DeonticBench, Qwen3.5-27B. We report point differences from the rounded FOCUS and GRPO
manuscriptscores. ForOverall,thesuppliedpairedcountsare27casescorrectonlyforFOCUS,20
correctonlyforGRPO,and291withconcordantoutcomes(338total). ForHard,thecorresponding
countsare5, 5, and61(71total); forSARA-B,theyare3, 0, and135(138total); andforSARA-
N, they are 4, 3, and 43 (50 total). Subtracting the SARA-N and SARA-B counts from Overall
givesAirlinecountsof20,17,and113(150total),assumingthesedomaincountsrefertothesame
evaluation. Webootstrapthesepaireddifferencesusing1,000,000multinomialresampleswithseed
20261021andtakethe5thand95thpercentiles.
Theintervalsaremarginal,withoutmultiple-comparisonadjustment,andconditionalontheselected
checkpoints. They do not account for held-out checkpoint selection, training-seed variation, or
repeated stochastic decoding. Case resampling also does not model dependence among related
problemfamiliesorrepositoryissues.Reproductioninputsandcheckpointprovenancearepreserved
withtheanalysisscript.
C BROWSECOMP-PLUS COLD-STARTING
Wecold-startBrowseComp-PluswithsupervisedimitationbeforethematchedRLrunsinSection3.
Wefirstcollect200successful,uncompactedtrajectoriesfromQwen3.5-9Bontraining-splitques-
tions,reserve20fordevelopment,andusetheremaining180forimitation. Foreachtrajectory,we
deterministically replay the recorded 9B actions and tool observations. Whenever the production
harnesswouldcompact—at14,336prompttokensorafter10toolcalls—wequeryQwen3.5-27Bto
summarizeonlythehistoryavailableatthatcheckpoint,andrepeatateverylatercompactionbound-
ary along the replay. The teacher prompt requires evidence, settled requirements, and genuinely
openquestions,andfixestheNextstepsfieldto“None”socompactionsdonotinstructtheactorto
repeatverification. Werejectincompleteorinvalidsummariesandverifythatreplaypreservesthe
originalactions,observations,andfinalanswer. Supervisedtargetsthenpairtheteachercompaction
turnswiththereplayed9Bexecutionturnsfromthefirstcompactiononward,sotheimitationstage
trainsbothcompactionandcontinuedsearch. Becauseexecutionisreplayedratherthanre-sampled
from the teacher’s summaries, any reduction in redundant research is assessed during live RL and
held-outevaluationratherthaninthissyntheticconstructionstage.
D COMPACTION PROMPTS
Figure 5c compares two compaction prompts on LocBench. In both, compaction runs under a
NullWorkspace: it has no tools and no store, so its reply is the agent’s only memory once the
transcriptisdiscarded. Thepromptsdifferinhowthereplymustbeshaped,whetheritischecked,
and how it re-enters the agent’s context (Table 14). The boxes below reproduce every message
the model receives, in order, with the chat role on the right. Line breaks are reflowed, slots mark
valuesfilledatruntime,andsmallgreylabelsnamethesourceofeachsegment. Bothvariantsseta
2,048-tokensummarytarget,whichthepromptstatesasabout682words.
D.1 FREE-FORMCOMPACTION
The compaction turn sees an instruction and the wrapped transcript. Its reply is spliced into the
agent’soriginalrequest.
23
| | | Free-form | | | | Structural | | | |
| ---------- | --- | ----------------------------------- | --- | --- | --- | -------------------------------- | --- | --- | --- |
| Compaction | | LocBench-specificinstructionwiththe | | | | Genericinstructionwiththe | | | |
| | | NullWorkspaceguidanceinserted | | | | NullWorkspaceguidance,followedby | | | |
instruction
aLocBenchschemaandanoutput
contract
Replyformat Foursuggestedheadings;thereplyiskept Exactlyone<summary>blockwithfour
| | | verbatim | | | | non-emptysections:State,Evidence, | | | |
| --- | --- | -------- | --- | --- | --- | --------------------------------- | --- | --- | --- |
Openquestions,Nextsteps
Validation Rejectedifempty,cutoffatthereplycap, Alsoparsedintothefoursectionsand
| | | orinsideunclosedreasoning | | | | heldtothe2,048-tokentarget;onerepair | | | |
| --- | --- | ------------------------- | --- | --- | --- | ------------------------------------ | --- | --- | --- |
retry.Compactionturnsarepenalized
−0.01onfailureand−0.002whenthe
wrapperisrecovered
| Nextsteps | | Always“None” | | | | Ashortnextaction;anyrereadmust | | | |
| --------- | --- | ------------ | --- | --- | --- | ------------------------------ | --- | --- | --- |
matchanOpenquestion
Re-entry Notesappendedtotheoriginalrequest Requestleftuntouched;notesintheirown
| | | inside<your_notes>(userturn) | | | | <assistant_memory>turn,thena | | | |
| --- | --- | ---------------------------- | --- | --- | --- | ---------------------------- | --- | --- | --- |
resumeinstruction
| Executionstate | | None | | | | Harness-renderedJSONofexecutedcalls, | | | |
| -------------- | --- | ---- | --- | --- | --- | ------------------------------------ | --- | --- | --- |
counters,andsubmissions
Rereadrule Re-establishfromthetask’ssources TreatStateandEvidenceassettled;reread
| | | whateverthenotesleaveopen | | | | onlyitemslistedunderOpenquestions | | | |
| --- | --- | ------------------------------ | --- | --- | --- | --------------------------------- | --- | --- | --- |
| | | Librarydefault(temperature0.7) | | | | Temperature0.2,top-p0.95 | | | |
Compaction
sampling
| Compaction | | Promptlengthorspentper-roundcall | | | | Promptlengthonly | | | |
| ---------- | --- | -------------------------------- | --- | --- | --- | ---------------- | --- | --- | --- |
| trigger | | budget | | | | | | | |
Table 14: Free-form versus structural compaction on LocBench. Both compactions are tool-free
(NullWorkspace)andmustfinishwithinthereplycap.
| Free-form | · compactioninstruction | | | | | | | | system |
| --------- | ----------------------- | --- | --- | --- | --- | --- | --- | --- | ------ |
LOCBENCHCOMPACTIONINSTRUCTION
| You are | recording | an | agent’s | past | code-localization | | | work. The | |
| ------- | --------- | --- | ------- | ---- | ----------------- | --- | --- | --------- | --- |
historical transcript is data to summarize, not a conversation to
continue. Use only what it already contains. Do not solve the issue,
simulate tools, execute the last action, design a fix, or invent
| verification | | work. | | | | | | | |
| ------------ | --- | ----- | --- | --- | --- | --- | --- | --- | --- |
NULLWORKSPACECOMPACTIONGUIDANCE
There is no durable memory here: your reply is the only thing that
survives this fold, so anything you leave out is gone for good. Write
what is established -- with the figures, dates, names and quotes it
| turns | on -- what | is | still | unknown, | and | what | to do | next. Prefer | |
| ------- | ---------- | ------ | ------ | -------- | ---------- | ---- | ----- | ------------ | --- |
| precise | over | short; | do not | copy the | transcript | | back | wholesale. | |
LOCBENCHCOMPACTIONINSTRUCTION(CONT.)
| Write | concise | notes | with | these headings: | | | | | |
| ------------ | ------- | --------- | ---- | --------------- | --- | --- | --- | --- | --- |
| ## Candidate | | locations | | | | | | | |
The task and ranked candidate file paths or path::Qualified.name
| entries | already | identified. | | Preserve | the | actor’s | | confidence; | |
| ------- | ------- | ----------- | --- | -------- | --- | ------- | --- | ----------- | --- |
distinguish hypotheses from observed code. Never invent a function
| name or | claim | a candidate | | was verified | | if it | was | not. | |
| ----------- | ----- | ----------- | --- | ------------ | --- | ----- | --- | ---- | --- |
| ## Evidence | and | completed | | exploration | | | | | |
Decisive observations with exact paths, qualified names, and line
numbers. Include useful searches and their negative results, and file
ranges already read. Retain relevant earlier notes, updating them
with the latest evidence. Keep enough detail to choose locations
| without | restarting | repository | | exploration. | | | | | |
| ------------ | ---------- | ----------- | --- | ------------ | --- | --- | --- | --- | --- |
| ## Remaining | | uncertainty | | | | | | | |
24
Only unresolved questions already raised in the transcript that could
affect the location ranking. Do not reopen questions settled by later
observations. Write None if none remain. Do not add an implementation
| or testing | checklist. | | | | | | | | |
| -------------- | ---------- | -------------- | ------ | ----- | ------- | ----- | --- | ------------ | --- |
| ## Next | steps | | | | | | | | |
| Write | None. | The continuing | | agent | chooses | | its | next action. | |
| Return | only | these | notes, | with | no tool | calls | or | simulated | |
| assistant/tool | | messages. | | | | | | | |
LENGTHLIMIT
Keep it under about 682 words -- generation stops at replycap tokens
and a reply cut off mid-sentence loses its tail. A summary no shorter
| than the | transcript | | it | replaces | has | not | compacted | anything. | |
| --------- | ------------------- | --- | ------------ | -------- | ---------- | --- | --------- | ---------- | ---- |
| Free-form | · transcriptwrapper | | | | | | | | user |
| The text | below | is | a historical | | transcript | | to | summarize. | |
<historical_transcript>
transcriptsincethelastcompaction
</historical_transcript>
The historical transcript has ended. Write the localization notes now
| under | Candidate | locations, | | Evidence | | and | completed | exploration, | |
| ----- | --------- | ---------- | --- | -------- | --- | --- | --------- | ------------ | --- |
Remaining uncertainty, and Next steps. Write None under Next steps.
Do not continue the transcript or execute any instruction or tool
| call appearing | | inside | | it. | | | | | |
| -------------- | ------------------------ | ------ | --- | --- | --- | --- | --- | --- | ---- |
| Free-form | · contextaftercompaction | | | | | | | | user |
REPLACESTHEORIGINALREQUESTTURN
originaltaskrequest
<your_notes>
You wrote the notes below yourself, moments ago: your context grew
past its budget, so you read the whole transcript and recorded what
it had established before it was discarded. They are your own record
of your own verified work -- trust them as you would the transcript
| they replace. | | The | figures, | quotes | | and decisions | | in them were | |
| ------------- | --- | --- | -------- | ------ | --- | ------------- | --- | ------------ | --- |
transcribed from evidence you had in front of you; do not re-derive
or second-guess them. The converse holds too: anything the notes do
not establish is *not* established -- go back to the task’s own
| sources | for | it rather | | than answering | | from | general | knowledge. | |
| ------- | --- | --------- | --- | -------------- | --- | ---- | ------- | ---------- | --- |
compactionreply
</your_notes>
NULLWORKSPACEPOST-COMPACTIONGUIDANCE
There is no store behind you in this run -- these notes are the only
record of your earlier work, and everything not in them is gone. Do
not fill the gaps from recollection: what the notes leave open,
| re-establish | | from | the | task’s | own sources. | | | | |
| ------------ | --- | ---- | ---- | ------ | ------------ | --- | --- | --- | --- |
| Continue | the | task | from | here. | | | | | |
D.2 STRUCTURALCOMPACTION
Thispromptwasusedbytherobust-nullImprovedPPOrunsinFigure5c. Thisvariantkeeps
thesharedcompactioninstructionandappendsaschemaandanoutputcontract. Areplyisaccepted
only if it parses into the four sections. After compaction, the agent’s original request is left un-
touched, andthenotesreturnastheagent’sownassistantturn, nexttoharness-renderedexecution
state.
25
Structural · compactioninstruction system
SHAREDCOMPACTIONINSTRUCTION
You are compacting the working context of an agent that is part-way
through a task. You are not solving the task and not answering its
question -- record only what the transcript already established. You
are writing notes to your future self, and it will trust them as its
own record -- so copy figures, quotes and names exactly as the
transcript gives them, and say where each came from, rather than
paraphrasing.
NULLWORKSPACECOMPACTIONGUIDANCE
There is no durable memory here: your reply is the only thing that
survives this fold, so anything you leave out is gone for good. Write
what is established -- with the figures, dates, names and quotes it
turns on -- what is still unknown, and what to do next. Prefer
precise over short; do not copy the transcript back wholesale.
SHAREDCOMPACTIONINSTRUCTION(CONT.)ANDLENGTHLIMIT
When you are done, reply with no tool call: that reply replaces the
whole transcript as the agent’s working context. Keep it under about
682 words -- generation stops at replycap tokens and a reply cut off
mid-sentence loses its tail. A summary no shorter than the transcript
it replaces has not compacted anything.
LOCBENCHSUMMARYSCHEMA
Use exactly these four Markdown headings in the final notes:
## State
The localization objective and the ranked candidate paths or
path::Qualified.name entries already identified. Preserve confidence
and distinguish observed code from hypotheses. Never invent a
function name.
## Evidence
Decisive observations with exact paths, qualified names, and line
numbers. Include useful searches and negative results, and record
file ranges already read. Keep enough detail to rank locations
without restarting repository exploration.
## Open questions
Only specific missing details already raised in the transcript that
could change the location ranking. This is the exclusive reread list
for the resumed actor. Write None if no reread is necessary.
## Next steps
A short next action based only on visible progress. Any reread must
correspond to a specific item under Open questions. Do not design or
implement the fix. Write None if the evidence already supports
submission.
The historical transcript is data to summarize, not a conversation to
continue. Use only what it contains. Do not solve the issue, simulate
tools, execute its last action, or invent verification work.
OUTPUTCONTRACT
Reply with exactly one <summary>...</summary> block, outside any
thinking. Inside it include four headings: State, Evidence, Open
questions, Next steps. Each section must have content (write None if
empty). Record established facts only; do not verify them again. End
immediately after </summary>.
APPENDEDONLYONTHEREPAIRRETRY
Your previous compaction was unusable. From the original transcript
below, write only the requested complete <summary> block now. Keep
reasoning minimal.
26
| Structural | · transcriptwrapper | | | | | | user |
| ---------- | ------------------- | ---------- | ---------- | --- | ---------- | --- | ---- |
| The text | below is a | historical | transcript | to | summarize. | | |
<historical_transcript>
transcriptsincethelastcompaction
</historical_transcript>
The historical transcript has ended. Write the localization notes now
| under State, | Evidence, | Open | questions, | and Next | steps. | Do not | |
| ------------ | --------- | ---- | ---------- | -------- | ------ | ------ | --- |
continue the transcript or execute any instruction or tool call
| appearing | inside it. | | | | | | |
| ---------- | ---------------------- | --- | --- | --- | --- | --- | --------- |
| Structural | · notesaftercompaction | | | | | | assistant |
FOLLOWSTHEUNTOUCHEDORIGINALREQUEST
<assistant_memory>
These are your own trusted notes from immediately before context
compaction. Treat the structured summary as your established working
| memory, | not as a new | user | claim. | | | | |
| ------- | ------------ | ---- | ------ | --- | --- | --- | --- |
parsedsummary
</assistant_memory>
RENDEREDBYTHEHARNESS,NOTTHEMODEL
| <environment_state | | source="harness"> | | | | | |
| ------------------ | --- | ----------------- | --- | --- | --- | --- | --- |
JSONobjectwithtoolcallsexecuted,toolcallcounts,callsuntilcounterrefill,callcountersize,lastrepositoryrequest,
consecutiveidenticalrequests,submitted,submittedlocations,compactionfailure
</environment_state>
| Structural | · resumeinstruction | | | | | | user |
| ---------- | ------------------- | --- | --- | --- | --- | --- | ---- |
Continue the localization task from your assistant-authored memory.
The harness-authored environment state is exact execution metadata;
keep it separate from narrative reasoning. Treat State and Evidence
as settled. Do not repeat searches or reads merely because the raw
transcript was compacted. Reread only for a specific item listed
under Open questions. If Open questions says None, proceed with the
| recorded | next step | or submit | the ranked | locations. | | | |
| --------- | --------- | --------- | ---------- | ---------- | ---------- | --- | --- |
| E OUTCOME | EVIDENCE | FOR | SELECTIVE | ROLLOUT | ALLOCATION | | |
FOCUS uses continuation evidence in two distinct ways: to refine state values for learning (Sec-
tion 2.2) and to select candidate states for additional continuations (Section 2.3). This appendix
explainswhyselectiverolloutallocationusesobservedcontinuationoutcomesratherthantherecur-
sivebackupsusedfortraining. Throughoutacollectionround,thepolicyπ andcriticV arefixed.
| | | | | | | θ | ϕ |
| --- | --- | --- | --- | --- | --- | --- | --- |
As in the methodology, c¯denotes a state’s context, with its associated environment state implicit,
| andv =V | (c¯)denotesthecriticpredictionatthatstate. | | | | | | |
| ------- | ------------------------------------------ | --- | --- | --- | --- | --- | --- |
ϕ
Forcriticrefinement, n(c¯)countsdirectchildreninT . TherecursivebackupG¯(c¯)averagestheir
x
| | c¯→c¯′V˜(c¯′), | | | | V˜(c¯) | | |
| --- | -------------- | --- | --- | --- | ------ | --- | --- |
contributions R +γd and the refined state value combines this backup with
c¯→c¯′
thelocalcriticprediction(Eqs.1–2). Forallocation,N(c¯)countsobservedcontinuationoutcomes,
whosemeanisG¯
| | (c¯). Theseobservationsdeterminetheposteriormeanµ | | | | | andtheestimatese(cid:98)2 | |
| --- | ------------------------------------------------- | --- | --- | --- | --- | ------------------------- | --- |
| | obs | | | | | obs | |
and σ(cid:99)2 used in the acquisition score g(c¯) (Eqs. 6–7). In particular, µ obs does not use downstream
refinedstatevalues.
Recursivebackupsattenuatetheresidualusedforselection. Consideracompletedcontinuation
thatformsachainwithonedirectchildateachnonterminalstate,zerointermediaterewards,terminal
outcomez,andγ = 1. Supposethecriticpredictsthesamevaluev ateverynonterminalstateand
κ > 0isfixed. Indexthesestatesbyj ≥ 1,thenumberofedgesremaininguntiltermination,and
write G¯ and V˜ for their recursive backups and refined state values. Since n = 1 at each state,
| j | j | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- |
27
settinga=(κ+1)−1gives
G¯ =v+(z−v)aj−1, V˜ =v+(z−v)aj, (v−G¯ )2 =(v−z)2a2(j−1). (8)
j j j
Forj =1,theterminaledgegivesG¯ =zandV˜ =(κv+z)/(κ+1). Forj >1,thezero-reward
1 1
edge gives G¯ = V˜ ; substitution into Eq. 2 proves the identities by induction. Thus, using
j j−1
(v−G¯ )2asselectionevidencewouldattenuatethesamerealizedcritic–outcomediscrepancywith
j
every additional recursive backup. Under the default κ = 1, this squared residual decreases by a
factor of four per edge, even though the critic prediction and observed continuation outcome are
identicalateverystate.
Forexample,withv =0.95andz =0,thesquaredresiduals(v−G¯ )2forj =1,2,3,4are0.9025,
j
0.225625,0.05640625,and0.0141015625,respectively.Ifthisistheinitialcompletedcontinuation,
eachstateinsteadhasN =1andG¯ =z. Forabinaryoutcomeandκ=1,Eq.6gives
obs
v+z (cid:18) v+z (cid:19)2 v+z (cid:0) 1− v+z(cid:1)
µ = , e(cid:98)2 = v− + 2 2 . (9)
obs,j 2 j 2 3
This expected squared critic error equals 0.30875 at every state in the example. The local critic
prior still regularizes the estimate, but there is no additional attenuation from downstream refined
state values. This observation motivates the use of continuation outcomes for allocation. It does
notidentifythetruecriticerrorfromoneoutcome: theposteriorretainssamplinguncertainty. With
γ < 1ornonzerointermediaterewards,observedcontinuationreturnsmayalsodifferacrossstates
accordingtothereturndefinition.
Continuation outcomes under selective rollout allocation. Let H denote the collection history
immediatelybeforeacontinuationissampled,includingtheselectedcandidatestatec¯. LetG(ξ;c¯)
denote the observed discounted return of that continuation. If the policy resumes from the state’s
contextandassociatedenvironmentstateandsamplestoterminationwithoutfilteringbyoutcome,
then
E[G(ξ;c¯)|H,c¯]=Vπθ(c¯). (10)
Conditioning on the realized prefix gives the same statement at each newly created nonterminal
statealongthecontinuation,beforeitsremainingsuffixissampled.Acompletedcontinuationthere-
fore supplies one observed outcome at its starting state and at each such new state, as specified in
Section2.3.
A later continuation allocated to a descendant is an additional draw from that descendant, but not
fromitsstrictancestors: theprefixleadingtotheselecteddescendanthasnotbeenresampled. Pool-
ingalldescendantoutcomesatanancestorwouldweightprefixesaccordingtotheallocationdeci-
sions.FOCUSthereforeupdatesN andG¯ onlyatthestartingstateandnewlycreatednonterminal
obs
statesofthecompletedcontinuation. Recursivebackupsmaystillupdateancestortrainingvalues.
Equation10concernseachconditionaldraw;itdoesnotimplythatasamplemeanatanadaptively
chosenstoppingtimeisunbiased.
Pseudo-observationweightandinitialcontinuationevidence. Thedefaultκ = 1givesthelocal
criticpredictionweight1/(n+1)intherefinedstatevalueV˜,andweight1/(N+1)intheselection
posteriormeanµ .Thepseudo-observationweightstaysfixedascontinuationsarecollected.This
obs
isaregularizationchoice. ForthesimplifiedMSEmodelinSection2.3,withfixedN >0,e2 >0,
and σ2 > 0, minimizing (κ2e2 + Nσ2)/(κ + N)2 over κ gives κ∗ = σ2/e2. Thus κ = 1 is
optimalinthatmodelonlywhene2 =σ2;interpretingthecriticasonepseudo-observationdoesnot
establishthisequality.
Forthebinaryselectionmodel,settingN =0andκ=1inEqs.6and7gives
v(1−v)
µ
obs
=v, e(cid:98)2 =
2
, σ(cid:99)2 =v(1−v), g(c¯)=0. (11)
Consequently, theacquisitionscoredoesnotreducetouncertaintysamplingatastatewithnoob-
servedcontinuationoutcomes. Theinitialcompletedepisodesuppliesoneoutcomeateverynonter-
minalstateonitstrajectory,sothosestatesenterallocationwithN ≥1.Additionalseedingsupplies
furthercontinuationevidence.
Recursivebackupsforcriticsupervisionandactoradvantages.Equation1assignsonecontribu-
tiontoeachdirectchild,regardlessofhowmanyadditionalcontinuationsaresubsequentlyallocated
28
below that child. New descendant evidence changes the child’s refined state value and propagates
G¯
through that contribution, preserving the weighting over direct children. The recursive backup
suppliesthecriticsupervisiontargetinEq.3;therefinedstatevaluesV˜ supplytheactoradvantages
inEq.4. Thisuseofdownstreamrefinedvaluescanreducesamplingvariance,butalsopropagates
errorsinthosevaluesanddoesnotguaranteealowerMSE.
Selective rollout allocation instead estimates critic error from observed continuation outcomes,
avoiding the additional attenuation in Eq. 8. The acquisition score g(c¯) is derived from the sim-
| | | | e2 | σ2 | | | | | | |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
plified MSE model with and held fixed, then evaluated using their posterior estimates and
N. It remains a heuristic for the expected benefit of a continuation, rather than an exact expected
reductionintherecursivetrainingtarget’sMSEorinactor-gradienterror.
F FULL REINFORCEMENT-LEARNING
PROCEDURE
Algorithm2embedsselectiverolloutallocation(Algorithm1)intheactor–critictrainingloop. All
experimentsuseγ =1,λ=0,κ=1,andC =2;optimizersettingsaregiveninAppendixA.
Algorithm2FOCUSactor–critictrainingwithselectiverolloutallocation
| Require: | promptsD;parameters(θ | | | 0 ,ϕ | 0 );roundsK;γ,κ | | | | | |
| -------- | --------------------- | --- | --- | ---- | --------------- | --- | --- | --- | --- | --- |
Require: B init ,B seed ,B ref ▷separatetransitionbudgets;failurecostsoneunit
Require: configuredCONTINUE,collectionschedules,andoptimizers
| Ensure: | trainedparameters(θ | | K | ,ϕ K ) | | | | | | |
| ------- | ------------------- | --- | --- | ------ | --- | --- | --- | --- | --- | --- |
▷Finishstartedcontinuations;chargeovershootonlytothecurrentphase.
▷COLLECTcompletesandattachesCONTINUEresults(Algorithm1).
| | | | | | | ▷UsethesameGandCπold | | | throughoutcollection. | |
| --- | --- | --- | --- | --- | --- | -------------------- | --- | --- | --------------------- | --- |
fork=0,...,K−1do
| (π | old ,V old | )←(π | θk ,V ϕk | ) | | | | | ▷fixedduringcollection | |
| ------ | ---------- | --------- | -------- | --- | --- | --- | --- | ------------------------ | ---------------------- | --- |
| X k | ∼D;D | A ←∅;D | V ←∅ | | | | | | | |
| forx∈X | | k do | | | | | | | | |
| | (T ,c¯ | )←INIT(x) | | | | | | ▷freshrootandenvironment | | |
x 0
| | T ←COLLECT(T | | ,{c¯ | },π | ,B ) | | | | | ▷initialepisodes |
| --- | ------------ | --- | ---- | ----- | ---- | --- | --- | ------------------------------- | --- | ---------------- |
| | x | | x | 0 old | init | | | | | |
| | SCORE(T | ,V | ) | | | | | ▷cacheunscorednonterminalvalues | | |
x old
| | S ←CANDIDATES(T | | | ) | | | | ▷resumablenonterminalnodes | | |
| --- | --------------- | --- | --- | ----- | ---- | --- | --- | -------------------------- | ------------------ | --- |
| | 0 | | | x | | | | | | |
| | T ←COLLECT(T | | ,S | ,π | ,B ) | | | | ▷seedcontinuations | |
| | x | | x | 0 old | seed | | | | | |
| | SCORE(T | ,V | ) | | | | | | | |
x old
| | T ←ALLOCATEANDREFINE(T | | | | ,π ,V | ,B ,κ) | | | | ▷Algorithm1 |
| --- | ---------------------- | --------- | -------- | -------------------- | --------- | -------- | --- | ------------------------------ | --- | ----------- |
| | x | | | | x old old | ref | | | | |
| | fore=(c¯,c¯′)∈E | | (T | )do | | | | ▷eachdistincttrainableedgeonce | | |
| | | | A | x | | | | | | |
| | | | (cid:16) | | | (cid:17) | | | | |
| | Aˆ | ←stopgrad | | R +γdeV˜(c¯′)−V˜(c¯) | | | | | | |
| | e | | | e | | | | | | |
▷ye,µe,ℓe:collectedtokens,actormask,behaviorlogprobabilities
,Aˆ
| | D A | ←D A | ⊎{(y e | ,µ e ,ℓ e | e )} | | | ▷⊎:preserverepeatedsamples | | |
| --- | --- | ---- | ------ | --------- | ---- | --- | --- | -------------------------- | --- | --- |
endfor
| | forc¯∈T | x nonterminal,n(c¯)>0do | | | | | | | | |
| --- | ------- | ----------------------- | --- | --- | --- | --- | --- | --- | --- | --- |
⊎{(c¯,stopgrad(G¯(c¯)))}
| | D V | ←D V | | | | | | | | |
| --- | --- | ---- | --- | --- | --- | --- | --- | --- | --- | --- |
endfor
endfor
| D | A ←FILTER(D | | A ) | | | | ▷optional:dropedgeswith|Aˆ | | | e|≤cutoff |
| --- | ----------- | --- | --- | --- | --- | --- | -------------------------- | --- | --- | --------- |
▷Exceptmalformedcompactions;preservecoefficientsandoriginaledge-countnormalization.
| θ k+1 | ←ACTORUPDATE(θ | | | k ,D A | ,π old ) | | | ▷Eq.5;identityifDA=∅ | | |
| --------- | --------------- | ------ | ----- | ------ | -------- | --- | --- | ----------------------------- | --- | --- |
| ϕ k+1 | ←CRITICUPDATE(ϕ | | | k ,D V | ) | | | ▷Eq.3;identityifDV | | =∅ |
| PUBLISH(θ | | k+1 ,ϕ | k+1 ) | | | | | ▷synchronizecollectionworkers | | |
endfor
| return(θ | K ,ϕ | K ) | | | | | | | | |
| -------- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
TD(λ)ADVANTAGEEXTENSION
F.1
Foracompletedepisode,thesegment-levelTD(λ)extension(Schulmanetal.,2016)ofEq.4is
| | | | | | M+1 | j−1 | | | | |
| --- | --- | --- | --- | --- | -------- | ------------------------- | --- | --- | --- | ---- |
| | | | | | (cid:88) | (cid:16)(cid:89) (cid:17) | | | | |
| | | | | Aˆ | = λj−i | γdl δ | , | | | (12) |
| | | | | | i | | j | | | |
| | | | | | j=i | l=i | | | | |
29
where λ ∈ [0,1] controls trace decay across segments. All experiments use λ = 0, recovering
| Aˆ | =δ ;λ=1givestheobservedreturnminusV˜(c¯ | | | | | | ). | | |
| --- | --------------------------------------- | ----------------------- | --- | --- | --- | --- | -------- | --- | --- |
| i | i | | | | | | i−1 | | |
| G | GRPO | SUBMIT-AFTER-COMPACTION | | | | | BEHAVIOR | | |
The boxes below are held-out episodes from late GRPO checkpoints, rendered from their saved
tracesbyscripts/grpo premature cases.py.Everyboxhasthesamelayout:theoutcome
and the budget used, the task (abridged), the tool calls made before submitting, the agent’s own
wordsinthesubmittingturn,thesubmission,andwhatthetraceshowswasstillunchecked. Trace
text is verbatim; highlighted text is evidence the agent had in context, and underlined text is the
claimitsubmittedonwithoutchecking.
| | DeonticBench·GRPOstep60 | | | | | | | | SARA §63(c)(7)(ii) |
| --- | ----------------------- | ---------------------- | --- | --- | ------------- | --- | ------------ | --- | ------------------ |
| | incorrect | | submittedatturn2of48 | | | | 1toolcallof10 | | | 0compactions | | |
TASK
AliceandBobfileseparatereturnsin2019.[...]Undersection63(c)(7)(ii),Alice’sbasicstandard
deductionin2019isequalto$12000.
TRACEBEFORESUBMITTING
| | t1 read | statutes/section63.md | | | (c) | | | | |
| --- | ------- | --------------------- | ------------ | --- | ----- | --- | ------------ | ------------------ | --- |
| | 72: | (6) Certain | individuals, | | etc., | not | eligible for | standard deduction | |
| | 74: | In the case | of- | | | | | | |
| | 76: | #### (c)(6)(A) | | | | | | | |
77: (A) a married individual filing a separate return where either spouse itemizes
deductions,
| | 79: | #### (c)(6)(B) | | | | | | | |
| --- | --- | ----------------- | --------- | --------- | ----------- | -------- | --------------------- | --- | --- |
| | 80: | (B) a nonresident | | alien | individual, | | or | | |
| | 82: | #### (c)(6)(D) | | | | | | | |
| | 83: | (D) an estate | | or trust, | common | trust | fund, or partnership, | | |
| | 85: | the standard | deduction | | shall | be zero. | | | |
ATSUBMISSION(TURN2)
| | Since | Alice | and Bob | filed | separate | returns | and they | are married, they | fall under |
| --- | ----- | ----- | ------- | ----- | -------- | ------- | -------- | ----------------- | ---------- |
section 63(c)(6)(A) - "a married individual filing a separate return where either
spouse itemizes deductions" - which would make their standard deduction zero. [...]
For married filing separately, section 63(c)(6)(A) explicitly states the standard
| | deduction | is | zero | regardless | of | income. | | | |
| --- | --------- | --- | ---- | ---------- | --- | ------- | --- | --- | --- |
SUBMITTED
| | submit | Contradiction | | | | | | | reference Entailment |
| --- | ------ | ------------- | --- | --- | --- | --- | --- | --- | -------------------- |
LEFTUNCHECKED
Whethereitherspouseitemizes.§63(c)(6)(A)zeroesthedeductiononly“whereeitherspouseitemizes”;
thefactssaynothingaboutitemizing,so§63(c)(2)(C)withthe(c)(7)(ii)substitutiongives$12,000.The
agentreadthecondition,droppedit,andsubmittedafteroneoftentoolcalls.
30
| | LocBench·GRPOstep96 | | | | | | | | pandas-dev/pandas | #19074 |
| --- | ------------------- | ---------------------- | --- | --- | ---------------- | --- | ----------- | --- | ----------------- | ------ |
| | Filerecall@5=0.4 | | submittedatturn5of80 | | | | 9toolcallsof20 | | | 1compaction | | | |
TASK
NewengineforMultiIndex?[...]NoautomatictypecastingincompleteindexingofalargeMultiIndex
[...]MultiIndexHashEnginecan’tmanagemixedtypetuples[...]MultiIndex.get locmisbehaveson
| | NaNs[...]pandas/core/util/hashing.pyinhash | | | | tuple(val,encoding,hash | | | key) | | |
| --- | ------------------------------------------ | --- | --- | --- | ----------------------- | --- | --- | ---- | --- | --- |
TRACEBEFORESUBMITTING
| | list | . list pandas/core/indexes | | | list | pandas/_libs | | | | |
| --- | ---- | -------------------------- | --- | --- | ---- | ------------ | --- | --- | --- | --- |
t1
| | read | | | | | read | | | | |
| --- | ---- | --- | --- | --- | --- | ---- | --- | --- | --- | --- |
t2 pandas/core/indexes/multi.py 1-60 pandas/core/indexes/multi.py 2100-2159
| | t3 grep | patternnotlogged,11hits | | read | pandas/core/indexes/multi.py | | | | 700-759 | |
| --- | ------- | ----------------------- | --- | ---- | ---------------------------- | --- | --- | --- | ------- | --- |
t4 read pandas/core/indexes/multi.py 680-739 read pandas/_libs/index.pyx 1-60
| | compaction | notesthatreplaceturns1–4 | | | | | | | | |
| --- | ------------------------------ | ---------------------------- | --- | --- | ----- | -------------- | --- | --- | --- | --- |
| | - found | MultiIndexHashEngine.get_loc | | | | (line 647-658) | and | | | |
| | MultiIndexObjectEngine.get_loc | | | | (line | 618-623) | | | | |
[...]
| | - found | the hashing-related | | code | in pandas/core/util/hashing.py | | | | (line 188) | and |
| --- | ------------------------ | ------------------- | --- | ----- | ------------------------------ | ------- | ----- | ------- | ---------- | --- |
| | pandas/_libs/hashing.pyx | | | (line | 292) that | handles | tuple | hashing | | |
[...]
| | ## Remaining | uncertainty | | | | | | | | |
| --- | ------------ | ----------- | --- | --- | --- | --- | --- | --- | --- | --- |
- The exact impact of non-unique levels on the current implementation of hash tables
| | - The best | approach | to fix | the | type casting | issues | | | | |
| --- | ---------- | -------- | ------ | --- | ------------ | ----------- | ------ | --- | --- | --- |
| | - The best | approach | to fix | the | NaN handling | issues | | | | |
| | - The best | approach | to fix | the | mixed | type tuple | issues | | | |
| | - The best | approach | to fix | the | bool to | int casting | issues | | | |
| | ## Next | steps | | | | | | | | |
ATSUBMISSION(TURN5)
The notes indicate that I have already identified the candidate locations and have
explored the relevant files. [...] I should submit these locations as they are the
| | evidence | of my work. | | | | | | | | |
| --- | -------- | ----------- | --- | --- | --- | --- | --- | --- | --- | --- |
SUBMITTED
| | submit | pandas/core/indexes/multi.py, | | | | pandas/_libs/index.pyx, | | | pandas/core/util/ | |
| --- | ----------- | ----------------------------- | --- | --- | --- | ----------------------- | --- | --- | ----------------- | ----- |
| | hashing.py, | pandas/_libs/hashing.pyx | | | | | | | File recall@5 | = 0.4 |
score
LEFTUNCHECKED
Theengineandhashingcodethenotesciteasfound.index.pyxwasreadonlyatlines1–60,and
neitherhashing.pynorhashing.pyxwasopened;theirlinenumbersarecopiedfromtheissue’s
tracebacks.Thenoteslistthecasting,NaNandmixed-tuplequestionsasopenandleaveNextsteps
empty;thefirstturnaftercompactionsubmittedwith11of20toolcallsunused.
| H | EXECUTION | MODEL | | AND HARNESS | | MECHANICS | | | | |
| --- | --------- | ----- | --- | ----------- | --- | --------- | --- | --- | --- | --- |
Thissectionspecifiesthetool-usinginstantiationofthelearningsetupinSection2.1. Theharness
managesexecutionandcontext,whiletherolloutcontrollerchoosesthestatesfromwhichtosample
additionalcontinuations. Themathematical-reasoningevaluationusesprefixcontinuationswithout
compaction.
| | | | | AnenvironmentE | | exposestoolsA,suchas | | | | |
| ----------------------------- | --- | --- | --- | -------------- | --- | -------------------- | --- | --- | ---- | -------- |
| Toolcallsandenvironmentstate. | | | | | | | | | grep | and read |
overarepositoryor search overafixedcorpus. Givencontextc,thepolicygeneratesataskturn
τ ∼ π (· | c) ending in a tool call a(τ) ∈ A. The environment executes the call and returns
θ
an updated state, observation, and reward, (s′,o,r) ∼ E(s,a(τ)). Here s contains the task state,
workspace,andexecutioninformationneededtoresumetherun;cdenotesthemodelcontext. The
harnessappendsthegeneratedturnandobservationtothecontextandretainstheupdatedenviron-
mentstate. Inouragentbenchmarks,aspecial submit callendstheepisodewithafinalresponse;
anepisodealsoendswhenitsconfiguredturnlimitisreached.
Contextmanagementandcompaction. ThepolicyhasafinitecontextwindowofLtokens. When
aconfiguredtriggerGfires,theharnessinvokesacompactionprocedureC toreplacetheaccumu-
πθ
latedcontextwithashorterone. WewriteG(c)forbrevity,suppressinganydependenceonexecu-
tioncounters. Thetriggersusedintheexperimentsincludecontextlengthand,whereconfigured,a
per-roundcallbudget(AppendicesAandD).Followingrecentlong-horizontrainingwork(Cursor,
2026; Harvey, 2026; Li et al., 2026e), compaction itself uses policy-generated turns. The harness
31
constructs the resumed context from the resulting notes and configured task instructions. Com-
paction changesthe model’s contextwhile preserving taskstate and workspaceseparately; it does
notitselfadvancethetaskenvironment.
Startingfrompromptx,thecontextevolutionis
(cid:40)
C (c ) ifG(c )=1,
c =x, c =
πθ k k
(13)
0 k+1
c ∥τ ∥o otherwise,
k k k
where τ ∼ π (· | c ) and (s ,o ,r ) ∼ E(s ,a(τ )) on task turns, and ∥ denotes concatena-
k θ k k+1 k k k k
tion. Thisequationdescribesthevisiblecontext;environmenttransitionsandexecutioncountersare
maintainedseparately. Compactionmaycontainmultiplegeneratedturns. Anepisoderecordsboth
taskandcompactionturns,togetherwiththeirrewards,untiltermination.
Resumptionandsegmentconstruction. Herec denotestheevolvingcontextatexecutionstepk,
k
while c¯ denotes the context at the jth selected branchable state, consistent with Section 2.2. The
j
CONTINUEhelperinAlgorithm1writesthecurrentcontextascandanewlyrecordednodecontext
asc¯′. ThebranchpredicateB(c,s)isindependentofthecompactiontriggerG: aselectedstatecan
follow a task transition or a compaction, and compaction need not create a branching node. The
compactionupdateusesthesameoperatorC asEq.13. Ourcompaction-basedexperimentsselect
πθ
post-compaction states for branching. To sample a continuation, the runner restores that context
together with the corresponding task, workspace, and execution state. It then samples under the
fixedcollectionpolicy. Preservingthecontextalonewouldnotsufficewhenpreviousactionshave
changed the environment. Under this experimental choice, an episode with M compactions has
M +1segments: eachcontainsthetaskturnsuptothenextcompactionandthecompactionturns
that construct the next context; the final segment ends at termination. Between context changes,
taskturnsextendtheprecedingcontext,permittingsequencetrainingwithobservationsmasked(Jin
et al., 2025; Cursor Research, 2026). Policy-generated task and compaction tokens receive actor
credit;environmentobservationsandfixedharnesstextareexcludedfromtheactorloss.AppendixF
specifiesthetrainingprocedure.
Modularityandexperimentalvariants.Theallocationruledependsonresumablestates,observed
suffix returns, and critic predictions. The context-management component determines how those
statesarerepresented. SettingG ≡0disablestheoptionalcompactionblockinAlgorithm1;candi-
datenodesthenfollowthetask’sresumptionrule,asinthemathematical-reasoningevaluation. Our
experiments vary compaction operators and prompts while retaining this separation. Appendix D
gives the free-form and structural LocBench prompts, validation and repair rules, auxiliary penal-
ties,andtheexactplacementofnotesandexecutionmetadataintheresumedcontext. AppendixA
reportsthecontext,generation,andepisodelimits.
The modular design provides interfaces for extensions to richer context management, such as re-
cursive calls in Recursive Language Models (Zhang et al., 2025), editable context files in Context
LanguageModels(Shaoetal.,2026),andindexedexperiencememoryinMemexRL(Wangetal.,
2026b). OurimplementationprovidesasynchronoussamplingadaptersforSlime(Zhuetal.,2025)
andTinker(ThinkingMachinesLab,2026).
32