paperarXiv (systems, data, ML)·27 Oct 2026·Zhengping Jiang

FOCUS: Active Critic Learning for Long-Horizon Language Model Agents

FOCUS trains critics actively to give better intermediate value estimates for long-horizon LLM agents with sparse rewards.

Read original on arxiv.org Read PDF

llm-agents Reinforcement Learning

Depth 4/5Score 4.94

Paper (from PDF)

Read full text

FOCUS: ACTIVE CRITIC LEARNING FOR LONG- HORIZON LANGUAGE MODEL AGENTS ZhengpingJiang WilliamJurayj LiaoyaqiWang ChunshengZuo AnqiLiu BenjaminVanDurme JohnsHopkinsUniversity {zjiang31, aliu.cs, vandurme}@jhu.edu ABSTRACT Sparseoutcomerewardsmakeitdifficultforlong-horizonlanguage-modelagents toassigncredittoearlieractions. Learnedcriticsprovidevalueestimatesforin- termediatestates,buttheseestimatescanbeinaccurate,whileestimatinginterme- diate values with repetitive continuation samples from points within a trajectory incurs substantial costs. To mitigate these issues, we introduce FOCUS, an RL methodthatusesselectivebranchingtohelpthecriticnetworklearnitsvaluees- timator. Therolloutcontrollerfacilitatesactivelearningforthecriticbysampling continuationsfromstatesthatareexpectedtomostreducethecritic’sestimation error. These refined value estimates then score the segment between consecu- tivestateswithaTD(λ)advantage,whilethesampledevidencesuppliestraining targetsforthecritic. FOCUSsupportslong-horizonagenticreinforcementlearn- ingthroughamodularcontext-managementinterfacethatdecouplescontextman- agement from rollout allocation. Our experiments show that using compaction- inducedsegmentboundariesforvalueestimationandcreditassignment,FOCUS outperforms GRPO and VinePPO with matched per-step generation budgets on legalreasoning,deepsearch,andcodelocalizationbenchmarks,improvingheld- out accuracy at Qwen3.5-9B scale by 7.2 points over the strongest baseline on BrowseComp-Plus (55.4 vs. 48.2) and by 4.1 points on DeonticBench (56.5 vs. 52.4). 1 INTRODUCTION Language-modelagentsareincreasinglytrainedtosolvetasksthatunfoldoverdozensorhundreds oftoolcalls: resolvinganissueinarealrepository(Jimenezetal.,2024;CursorResearch,2026), completing a task on a live website (Zhou et al., 2024b), or answering a question that requires manyroundsofsearchandreading(Chenetal.,2026;KimiTeam,2026). ReinforcementLearning with Verified Rewards (RLVR) is the standard way to improve such agents, but long trajectories meanbothsamplingandcreditassignmentarecostly. Asingleverifiablerewardmustbedistributed acrosseverytokenwithinarolloutevenwhenmanyhadlittleimpactonthefinaloutcome. Tolearn efficiently,weneedproperintermediatevalueestimatesunderalimitedrolloutbudget. Theseestimatescancomefromalearnedcriticnetwork(Konda&Tsitsiklis,1999)orthroughMonte Carloapproximationfromagivenstate(Shaoetal.,2024).Alearnedcriticreusesinformationacross trajectories, but its predictions can be inaccurate. …

Explained

What happened

Training LLM agents with reinforcement learning on long tasks (dozens or hundreds of tool calls) has a credit problem: there is one reward at the end, like "answer correct", and it must be spread across every step, most of which did not matter. FOCUS (Focused Optimization on Critical Underperforming Segments), from Johns Hopkins, spends the rollout budget where it most improves those per-step estimates.

Steps come from context compaction. Long-running agents periodically compress their history into a summary and resume from it. FOCUS treats each segment between compactions as one step, and each compaction point as a state you can fork from.
A critic scores each state. A learned value network predicts the chance of success from that point.
Branch where the critic is most likely wrong. The rollout controller compares critic predictions with real outcomes and samples extra continuations from states where more evidence is expected to cut the critic's error the most. This is active learning for the critic.
Train both sides. The refined values give each segment a TD(lambda) advantage for the policy update, and the sampled continuations become training targets for the critic.
Results: with matched per-step generation budgets, FOCUS beats GRPO and VinePPO on legal reasoning (DeonticBench), deep search (BrowseComp-Plus) and code localization (LocBench). At Qwen3.5-9B scale it gains 7.2 points on BrowseComp-Plus (55.4 vs 48.2) and 4.1 points on DeonticBench (56.5 vs 52.4).

Why it works

There are two classic ways to value an intermediate state. A learned critic is cheap and reuses knowledge across episodes, but it can be badly wrong. Monte Carlo sampling (run several continuations from the state and average the reward) is accurate but expensive, because each sample is a full LLM rollout.

FOCUS mixes them and treats sampling as a budget to allocate. Most states are easy and the critic is fine. A few are uncertain or mispredicted, and those are where extra rollouts pay off. Using compaction points as states also gives natural, coarse steps, so credit is assigned per meaningful chunk of work rather than per token.

The bigger pattern

This is budgeted exploration, the same idea behind active learning, bandits and adaptive sampling in A/B tests: spend measurement where uncertainty is highest. In RL terms it sits between critic-free methods like GRPO (cheap, noisy credit) and full tree search (accurate, costly).

What is new is tying the step structure to context compaction, which long-horizon agents already do for memory reasons, and keeping context management separate from rollout allocation so either can change.

What it means in practice

If you fine-tune agents with RL, outcome-only rewards on long tasks waste compute. Adding a critic with targeted branching can buy accuracy at the same budget.
Design agent loops with clean checkpoint states (compaction summaries, saved tool state). They help with resumability, debugging and training alike.
For evaluation, sample more runs on tasks where results are uncertain instead of evenly across a benchmark.
The same budgeting logic applies to offline replay and load testing: re-run where outcomes vary most.