MiRL: Error-Targeted Reinforcement Learning for Correcting Terminal Agent Misbehaviors

What happened

Terminal agents solve multi-step tasks by running shell commands. They usually get better by training on more environments: one strong recipe, TMax, samples from 14.6K environments and takes two to three days on eight H100 nodes. Researchers at UNC Chapel Hill tried a more targeted route called MiRL (misbehavior-driven RL): find the agent's recurring mistakes, then train directly at the moments they happen.

1. Profile errors. They ran Qwen3.5-4B on 2,017 environments. An error localizer flagged wrong steps, a teacher model corrected each one so the run could continue and reveal later errors, and an annotator model grouped wrong steps into types.
2. Pick generalizable misbehaviors. Eight error types, such as "makes false claims about tests", "repeats unnecessary checks", "retries without fixing the cause" and "fails to produce runnable commands", covered 52.1% of all wrong steps and showed up in 75.4% of environments.
3. Start rollouts at the mistake. Instead of starting each training rollout from the beginning, MiRL rebuilds the container state by replaying the commands just before a misbehavior, keeping the prefix only if every command returns the same output as recorded.
4. Reward both outcome and behavior. A rollout earns reward only if the tests pass and a behavior judge confirms it avoided that specific misbehavior.

Results:

• On held-out prefixes, the trained agent repeated the targeted misbehaviors 43% to 57% less often.
• With the same 189 training environments, MiRL beat TMax-189, PivotRL and an outcome-augmented baseline by at least 4.2 points on Terminal-Bench Lite and 2.2 points on Terminal-Bench 2.1.
• It beat the full released TMax-4B on Terminal-Bench Lite (51.8% vs 47.4%) and matched it on Terminal-Bench 2.1 (21.3% vs 20.6%) while using 35% of its rollouts.
• Ablations: starting from misbehavior prefixes added 7.9 points and the behavior reward added 6.5.

Why it works

A failed 50-step task usually fails because of one or two bad steps. Training from step 1 wastes most compute on steps the agent already does well, and the agent may never even reach the bad state. Restarting at the prefix puts every rollout exactly where learning is needed.

Targeting error types that recur across tasks, rather than one task's bug, means the fix generalizes. The extra behavior check closes the loophole where an agent passes tests by luck while still, for example, claiming tests passed without running them.

The bigger pattern

This is error-driven improvement, the same loop good engineering teams use: classify incidents, find the few root causes behind most of them, and fix those. In ML it combines two known tools, step-level error analysis and RL from intermediate states, and shows they beat brute-force data scaling on efficiency.

What it means in practice

• If you run coding or ops agents, log trajectories and classify failures. A handful of types will dominate, and "claims tests passed without running them" is common.
• Add cheap checks for those behaviors in your harness: verify that test commands actually ran, cap retries without changes.
• For fine-tuning, replaying to a failure state and training from there is far cheaper than full-episode rollouts.
• Make agent environments replayable (deterministic containers, recorded command outputs). It helps debugging and training alike.