Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox
ThinkingBox turns database end-state checks into RL rewards for post-training agents, with training code released.
Extract
This is the follow up to our first blog post, The Agent Said It Was Done. The Database Disagreed. ThinkingBox-Training is now available on GitHub. We built ThinkingBox to evaluate an agent's work by comparing the final database state with the expected outcome. We then used the same kind of executable signal as a reward on training tasks outside ThinkingBox-Bench. For Qwen3.8-27B, average success rose from 51.70% to 60.78%, while tasks completed correctly on all twenty attempts rose from 38 to 107 with 50 GRPO steps. Figure 1. ThinkingBox turns executable checks of terminal database state into rewards for post-training agents on tasks outside ThinkingBox-Bench. This is a joint blog by Microsoft and Hugging Face, special thanks to Mirco Milletari and Xiang Jiang from Microsoft, Sergio Paniego from Hugging Face and our former intern Zhuochun Li (University of Pittsburgh) for co-authoring/reviewing efforts. We built ThinkingBox to catch agents that say they are done when they are not. Each of its 507 tasks in ThinkingBox-Bench (our benchmark dataset) hands an agent a real backend and a set of tools, lets it work, then checks what really changed in the database. In the first post, we caught a model closing a customer's ticket as resolved while the carrier exception behind it was still open. Nine competent-looking tool calls, one wrong database row. Then we noticed what we had built. For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function.+ So we constructed a training pool from ThinkingBox-format tasks outside the benchmark and used their executable outcomes as rewards. The catch was choosing which tasks are suitable for agent training: most tasks are either already solved or hopeless for the model in front of you, and both teach it nothing. Difficulty is policy-relative: a task is hard for the current model only until that model learns it. We call this offline policy-relative difficulty. We estimated each task's success probability once using the base policy before training and kept the selected training pool fixed throughout the run. What happened after training We used offline policy-relative difficulty to build the training pool, then fine-tuned Qwen3.8-27B with GRPO for 50 steps. To recap, we evaluate each of the 507 benchmark tasks 20 times. pass@1 measures average success across all 10,140 attempts. pass@20 measures the share of tasks solved at least once. …
Explained
What happened
Microsoft and Hugging Face turned their agent benchmark into a training signal.
- ThinkingBox-Bench has 507 tasks. Each one starts a real backend, gives the agent MCP tools and a simulated user, and at the end checks the final database state, not the tool calls.
- The team noticed that a check returning 1 or 0 from the database is already a reward function.
- They built a separate pool of tasks in the same format and picked 187 of them by running the base model first and keeping tasks it solved sometimes but not always. They call this offline policy-relative difficulty.
- They trained
Qwen3.8-27Bwith GRPO for 50 steps (18 groups of 8 rollouts per step, 24 H100s). Every rollout was a full episode with a fresh backend. - On the held-out benchmark, average success rose from 51.7% to 60.8%. Tasks solved on all 20 attempts rose from 38 to 107. Tasks solved at least once fell slightly, from 453 to 446.
- Cost per task that passes every time dropped from $24.36 to $8.00, close to much larger closed models.
Why it works
Two choices do the heavy lifting.
- Verifiable outcomes. The reward checks the record that matters, for example whether a booking was changed without manager approval. The agent cannot win by sounding done. It also may find a different valid path, since only the end state is graded.
- Useful difficulty. Tasks the model always solves or never solves give GRPO no signal, because every rollout in the group gets the same reward. Keeping only the tasks with mixed results makes every training step count.
The training and evaluation code also parse model output with the same interpreter, so the reward measures the same agent that gets evaluated.
The bigger pattern
This is RL with verifiable rewards applied to business workflows instead of math or code. The result matches a known effect called sharpening: RL makes the model more consistent on paths it already knew, while some rare successful paths disappear. Nothing here is a new algorithm. What is new is a clear recipe for agent tasks plus evidence that pass@1 hides reliability: two models with nearly equal average scores can differ by 2x on tasks they always get right.
What it means in practice
- When you evaluate an agent that writes to production data, run each task many times and report the share that passes every time, not just the average.
- Grade on the state of the system after the run. For a Postgres-backed service, write checks as SQL queries on the final rows.
- The same checks can be reused as rewards if you fine-tune a smaller model, which can cut serving cost a lot.
- Expect a tradeoff: more repeatable behavior, slightly less coverage of rare cases. Keep fallbacks or human review for the long tail.