On-Policy Self-Training in Large Language Models

The paper introduces On-Policy Self-Training (OPST), a data-free post-training method for large language models that generates synthetic problems and uses teacher-student distillation. OPST outperforms traditional methods like SFT and GRPO, achieving up to 10.2 points improvement in MMLU benchmark, and up to 6.9 points in VQAv2; it relies on synthetic context and parameters filtering, not literal target answers Experiments show OPST's effectiveness without curated supervision or answers across various benchmarks, including transfer to coarse Qwen3-4B model Problem setting and central contribution On-Policy Self-Training (OPST) addresses data-free post-training for language and vision-LLMs. Existing self-improvement methods typically eliminate one source of supervision while retaining another: SFT requires curated demonstrations, RLVR requires verifiable answers, on-policy distillation requires a stronger teacher, and on-policy self-distillation generally requires privileged reference solutions. OPST removes both the curated problem distribution and externally supplied answers. Its only textual input is a set of coarse topic descriptors, such as “Abstract Algebra” or “Commonsense Causal Reasoning.” From these descriptors, the model generates its own questions, rationales, and candidate answers, filters the resulting data, and trains through on-policy self-distillation (2610.32770). The paper’s principal claim is that a model can improve by exploiting latent capabilities already encoded in its parameters, provided that synthetic information is used as privileged context for a teacher rather than treated as an unquestionable training target. The distinction is important. SFT on a generated rationale assumes that every generated token is correct; reinforcement learning based on agreement with a generated answer compresses the information into a scalar reward. OPST instead conditions an EMA teacher on the synthetic rationale and answer, while the student sees only the question. The teacher consequently supplies token-level distributions along trajectories sampled from the student’s current policy. This permits the teacher to complete omitted reasoning steps, reinterpret imperfect candidate answers, and express supervision in its own distribution rather than in the surface form of the synthetic trace. The contribution is therefore not merely synthetic data generation. It is the combination of self-generated curriculum construction, intrinsic filtering, …