Scaling Continual Learning with On-Policy Self-Distillation

The Breakthrough

On-policy self-distillation (OPSD) replaces external teacher models and parallel rollouts with a self-referential training loop where a student model learns by matching its own log-probabilities against a version of itself prompted with privileged information (hints), enabling per-token optimization on online task distributions.

What Actually Worked

• Self-Distillation via Hinting: Instead of relying on a larger teacher model, the student is prompted with a "hint" (e.g., a golden solution or environmental guidance) to create a more capable teacher state. The model is then trained to match the log-probabilities of the unhinted student to the hinted teacher.
• Step-Level KL Weighting: To prevent the "but wait" problem—where long-horizon models collapse into hedging behaviors like "maybe" or "but" due to teacher course-correction—the training applies a divergence-based weighting factor to each token. This allows the model to ignore or mildly shift distributions when a trajectory diverges significantly from the expected path.
• Residual Guidance: To mitigate "hint leakage" (a form of reward hacking where the model shortcuts reasoning by stating the answer provided in the hint), the system uses a linear combination of partial and full hints. This prevents the model from being pushed into out-of-distribution territory by overly strong guidance.
• Vocabulary-Wide Optimization: Unlike RL methods that sharpen only the sampled token, OPSD optimizes across the entire 65k vocabulary at every step, allowing the model to shift its distribution toward better tokens even if they were not the ones originally sampled.

Context

Traditional reinforcement learning methods like GRPO require massive parallel infrastructure and collapse complex trajectories into a single scalar reward, which is inefficient for real-world agentic tasks. OPSD addresses these bottlenecks by enabling on-policy learning from single examples with dense, per-token feedback. This approach is particularly effective for long-horizon tasks, such as agentic tool-calling, where the model must maintain coherence over 100+ steps without collapsing into suboptimal hedging behaviors.

Content References

• Tool: OpenClaw, mentioned as an example of an open-source project compatible with OPSD.