The Shift Toward Continual Learning in AI Agents

Dwarkesh Patel2026-06-26go watch the original →

the gist

Current AI progress relies on RLVR in reproducible environments, but future scaling requires 'continual learning'—distilling real-world deployment experience back into model weights via techniques like On-Policy Self-Distillation (OPSD) and test-time 'dreaming'.

The Limitations of Current Training Paradigms

Current AI research focuses on Reinforcement Learning from Verifiable Rewards (RLVR) across containerized, deterministic environments. While this produces agents capable of solving complex tasks in coding or math, it struggles with 'real-world' domains like business operations or politics. These domains are not easily 'grindable' because they lack replayable simulators and require long-horizon, reset-free interaction. The current reliance on pretraining and short-horizon RLVR creates a sample-efficiency gap that cannot be closed simply by scaling compute if the training targets remain unverifiable.

Distilling Experience via OPSD and Dreaming

To move beyond static models, AI must transition to continual learning, where deployment experience is distilled back into the base model weights. This avoids the memory constraints of infinitely expanding context windows and the sample inefficiency of naive supervised fine-tuning.

On-Policy Self-Distillation (OPSD): Instead of training on raw transcripts, OPSD encourages the base model to match the predictions of a 'veteran' model that has already accumulated context-based experience. This provides a denser supervision signal than RL while maintaining the sparsity of updates, preventing the model from overwriting existing knowledge.
Dreaming (Test-Time Training): This speculative approach involves the model using inference-time compute to generate its own RL environments or simulations based on real-world observations. By rehearsing alternative strategies against these self-generated simulators, the model can achieve orders of magnitude more 'experience' than it receives from real-world data alone.

The 2027 Paradigm

By 2027, the primary driver of AI improvement may shift from pre-release training to post-deployment learning. In this scenario, agents are deployed to perform real-world work, and successful sessions—validated by user feedback—are distilled into the base model. This creates a feedback loop where the model improves incrementally across diverse, organization-specific tasks, allowing it to expand its capabilities into domains far beyond its initial training distribution.