Continual Learning via Distillation and Hinting
AI Engineergo watch the original →
the gist
Applied Compute uses a 2x2 matrix of offline/online traces and hints to perform continual learning in enterprise agents, enabling behavior modification without requiring golden datasets.
The Distillation Spectrum
Applied Compute categorizes continual learning based on two axes: the source of the production traces (offline vs. online) and the source of the hints (offline vs. online). Offline distillation processes static batches of production traces, while online distillation integrates inference and training into a continuous flywheel. The goal is to improve agent behavior without relying on golden answers or perfect rubrics.
The 2x2 Distillation Matrix
- Offline Traces + Offline Hints: Best for immediate value. By applying static rubrics to existing production logs, agents can be nudged toward specific behaviors, such as reducing refund frequency or modifying reasoning patterns.
- Online Traces + Online Hints: The most scalable approach for raising performance ceilings. By dynamically constructing hints based on the current on-policy rollout, the system adapts to specific agent behaviors in real-time, which is critical for out-of-distribution tasks like specific hyperlink formatting.
Implementation Techniques
- Per-Step Hinting: Instead of injecting hints at the start of a rollout, use an LLM judge to identify the specific turn where a correction is needed. Distillation is most effective when applied to the immediate next step rather than the entire sequence, as the KL divergence signal weakens as the model moves further from the hint.
- Relevance Masking: Use an LLM judge to sample and mask tokens during distillation. This prevents the student model from overfitting to the teacher's stylistic preferences or filler words, focusing learning only on the relevant behavioral changes.
- On-Policy Stepping: When working with offline traces, performing a single on-policy step allows the model to generate a more relevant hint, which significantly improves the pass rate compared to purely offline methods.
Performance Results
- SWEBench Reasoning: By injecting a hint to finalize tasks before turn 40, the task completion rate increased from 22% to 60% without degrading the test pass rate.
- Hyperlink Formatting: Online hinting improved correct formatting from 15% to 80%, whereas offline hinting showed significantly lower gains for the same out-of-distribution task.