Distillation vs. Output Harvesting in LLM Training

The Mechanics of Knowledge Distillation

Knowledge distillation is the process of compressing a large, computationally expensive teacher model into a smaller student model. In image classification, this involves training the student to replicate the teacher's full probability distribution across all classes, rather than just the hard label. By dividing the teacher's output logits by a temperature parameter, the distribution flattens, revealing 'dark knowledge'—the non-zero probabilities assigned to incorrect classes that provide insight into how the model generalizes. To achieve true capability transfer in LLMs, the student must similarly have access to the teacher's log probabilities for the entire vocabulary during training.

Distillation Types and Limitations

Distillation methods vary based on the level of access to the teacher model:

• Logit/Feature Distillation: Requires access to internal weights, activations, or full log probabilities. This is the only method that effectively transfers reasoning capabilities.
• On-Policy Distillation: The teacher acts as a judge or reward model, evaluating the student's generated tokens during training. This approach is highly efficient, requiring roughly 1,800 GPUs compared to 18,000 for standard reinforcement learning, but it necessitates access to the teacher's log probabilities.
• Sequence-Level Distillation (Output Harvesting): This method involves sampling finished outputs from a teacher API and using them for supervised fine-tuning. Because this lacks access to the teacher's internal probability distribution, it functions as imitation learning rather than capability transfer. It copies the style and behavior of the teacher but fails to replicate the underlying reasoning logic.

The Reality of Modern Training Pipelines

Modern LLM development relies on a multi-stage pipeline: pre-training, mid-training, supervised fine-tuning (SFT), and reinforcement learning (RL). Pre-training and mid-training define base capabilities and are typically locked months before a model release. While output harvesting is often conflated with distillation in news reports, it is insufficient to bridge the gap between a base model and a frontier-level model. For instance, achieving frontier-level performance via RL typically requires 20 to 40 million rollouts, a scale that is practically impossible to achieve through public API access due to rate limits and the lack of logit visibility.