Scaling Compute on Context: The Search for Model Depth

The Limitation of Public-Data Scaling

Modern AI progress relies on three axes: more data, more compute, and larger models. These axes are currently tethered to public datasets like Wikipedia, GitHub, and arXiv. Because these models are trained on public information, they lack the depth required to understand private, unstructured data such as personal emails, company meeting transcripts, or proprietary codebases. While naive fine-tuning on such data can achieve near-zero loss, it results in model collapse where the model fails to generalize or answer questions outside of the exact training examples.

The Search for Recursive Depth

To move beyond the limitations of standard fine-tuning, researchers are exploring methods to scale compute on private context. Current approaches include:

• KV Compaction: Compressing long-context data into succinct key-value representations to fit within the model window.
• On-Policy Distillation: Updating the model during training to treat the private corpus as if it were in-context, often using synthetic question-answer pairs generated from the source data.
• Continued Pretraining: Crafting synthetic data conditioned on the private corpus to simulate pretraining, though this risks overwriting existing base-model knowledge.

These methods eventually hit a synthetic data wall where the model saturates and stops learning. The objective is to replicate the success of systems like AlphaGo, where the training process becomes recursively harder as the model improves. By creating a feedback loop where the model generates better training questions as it gains depth, researchers aim to move from a fixed data budget to a system where adding compute consistently increases the model's understanding of a specific domain.