Data Markets and the Shift to Process-Based Training

The Shift to Process-Based Data

Model performance is currently limited by a lack of high-quality, process-based data. While static data (rows in an ERP) served the 2023 era of next-token prediction, modern expert-level performance requires process-based data: the trajectory, reasoning traces, and sequence of decisions that professionals use to complete work. The author distinguishes between Type 1 data (real-world workflows like GitHub commits or session replays) and Type 2 data (contrived examples generated by experts in artificial settings). Type 2 data is often sold as Type 1, but it fails to scale quality because it lacks the inherent realism of live business workflows.

Verifiability as a Bottleneck

The ease of training a model for a specific domain is governed by "Verifiers Law," which posits that training difficulty is proportional to task verifiability. Verifiability is measured across three axes: asymmetry (decomposability into checkable steps), veracity (consensus on correctness), and proliferation (availability of fresh, verified examples). Coding succeeded as an early AI application because GitHub provided high scores across all three axes. Domains like biology, security, and finance are currently harder to automate because they lack these public, verified reasoning traces, forcing labs to aggressively acquire private enterprise logs.

The Failure of Benchmark-Driven Development

Most current benchmarks suffer from "benchmark psychosis," where the measure becomes the target, leading to Goodhart’s Law with a profit motive. Labs and vendors often create contrived benchmarks, sell the data to climb those benchmarks, and then fail to test for long-horizon reasoning. The author’s internal testing shows that model performance often diverges significantly from public leaderboards. For instance, comparing GPT-4.8 and Opus 4.08 on finance tasks (AR waterfall reconciliation, LBO memos) reveals that models often trade off arithmetic accuracy for methodological correctness, proving that single benchmark numbers are merely noisy samples from unmeasured distributions.

Building Durable Data Pipelines

To build a defensible product, developers must decouple their infrastructure from foundation model labs. The author argues that data companies must pivot to enterprise-focused services, specifically by building "Antikythera mechanisms"—bespoke systems that translate messy business context into evaluation rubrics. Successful builders should focus on creating pipelines that allow for automated post-training whenever a new base model is swapped in, rather than relying on static datasets that depreciate as the frontier moves.