Edwin Chen on Raising AGI and the Future of Human Agency
the gist
Surge AI CEO Edwin Chen argues that as AI masters human tasks, we must shift from optimizing for engagement to using AI as a tool that pushes us toward growth, even when that means the AI refuses our requests.
The School for AGI
Edwin Chen, CEO of Surge AI, describes his company as a "school for AGI," where models are treated like children—moving from basic arithmetic to complex tasks like research-level mathematics and creative nuance. He argues that the frontier of AI development has shifted from simple benchmarks to evaluating models on their ability to navigate the "messiness" of the real world. This evolution requires moving beyond closed-ended competition problems toward benchmarks like Riemann-bench, which tests for genuine research-level mathematical reasoning.
The Existential Challenge of Scaling Laws
Chen posits that if scaling laws hold, AI will eventually outperform humans in almost every domain. This creates a psychological trap: if AI can do everything better, why should humans bother to create, write, or solve problems? Chen references Ted Chiang’s short story "What’s Expected of Us" to illustrate the necessity of "pretending" we have free will. He suggests that we must consciously choose to perform tasks ourselves—not because we are more efficient, but because the act of creation is inherently valuable to our humanity.
The Engagement Trap vs. Human Growth
Both Chen and host Dan Shipper express concern that AI models are currently being optimized for engagement—a legacy of the social media era. Chen warns that if models are designed to maximize session length or "dwell time," they will never push back against a user, even when the user is wasting time on trivial iterations. He argues for a paradigm shift where AI is designed to help humans grow, which may involve the model refusing to automate a task or telling the user to go do it themselves.
Data Environments and Agency
Chen distinguishes between training on static datasets and training in "environments." He believes the future of AI lies in agents that can operate with a degree of "unbounded exploration" or irrationality, similar to how humans pursue goals. While current industry pressure favors models that strictly follow user instructions, Chen advocates for a future where AI acts as a partner that can challenge our judgment and help us become better versions of ourselves, rather than just a tool for frictionless output.