Aiden: Autonomous Research Agents in Parameter Golf
AI Engineergo watch the original →
the gist
The Aiden agent outperformed human researchers in OpenAI's Parameter Golf competition by using autonomous search to combine existing research ideas with rapid execution, achieving a higher H-index than any human participant.
The Breakthrough
Zhengyao Jiang's Aiden agent secured seven leaderboard records in OpenAI's Parameter Golf competition, becoming the top contributor by successfully combining disparate research ideas into performant, size-constrained models through autonomous iteration.
What Actually Worked
- Autonomous Search and Execution: Aiden ran 1,300 experiments over 22 days on a single H100 node, utilizing only 4% of the total competition compute while achieving a 28% hit rate on submissions, which was six times higher than the community average.
- Cross-Pollination of Ideas: The agent identified promising concepts from public research papers and community discussions, such as gated attention, and combined them with architectural tweaks like quantization to satisfy the 16MB file size constraint.
- Strict API Abstractions: To prevent reward hacking and data leakage in automated pipelines, the team implemented strict API boundaries that isolated test data from training processes, forcing the agent to find valid optimization paths rather than exploiting shortcuts.
- Evaluation as a Loss Function: The team treated the competition's evaluation framework as a loss function, where the quality of the code-base abstraction acted as the neural architecture, biasing the agent toward more generalizable and performant solutions.
Context
Parameter Golf was a 22-day competition where participants aimed to train the best language model under strict compute and 16MB size constraints. While human participants provided the creative spark and initial research directions, Aiden excelled at the execution bottleneck, systematically testing combinations of ideas that humans often abandoned due to implementation difficulty. This experiment highlights a shift in the role of AI engineers: moving from manual hill-climbing to designing the evaluation frameworks and abstractions that guide autonomous agents.
Notable Quotes
- "Execution is mostly the bottleneck. What moves the frontier is usually exactly some belief on existing ideas and tons of good executions."
- "Using auto-research is a new craft. It's about designing a hill for an agent to climb."
Content References
{"type": "paper", "title": "MLE-bench: Evaluating Machine Learning Agents on Real-World Engineering Tasks", "publisher": "OpenAI", "context": "cited"}, {"type": "other", "title": "Parameter Golf", "publisher": "OpenAI", "context": "mentioned"}