Battle-Testing Sakana Fugu Ultra Orchestration
Nate Herk | AI Automationgo watch the original →
the gist
Fugu Ultra is a multi-agent orchestration API that routes tasks to frontier models like Claude Opus and GPT. In a 38-task benchmark, it performed on par with Claude Opus 4.8 but proved 4.5 times slower and 5 times more expensive.
The Orchestration Model
Sakana AI's Fugu Ultra is not a standalone large language model but a multi-agent orchestration system. It functions as a single API endpoint that acts as a manager, breaking down complex tasks and delegating sub-tasks to specialized frontier models such as Claude Opus, GPT, and Gemini. Once the sub-agents complete their work, the system uses an additional LLM to synthesize the outputs into a final response. This approach mirrors dynamic workflows found in tools like Claude Code, where tasks are automatically routed to appropriate models based on their specific strengths.
Performance and Cost Analysis
To evaluate the system, the author ran 38 tasks comparing Fugu Ultra against Claude Opus 4.8. The tasks included puzzles, algorithmic challenges, and technical specifications. The results showed that 36 of the 38 tasks ended in a tie, with Claude Opus winning the remaining two. Despite the parity in output quality, Fugu Ultra incurred significant overhead in both time and cost. The total wait time for Fugu Ultra was 357 minutes compared to 80 minutes for Claude Opus. Financially, Fugu Ultra cost approximately $50, whereas Claude Opus cost $10 for the same set of tasks. The author concludes that while the orchestration pattern is a promising architectural direction for managing model unit economics, Fugu Ultra does not currently offer a performance advantage over using Claude Opus directly for general knowledge work.