SAGE: Adaptive Reasoning for Long-Form Video Agents
Y Combinatorgo watch the original →
the gist
SAGE improves video agent performance on long-form content by replacing reliance on unreliable temporal grounding with a tool-use architecture that integrates web search and transcripts, optimized via synthetic data and reinforcement learning.
Architecture and Tool Integration
Existing video agents struggle with long-form content because they rely heavily on temporal grounding models, which lack accuracy due to insufficient training on long-duration video data. SAGE addresses this by adopting a human-like approach that prioritizes world knowledge and context over raw visual processing. The system is equipped with external tools including web search and speech transcription, allowing it to bypass unreliable visual grounding when external signals provide more efficient or accurate information. This design significantly reduces runtime compared to previous systems that attempted to process entire video streams visually.
Synthetic Data and Training Methodology
To overcome the high cost of human-annotated video data, which can exceed $30 per hour, the team utilized Gemini 1.5 Flash to generate synthetic question-answer pairs and tool-call trajectories. This synthetic data serves two purposes: providing a cold-start for supervised fine-tuning (SFT) and enabling reinforcement learning (RL) to teach the model adaptive reasoning. The RL recipe uses an LLM-as-a-judge to evaluate open-ended responses. The reward function penalizes the model for unnecessary tool calls that result in incorrect answers and provides a +0.25 reward bonus when the model successfully uses visual temporal grounding to arrive at a correct answer. This encourages the model to learn when visual tools are reliable and when they should be avoided in favor of other information sources.