Building Native Multimodal Agents with Gemini

Multimodal Understanding and Context

Gemini models support native ingestion of text, code, images, audio, and video. Developers can use the File API to upload large assets, including YouTube URLs, directly into the model context. For long-form content, Gemini supports up to 1 million tokens, which translates to approximately nine hours of audio or one hour of video. To optimize costs during repeated queries on large files, developers should utilize context caching, which can reduce expenses by 90 percent.

Agentic Multimodal Generation

Rather than using hardcoded pipelines, developers can build agentic loops where a reasoning model (Gemini 1.5 Flash) decides which modalities to generate based on the input data. This is achieved through function calling, where the agent is provided with tool declarations for image and speech generation. The agent analyzes the synthesized content and invokes specific tools when it determines a visual diagram or audio summary is required. Native generation models, such as the image generation model (internally referred to as Nano Banana 2), leverage the world-understanding capabilities of the base Gemini architecture, allowing for tasks like generating images from annotated maps or correcting math homework with visual overlays.

Real-Time Interaction

The Live API utilizes a native audio-to-audio model based on Gemini 1.5 Flash. This architecture processes audio input and generates audio output directly, eliminating the latency and complexity of cascaded pipelines (e.g., ASR to LLM to TTS). This enables natural, low-latency conversational agents capable of processing simultaneous video and audio streams.