RAG is Evolving into Iterative Agentic Retrieval
AI Engineergo watch the original →
the gist
RAG is not dead; it is evolving from simple vector search into iterative, agentic retrieval that combines semantic search, full-text search, and filtering to provide context to LLMs efficiently.
The Shift to Iterative Agentic Retrieval
Retrieval-Augmented Generation (RAG) is not obsolete, but its definition has matured beyond a single-shot vector database query. Modern agentic search uses an iterative loop where an agent performs multiple, varied retrieval steps—including vector search, full-text search (BM25), globbing, regex, and metadata filtering—until it gathers sufficient context to solve a task. This approach treats embeddings as cached compute, allowing agents to perform lightweight lookups at runtime rather than relying on expensive, repetitive file-system grepping.
Performance Gains and Implementation
Cursor demonstrates the efficacy of this approach by indexing codebases upfront using Merkle trees to identify and update only changed files. This strategy provides measurable improvements in model performance:
- Internal benchmarks showed a 13.5% average increase in answer accuracy across models.
- The Composer model specifically achieved a 24% increase in answer accuracy.
- Online A/B testing revealed a 2.6% gain in code retention for large codebases and a 2.2% decrease in dissatisfied user requests.
These gains are significant because semantic search is not triggered for every query, meaning the impact on relevant queries is higher than the aggregate percentage suggests. As Jeff Dean noted, the goal is not to feed an LLM a trillion tokens at once, but to use retrieval to narrow down a massive corpus to the "right million" tokens for the current context.