Multiscale Indexing for RAG Retrieval

AI Engineer2026-09-16go watch the original →

the gist

Retrieval performance is query-dependent, making fixed chunk sizes suboptimal. By indexing documents at multiple window sizes and using Reciprocal Rank Fusion (RRF) to merge results, you can achieve 20-40% higher recall with minimal latency impact.

The Breakthrough

Retrieval accuracy is limited by the mismatch between fixed indexing chunk sizes and the varying granularity of user queries. By indexing the same corpus at multiple window sizes and merging the results using Reciprocal Rank Fusion (RRF), you can eliminate the performance penalty of choosing an arbitrary chunk size.

Multiscale Indexing Strategy

Duplicate the corpus: Create multiple versions of the dataset, each processed with a different chunk size (e.g., 50, 100, 200, 500, 1000, 2000 tokens).
Retrieve at scale: Query every version of the database for each user request.
Return documents, not chunks: To make rankings commensurable across different window sizes, return the parent document for every retrieved chunk.
Merge with RRF: Aggregate the document rankings from all versions using Reciprocal Rank Fusion to produce a single, unified result list.

Performance and Cost

Recall Gains: The multiscale approach consistently matches or outperforms the best possible fixed chunk size, yielding recall improvements of 20% to 40% across datasets like QMsum, NarrativeQA, Seinfeld, and FinanceBench.
Memory Overhead: This method requires two to five times the storage capacity compared to a single-index system, as the entire corpus must be stored for each chunk size.
Latency: Because retrieval from multiple indices can be executed in parallel and the RRF script is computationally inexpensive, the added latency is negligible.

Context

Standard RAG pipelines often rely on a single, fixed chunk size (e.g., 512 tokens) chosen at indexing time. This creates an information bottleneck: small chunks capture specific facts well but fail on broad queries, while large chunks capture context but lose precision. Because the optimal chunk size is a property of the query rather than the data, fixed-size indexing forces a permanent loss of information. The proposed multiscale approach treats retrieval as a voting problem across multiple granularities rather than a single search operation.