Benchmarking Vector Databases with FineWeb-10B

The Problem with Current Benchmarks

Most public vector database benchmarks are misleading because they use small datasets (e.g., 1 million vectors) that fit entirely in RAM, masking real-world performance issues like memory pressure, index build times, and tail latency. Furthermore, many benchmarks rely on approximate ground truth or synthetic vectors, which fail to capture the clustering complexity of real-world embeddings. Generating accurate ground truth for large datasets requires exhaustive, brute-force distance calculations, which is computationally expensive and rarely performed.

The FineWeb-10B Solution

Qdrant released FineWeb-10B, a dataset containing 10 billion vectors derived from Hugging Face's Common Crawl. Each document is embedded twice using the GTE multilingual base model: once as a dense vector for semantic meaning and once as a sparse vector for keyword-like retrieval. The dataset includes 120,000 queries with pre-computed ground truth for the top 1,000 results, allowing developers to measure recall at varying depths without re-running exhaustive computations. The accompanying open-source toolchain, Supernova, allows users to generate embeddings, load data, and perform load testing on their own hardware.

Performance Observations

Testing a 1-million vector slice of the dataset across Qdrant, Milvus, Elasticsearch, and pgvector reveals that performance varies significantly based on accuracy requirements. Elasticsearch proved to be the fastest at 94% recall (7.5ms), but it hit a performance ceiling and could not exceed 95% accuracy. In contrast, Qdrant and Milvus achieved up to 98% recall, demonstrating that while Elasticsearch is efficient for approximate retrieval, dedicated vector databases maintain higher precision at the cost of latency.