Scaling Legal Document Search with Namespace-Based Storage

The Breakthrough

Legora resolved a catastrophic 20-second P99 search latency by shifting from a sharded PostgreSQL architecture to a namespace-per-project model in Turbopuffer, which treats object storage as the primary data source and eliminates the need for complex partition management.

What Actually Worked

• Namespace Isolation: Each legal project is assigned its own namespace, which acts as an atomic unit of storage. This allows idle projects to reside in object storage at near-zero cost while active projects are cached in memory.
• Object Storage Native Writes: Turbopuffer writes data directly to object storage (S3) rather than relying on disk replication or Paxos, optimizing for the 200ms P99 latency of S3 blobs.
• Encryption and Compliance: By using the namespace as the unit of encryption, Legora provides enterprise clients with physical data isolation and customer-managed encryption keys, allowing for granular key revocation.
• Tree-Based Vector Indexing: Instead of graph-based navigation, which requires frequent random round trips to object storage, the system uses a tree-based clustering approach. This keeps root centroids in DRAM while pushing leaf nodes to SSDs or object storage, minimizing round trips.
• Cache-Free Operation: The team discovered that the performance of Turbopuffer without an encrypted SSD cache was sufficient for their needs, allowing them to bypass the complexity of implementing encryption at the disk-cache layer.

Before / After

• Latency: The P99 search and ingestion latency improved from 20 seconds back to approximately 100 milliseconds.
• Operational Complexity: The system moved from maintaining multiple regional ElasticSearch clusters and partitioned PostgreSQL instances to a single, unified Turbopuffer cluster that handles multi-tenancy natively.

Context

Legora initially scaled by sharding document chunks across 4,000 PostgreSQL partitions. This approach failed because active and inactive projects were co-located in the same partitions, causing the database to constantly thrash the cache by pulling massive, mostly-dead partitions into memory. The migration to Turbopuffer allowed the team to decouple storage from compute, enabling them to scale to billions of vectors while maintaining cost efficiency for a long tail of infrequently accessed legal documents.

Content References

• Tool: Turbopuffer, cited as the primary search engine solution.
• Tool: PostgreSQL, cited as the previous search infrastructure.
• Tool: Elasticsearch, cited as the initial search infrastructure.