Scaling the Hugging Face Hub to 3 Million Models

Search Architecture and Metadata Optimization

Hugging Face maintains search performance by decoupling metadata from binary model artifacts. Metadata is stored in MongoDB Atlas, while model files and datasets reside in AWS S3. To handle high-volume search queries, the team moved away from regex-based filtering, which failed to scale, and implemented Apache Lucene via the MongoDB Atlas $search operator.

• Tokenization: Model names are tokenized at insert time rather than query time. For example, meta-llama/llama-3.1-8b is split into an array of searchable tokens like meta, llama, and 3.18b.
• Read Optimization: The system uses a denormalized read-only collection in MongoDB specifically for listings, preventing heavy search traffic from impacting the primary write collection.
• Ranking: Results are sorted by a trending score calculated every five minutes, based on the sum of downloads and likes from the previous seven days.

Infrastructure and Scaling Strategy

The infrastructure utilizes a seven-node MongoDB cluster to distribute read load, while reserving the primary node exclusively for writes. A hidden analytics node replicates data from the primary but remains invisible to the application, serving as the target for heavy reporting and ad-hoc queries.

• Kubernetes Autoscaling: The Hub scales from 10 to 500 pods using a two-layer approach. Horizontal Pod Autoscaler (HPA) manages pod counts based on CPU and memory, while CastAI handles infrastructure-level node provisioning when capacity is reached.
• Event-Driven Scaling: The team is migrating from standard HPA to KEDA (Kubernetes Event-driven Autoscaling). Unlike HPA, which only monitors resource utilization, KEDA scales based on application-specific metrics like request-per-second and event-loop utilization, allowing the system to react to request queues that do not necessarily spike CPU usage.
• Horizontal Sharding: To prepare for future growth beyond the capacity of a single replica set, the architecture is designed for horizontal sharding, where data is partitioned across multiple shards to scale CPU, memory, and storage independently.