Epic Games' Lore: A Version Control System for Large Binaries

The Breakthrough

Epic Games released Lore, a version control system built in Rust specifically for large binary assets like textures and 3D models, which uses content-addressed chunking and Zstandard compression to avoid the storage and performance limitations of Git and Git LFS.

What Actually Worked

• Chunk-based storage: Lore breaks large binary files into smaller chunks, hashes them, and stores them in a content-addressed Merkle tree, allowing the system to reuse existing chunks when only parts of a file change.
• Local-first workflow: Users perform branching, committing, and switching locally without requiring a round-trip to the central server, maintaining responsiveness similar to Git.
• On-demand hydration: The system allows developers to work with massive repositories without downloading every asset, fetching only the files required for the current task.
• Simplified setup: The CLI allows for immediate local testing using a demo flag, which spins up a server without requiring external authentication or complex configuration.

Context

Game development projects often struggle with Git because binary assets do not benefit from text-based diffing, leading to bloated repositories and slow performance. While Git LFS attempts to mitigate this, it remains a workaround for a system designed for code. Perforce is the industry standard for large-scale game development but is expensive and complex to maintain. Lore attempts to bridge the gap by providing the central control of Perforce with the local-speed benefits of Git, though it is currently pre-1.0 and lacks Git interoperability or a native GUI.