Decoupling RL Rollout from Training Clusters via Sparse Weight Deltas

The Breakthrough

Training clusters are bottlenecked by the need to keep rollout workers in the same high-bandwidth, RDMA-connected cluster as the trainer. By identifying that 99% of rollout-visible weights remain bit-identical between training steps due to the interaction between Adam optimizer steps and low-precision rounding, the system can sync weights using sparse, lossless deltas rather than full model checkpoints.

What Actually Worked

• Exploiting the 'Push vs. Floor' dynamic: Adam updates (the 'push') are typically on the order of the learning rate, while the BF16 rounding boundary (the 'floor') is roughly theta/256. Because the update magnitude is often significantly smaller than the rounding boundary, the served weights remain unchanged in the rollout engine's view.
• Lossless Delta Encoding: Instead of shipping full FP32 checkpoints, the system computes a bit-level diff between version t and t-1. This patch includes the index, replacement bits, and metadata, allowing the rollout engine to reconstruct the exact same version as if it were in the trainer cluster.
• Version-Aware Sidecars: Rollout engines are augmented with a sidecar that manages versioning. If the engine is behind, the sidecar applies the missing transition patches; if the engine is up to date, it proxies requests directly.
• Asynchronous Weight Publishing: The trainer publishes immutable weight versions to a shared bulletin board. Rollout engines pull these versions independently, allowing them to reside in different regions or cloud providers without requiring a global all-reduce or tight coupling to the trainer's fabric.

Before / After

• Checkpoint Transfer Size: Reduced from ~500GB (full checkpoint) to ~500MB (sparse delta) per update.
• Weight Stability: Approximately 99% of weights remain bit-identical per training step across various model families.

Context

Standard reinforcement learning (RL) loops require the trainer and rollout workers to share a tightly coupled, high-bandwidth cluster to handle weight synchronization. This forces the entire system to scale based on the most restrictive, hard-to-acquire compute resource. By treating the rollout fleet as a distributed set of serving jobs that consume sparse weight patches, the system enables the use of elastic, global compute capacity for trajectory generation.

Content References

{"type": "tool", "title": "Stitch", "url": "https://github.com/moto-ai/stitch", "context": "mentioned"}