Scaling Krea 2: Infrastructure for Training and Serving

Hardware Reliability and Metrics

Training at scale requires moving beyond standard GPU utilization metrics, which often report 100% usage even when the hardware is underperforming. Instead, tracking tensor core utilization provides a more accurate proxy for actual computational throughput. Because silent failures in cross-node communication are common, building custom collection for InfiniBand and NVLink metrics is essential for identifying bottlenecks and hardware errors. Furthermore, any GPU exceeding 78 degrees Celsius should be immediately removed from the cluster, as thermal throttling on a single card can destabilize the entire training run. When failures occur, the most effective strategy is often to let the job crash and restart, as the same nodes frequently succeed on subsequent attempts without manual intervention.

Cluster Orchestration and Resource Management

To maximize GPU utilization, Krea.ai uses a unified cluster for both training and production inference. Training jobs are assigned higher priority through gang scheduling, which allows them to preempt production workloads. To prevent production downtime during preemption, the system uses a virtual kubelet to migrate inference tasks to external providers. This migration happens gradually rather than all at once to ensure service continuity. When GPUs become available again, a descheduler slowly migrates inference workloads back to the primary cluster to avoid the sudden service disruption that would occur with a hard 'no-execute' taint. This self-healing infrastructure allows researchers to launch training jobs without needing to manage GPU availability or production impact.