Homa: Reducing AI Cluster Tail Latency via Receiver-Driven Transport

The Breakthrough

John Ousterhout introduces Homa, a transport protocol designed to replace TCP and RDMA in data centers by treating network traffic as discrete messages rather than opaque byte streams, enabling the use of Shortest Remaining Processing Time (SRPT) scheduling to prioritize small coordination messages.

What Actually Worked

• Message-Based Architecture: Homa treats data as Remote Procedure Calls (RPCs) consisting of request and response messages, allowing the protocol to know message lengths and boundaries at the transport layer.
• Receiver-Driven Congestion Control: Instead of relying on senders to detect congestion via packet loss or ECN marks, the receiver controls the flow by issuing "grant" packets that pace the transmission of scheduled data, preventing queue buildup at the switch.
• Priority Queue Utilization: Homa maps messages to different hardware queues in modern data center switches, allowing short, latency-sensitive messages to bypass long-running transfers that would otherwise cause head-of-line blocking.
• Unscheduled Packet Transmission: Senders transmit the first few packets of a message immediately without waiting for a grant, ensuring that small messages complete with minimal round-trip time.

Before / After

In a benchmark comparing Homa to TCP across message lengths from 50 bytes to 1 megabyte:

• Short Message P99 Latency: Homa achieved <100 microseconds, while TCP exceeded 1 millisecond (a 13x improvement).
• Long Message Performance: Homa performed nearly 2x faster than TCP, despite prioritizing shorter messages, due to more efficient run-to-completion scheduling.

Context

Modern AI workloads, particularly inference and agentic applications, rely on frequent, small synchronization messages that are highly sensitive to tail latency. Traditional protocols like TCP and RoCE (RDMA over Converged Ethernet) struggle with these workloads because they are sender-driven and treat data as a continuous stream, leading to congestion-induced queue buildup (incast) at the top-of-rack switch. When these queues fill, short messages get trapped behind large data transfers, stalling GPUs and wasting compute resources.

Content References

• tool: Homa, John Ousterhout, Stanford, https://github.com/PlatformLab/Homa, recommended