Topology-Aware Data Movement for Disaggregated GPU Inference
Disaggregated LLM inference creates a datacenter networking problem that no existing system solves correctly. The authors design a topology-aware transfer orchestrator that discovers interconnect hierarchy at startup and selects optimal transport per transfer.
Intelligence analysis by Llama
The authors propose a topology-aware transfer orchestrator to optimize data movement for disaggregated GPU inference. This approach reduces transfer latency by up to 18x compared to uniform RDMA.
Imagine you have a big computer system with many parts that need to talk to each other. The authors of this paper found a way to make these parts talk to each other more efficiently, especially when they're far apart. This is important because it can help make big computer systems work faster and more smoothly.
Analysis
A Novel Approach to Data Movement for Disaggregated GPU Inference
The authors of this paper propose a novel approach to data movement for disaggregated GPU inference. Disaggregated LLM inference creates a datacenter networking problem that no existing system solves correctly. When prefill and decode run on separate GPU pools, the KV cache must be transferred between them. For a 70B model, this is 2.6 GB per request, exceeding 100 GB/s aggregate at production scale. Yet, existing systems like DistServe, Splitwise, and Mooncake all use uniform RDMA, ignoring the fact that bandwidth between two GPUs varies by 72x depending on their physical relationship.
Mechanisms for Topology-Aware Transfer
The authors design a topology-aware transfer orchestrator that discovers interconnect hierarchy at startup and selects optimal transport per transfer. Three mechanisms work together: (1) pipelined layer-by-layer transfer that overlaps transmission with ongoing prefill, hiding 60 to 85 percent of latency behind computation; (2) NVLink domain-aware placement for Mixture-of-Experts models that co-optimizes expert dispatch with KV cache locality; and (3) CXL 3.0 memory expanders as a shared overflow tier providing 6x capacity at 86x lower latency than NVMe.
Evaluation and Results
The authors present analytical bandwidth models, component implementations, and projected analysis across three architectures showing 3 to 18x transfer latency reduction over uniform RDMA. While full evaluation requires multi-node clusters with heterogeneous interconnects and CXL 3.0 hardware that is beyond academic resources and not yet available in GPU clouds, the authors' approach has significant implications for the development of efficient datacenter networking systems.
Key points
- The authors propose a topology-aware transfer orchestrator to optimize data movement for disaggregated GPU inference.
- The approach reduces transfer latency by up to 18x compared to uniform RDMA.
- The authors design three mechanisms to work together: pipelined layer-by-layer transfer, NVLink domain-aware placement, and CXL 3.0 memory expanders.
- The approach has significant implications for the development of efficient datacenter networking systems.
If this approach is widely adopted, it could lead to significant improvements in the efficiency and scalability of datacenter networking systems, enabling faster and more accurate machine learning inference tasks.
However, the widespread adoption of this approach may be hindered by the need for significant hardware upgrades, including the availability of CXL 3.0 memory expanders and heterogeneous interconnects.
