Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing

Abstract: Modern LLM serving systems confront inefficient GPU utilization due to the fundamental mismatch between compute-intensive prefill and memory-bound decode phases. While current practices attempt to address this by organizing these phases into hybrid batches, such solutions create an inefficient tradeoff that sacrifices either throughput or latency, leaving substantial GPU resources underutilized. We identify two key root causes: 1) the prefill phase suffers from suboptimal compute utilization due to wave quantization and attention bottlenecks. 2) hybrid batches disproportionately prioritize latency over throughput, resulting in wasted compute and memory bandwidth. To mitigate the issues, we present Bullet, a novel spatial-temporal orchestration system that eliminates these inefficiencies through precise phase coordination. Bullet enables concurrent execution of prefill and decode phases, while dynamically provisioning GPU resources using real-time performance modeling. By integrating SLO-aware scheduling and adaptive resource allocation, Bullet maximizes utilization without compromising latency targets. Experimental evaluations on real-world workloads demonstrate that Bullet delivers 1.26x average throughput gains (up to 1.55x) over state-of-the-arts, while consistently meeting latency constraints.

• The paper introduces Bullet, a novel system that dynamically orchestrates spatial-temporal GPU resource sharing to boost LLM serving throughput by an average of 1.26× without sacrificing latency.
• It employs SLO-aware scheduling and adaptive resource allocation to concurrently manage compute-intensive prefill and memory-bound decode tasks, optimizing GPU utilization.
• Experimental results reveal up to 1.55× throughput gains over current methods, offering a cost-effective solution for improving large-scale model serving efficiency.

Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing

Introduction

The paper "Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing" (2504.19516), introduces a novel technique to address inefficiencies in GPU utilization in LLM serving systems. It identifies inefficiencies in current practices due to the separation of compute-intensive prefill and memory-bound decode phases, resulting in underutilization of GPU resources. The authors propose an innovative system named 'Bullet' to enhance the efficiency of GPU utilization by leveraging dynamic spatial-temporal orchestration.

Methodology

Challenges and Root Causes

The paper elucidates the challenges in LLM serving systems, namely the inefficient trade-off between throughput and latency when combining compute-intensive prefill and memory-bound decode phases into hybrid batches. The root causes of inefficiencies are pinpointed as suboptimal compute utilization due to wave quantization and attention bottlenecks during the prefill phase. Additionally, the prioritization of latency over throughput in hybrid batches leads to wasted compute and memory bandwidth.

Proposed Solution: Bullet

The authors present Bullet, a spatial-temporal orchestration system designed to improve GPU utilization. Bullet enables concurrent execution of prefill and decode phases by dynamically provisioning GPU resources based on real-time performance modeling. This system incorporates SLO-aware scheduling and adaptive resource determination to maximize GPU utilization while adhering to latency targets. The spatial-temporal orchestration ensures precise phase coordination, eliminating inefficiencies and optimizing resource usage.

Experimental Evaluation

The evaluation of Bullet demonstrates significant improvements in throughput and resource utilization across real-world workloads. Experimental results reveal that Bullet achieves an average throughput gain of 1.26 times, with peaks up to 1.55 times compared to state-of-the-art methods. Importantly, these throughput gains are accomplished without compromising latency constraints, showcasing the system's efficacy in balancing throughput and latency.

Implications and Future Directions

The development and implementation of Bullet have practical implications for enhancing the performance and efficiency of LLM-serving systems. By optimizing the utilization of GPU resources, the system supports more efficient and cost-effective deployments in environments requiring high-throughput and low-latency. From a theoretical perspective, Bullet contributes to the understanding of dynamic resource allocation strategies, promoting advancements in spatial-temporal resource orchestration.

Future developments might focus on extending the proposed methodology to accommodate additional phases in LLM serving or exploring its applicability in diverse hardware architectures. Furthermore, the integration of more sophisticated machine learning models for real-time performance prediction could enhance Bullet's adaptability and efficiency in varying operational environments.

Conclusion

The paper provides compelling evidence for the effectiveness of spatial-temporal GPU resource sharing in LLM serving systems through the adoption of Bullet. By addressing the inefficiencies inherent in current serving strategies and explicitly optimizing resource allocation, the authors have demonstrated a significant leap in the throughput and utilization of GPU resources. The insights and methodologies presented set a critical foundation for future research and development in efficient large-scale model serving paradigms.