TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference

Abstract: As the model size continuously increases, pipeline parallelism shows great promise in throughput-oriented LLM inference due to its low demand on communications. However, imbalanced pipeline workloads and complex data dependencies in the prefill and decode phases result in massive pipeline bubbles and further severe performance reduction. To better exploit the pipeline parallelism for high-throughput LLM inference, we propose TD-Pipe, with the key idea lies in the temporally-disaggregated pipeline parallelism architecture. Specifically, this architecture disaggregates the prefill and decode phases in the temporal dimension, so as to eliminate pipeline bubbles caused by the phase switching. TD-Pipe identifies potential issues of exploiting the novel architecture and provides solutions. First, a hierarchy-controller structure is used to better coordinate devices in pipeline parallelism by decoupling the scheduling from execution. Second, the AI-based greedy prefill approach aggressively performs more prefills by predicting the output length and simulating the memory usage. Third, the inter-batch work stealing approach dynamically balances decode phase workloads between different batches to reduce bubbles. Forth, the spatial-temporal intensity comparison approach determines the optimal switch from decode to prefill by comparing the performance drop from reduced computational intensity with that from phase switching bubbles. Extensive experiments show that TD-Pipe effectively increases the throughput of LLM inference by up to 1.91x over the existing tensor parallel approach and 2.73x over the existing pipeline parallel approach on GPU nodes with only PCIe interconnection.

• The paper presents a novel architecture that disaggregates prefill and decode phases to mitigate pipeline bubbles in LLM inference.
• It employs a hierarchy-controller, AI-based greedy prefill, and inter-batch work stealing to dynamically optimize GPU resource usage.
• Experimental results demonstrate up to 1.91× and 2.73× throughput improvements over tensor and existing pipeline methods on bandwidth-constrained hardware.

TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference

Introduction

The rapid growth in the size and complexity of LLMs poses challenges for efficient model inference. Pipeline parallelism offers promising solutions due to its low communication requirements compared to tensor parallelism. However, existing pipeline parallelism techniques suffer from significant performance degradation due to pipeline bubbles caused by imbalanced workloads and complex data dependencies during the prefill and decode phases. "TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference" (2506.10470) addresses these challenges through a novel architecture that separates prefill and decode phases temporally, minimizing phase-switching overhead.

Figure 1: Bubbles in LLM pipeline parallel inference.

Architecture Overview

TD-Pipe's architecture aims to disaggregate pipeline execution temporally to eliminate bubbles caused by phase-switching. This approach involves several key innovations:

1. Hierarchy-Controller Structure: TD-Pipe incorporates a hierarchy-controller structure to asynchronously coordinate devices during pipeline execution, decoupling scheduling from execution.
2. AI-based Greedy Prefill: Predicts output length using AI models to perform aggressive prefills that maximize resource utilization without exceeding memory constraints.
3. Inter-Batch Work Stealing: Dynamically balances decode phase workloads across batches to minimize pipeline bubbles.
4. Spatial-Temporal Intensity Comparison: Determines optimal switch timing by comparing the computational drop from reduced decode spatial intensity with the temporal bubbles incurred by phase-switching.

   Figure 2: GPU Utilization in Pipeline Parallelism.

Experimental Results

TD-Pipe demonstrated a substantial increase in throughput during experiments on GPU nodes with limited interconnection bandwidth. Specifically, it achieved up to a 1.91$\times$ improvement over tensor parallel approaches and up to 2.73$\times$ over existing pipeline parallel approaches. The architecture is particularly advantageous in setups where NVLINK cables are unavailable, highlighting its broader applicability on commodity hardware nodes.

Implications and Future Work

This new approach to pipeline parallelism yields significant implications for high-throughput LLM inference, particularly in scenarios with relaxed latency constraints. TD-Pipe's hierarchical architecture and novel techniques open pathways for optimizing batch processing without sacrificing throughput, while efficiently managing resource allocation across multiple GPUs. Future work may focus on extending TD-Pipe's methodologies to other model types or exploring the adaptability of the temporally-disaggregated approach under varied network infrastructures or model configurations.

Figure 3: Overall performance.

Conclusion

TD-Pipe establishes a pioneering framework for utilizing pipeline parallelism in high-throughput LLM inference scenarios. By disaggregating the prefill and decode phases, TD-Pipe significantly reduces pipeline bubbles, leading to better resource utilization and throughput on bandwidth-constrained architectures. This research enhances the understanding of pipeline dynamics in model inference and offers a scalable approach for the deployment of increasingly large and capable LLMs.