TurboServe: Serving Streaming Video Generation Efficiently and Economically

Abstract: Streaming video generation is emerging as a new serving workload in which users interact with long-lived sessions that generate video progressively, chunk by chunk. Unlike offline video generation or typical LLM serving, streaming video generation must preserve session state across active and idle periods, repeatedly schedule ongoing sessions, and deliver each chunk under a tight latency target. This creates two key serving challenges in multi-user, multi-GPU environments: session duration heterogeneity, where long-running sessions make placement decisions suboptimal over time, and temporal user-demand heterogeneity, where the number of active sessions fluctuates sharply across bursts and idle periods. We present TurboServe, the first serving system designed specifically for streaming video generation workloads. TurboServe formulates serving as an online scheduling problem that jointly coordinates session placement and GPU provisioning. Its closed-loop scheduling algorithm combines a migration-aware placement controller, which rebalances sessions across GPUs to reduce the maximum per-chunk latency, with a load-driven autoscaling controller, which adapts the GPU budget to workload variation for improved cost efficiency. To support these decisions at runtime, TurboServe implements coalesced chunk processing for batching concurrent active sessions on the same GPU, GPU-CPU offloading for session suspension and resumption, and NCCL-based GPU-GPU migration for online rebalancing. We evaluate TurboServe on real-world production traces from Shengshu Technology across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs. Compared with baseline serving configurations, TurboServe reduces worst-case per-chunk latency by 37.5% and total GPU operating cost by 37.2% on average. Our code is publicly available at https://github.com/shengshu-ai/TurboServe.

• The paper introduces TurboServe, a system that reduces per‐chunk latency by 37.5% and GPU operating costs by 37.2%, demonstrating significant efficiency gains.
• It employs a closed‐loop scheduling framework combining migration‐aware session placement with adaptive GPU autoscaling to manage dynamic, bursty workloads.
• Empirical evaluations on real-world traces confirm near‐oracle performance with minimal overhead, validating its impact on efficiency and cost reduction.

TurboServe: Efficient and Economic Serving for Streaming Video Generation

Introduction

TurboServe introduces a specialized system architecture for serving streaming video generation models in dynamic, multi-user, multi-GPU environments, which fundamentally depart from prior, stateless inference paradigms. Streaming video generation, typified by models such as Sora and LongLive, produces output incrementally (chunk by chunk), requiring persistent user sessions and context retention across active and idle periods. Two critical heterogeneities are addressed: session duration (with sessions varying from brief clips to tens of minutes) and temporal demand fluctuation (highly bursty, unpredictable patterns). TurboServe tackles these with tightly integrated scheduling of session placement and elastic GPU provisioning, coordinated via closed-loop feedback to balance latency and cost.

Figure 1: Illustrates differences between stateless generation (top) and stateful, session-based streaming generation (bottom) in multi-user scenarios.

Figure 2: Streaming workload trace from Shengshu Technology, showing session duration distribution (left) and dynamic active session count across a 30-minute period (right).

System Design and Architecture

TurboServe formulates streaming serving as an online scheduling problem, with two interdependent controls: session placement across GPUs and real-time GPU autoscaling. The placement controller implements migration-aware min-max rebalancing, moving sessions away from overloaded GPUs to minimize bottleneck chunk latency. The autoscaling controller uses load feedback to provision or deprovision GPUs according to demand, carefully tuned with a volatility-to-parameter mapping to preserve latency SLOs under variable burstiness.

Runtime optimizations include:

• Coalesced chunk processing: Efficient batching of concurrent session chunk generation, increasing GPU throughput and utilization.
• Session state management: Fast GPU-CPU offloading for idle-session suspension/resumption, decoupling session lifetime from GPU residency.
• NCCL-based GPU migration: Low-overhead, RDMA-enabled session state transfer for rapid load redistribution.

  Figure 3: TurboServe system overview, showing closed-loop scheduler coordinating session placement, GPU provisioning, and session state management.

Scheduling Framework

TurboServe's closed-loop algorithm employs event-driven scheduling, jointly solving for session placement and GPU budget at every system event (arrival, departure, activity transition). Session placement is solved via local search for min-max latency, with migration cost modeled by an $\alpha$-$\beta$ transfer overhead. Autoscaling uses a proportional, hysteresis-based policy to set GPU budget ($M_{tar}$), dynamically tracking workload volatility and adapting target utilization ($\rho^*$) to maintain latency constraints while maximizing cost-efficiency.

Figure 4: Illustrative examples: scale-out (top), followed by session rebalancing; rebalancing (bottom), followed by scale-in and GPU removal.

Empirical Evaluation

TurboServe is evaluated on Shengshu Technology real-world traces, utilizing clusters up to 64 B300 GPUs and multiple video generation models. Two metrics benchmark performance: worst-case per-chunk latency (a hard user-facing constraint), and GPU operating cost.

• Latency Performance: TurboServe reduces maximum chunk-generation latency by 37.5% on average compared to baseline static or load/memory-aware serving configurations.
• Cost Efficiency: It achieves an average 37.2% GPU operating cost reduction, simultaneously satisfying latency constraints.

These gains derive from the joint effect of session migration and autoscaling, confirmed via ablation—removing either severely degrades efficiency (migration: +15% cost; autoscaling: +43% cost).

Figure 5: End-to-end experimental results: TurboServe outperforms baselines on both latency and cost across models, traces, and clusters.

Figure 6: Ablation study: session migration and autoscaling both critical for optimal cost efficiency under fixed latency.

Scheduling and Overhead Analysis

TurboServe's migration-aware min-max rebalancing algorithm achieves near-oracle placement quality (≤ 6.5% latency gap), with negligible runtime overhead (typically <2% of chunk generation time). Autoscaling policy incurs only 6.1% average cost gap compared to offline oracle, validating its practical effectiveness.

Migration overhead remains consistently low (2-3% of per-chunk latency), enabling aggressive rebalancing without substantial interruption. Detailed runtime breakdowns show continuous, responsive autoscaling and session migration, reinforcing the necessity for both coarse-grained resource adaptation and fine-grained session management.

Figure 7: Characterization of GPU load imbalance, resource underutilization during low demand, and overutilization during demand bursts.

Figure 8: Case study quantifying the impact of session management approaches (A1–A3) on latency and cost.

Implications and Future Directions

TurboServe sets a new standard for serving architectures in the streaming generation regime, integrating tightly-coupled controls that are robust to both session and temporal workload heterogeneity. The formalization and closed-loop scheduling approach are highly generalizable to emerging generative modalities—streaming audio, multimodal agent sessions, or interactive long-form content—where persistent state, low-latency guarantees, and elastic resource use are required. As generative models push toward longer context, more interactive sessions, and real-time outputs, similar session-centric coordination and adaptive provisioning will become essential system primitives.

Potential research directions include extension to multi-cluster/federated scenarios, advanced migration policies considering session affinity or resource heterogeneity, and integration with serverless or decentralized GPU pools for further elasticity.

Conclusion

TurboServe advances system-level support for streaming video generation, achieving strong numerical improvements in latency and cost while leveraging migration-aware session placement and adaptive GPU autoscaling in a unified closed-loop system. Its design addresses both theoretical control coupling and practical runtime efficiency, establishing a template for stateful, interactive, and resource-constrained generative inference serving at scale.