Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving

Abstract: Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering. In practice, this creates a new asymmetry: under bursty, heavy-tailed workloads prefill nodes saturate while decode nodes have compute underutilized, and on a production-style A100 cluster with 2 prefill and 2 decode nodes (2P2D), we find that prefill execution accounts for only 2-23% of P95 Time-to-First-Token (TTFT). Queuing and inter-node GPU-GPU KV-cache transfer account for the rest. We present a proactive prefill-deflecting scheduler that lets decode nodes serve prefill phase of requests as chunked-prefill steps interleaved with their in-flight decode batches. For each queued request, we estimate the TTFT it would see on the prefill node, and on every decode node, search for the largest chunk schedule that keeps in-flight decodes within their Time-Between-Tokens (TBT) SLO and deflect when the decode path helps tail latency. Because the prefill phase of deflected requests runs in place on the decode node, the inter-node KV transfer is eliminated. Implemented on vLLM and evaluated on production-style traces with DeepSeek-V2-Lite, our approach reduces P95 TTFT by upto 81% and raises SLO attainment by upto 79% over state-of-the-art disaggregated schedulers, at sub-millisecond per-request routing cost.

• The paper presents Kairos, a proactive deflection strategy that reroutes prefill tasks to underutilized decode nodes, significantly reducing tail latency.
• It leverages lightweight, per-request analytical models to predict execution latency, achieving up to an 81% reduction in TTFT and a 79% higher SLO attainment than baseline systems.
• The approach integrates seamlessly with existing LLM serving stacks like vLLM, enabling fine-grained load balancing without requiring any modifications to the model.

Load-Aware Prefill Deflection in Disaggregated LLM Serving: The Kairos Approach

Introduction and Motivation

Disaggregated LLM serving architectures isolate prefill and decode phases onto separate GPU node pools for resource efficiency, but this separation introduces pronounced asymmetries under bursty, heavy-tailed production workloads. Empirical analysis demonstrates that prefill nodes saturate while decode nodes remain underutilized. Critically, in such systems, prefill queuing and inter-node KV-cache transfer dominate end-to-end user-perceived latency (P95 TTFT), with prefill execution itself accounting for only 2–23% of TTFT. This bottlenecking leads to unnecessarily long queuing delays and persistent underutilization of compute resources on the decode side (Figure 1).

Figure 1: Decode step latency increase from chunked prefill injection. Step latency remains flat ($\Delta T_{\mathrm{step}} \leq 30$ ms) for chunk sizes up to $512$ tokens across workloads and batch sizes.

This context motivates an interrogation of how, when, and whether prefill tasks can be intelligently rerouted (“deflected”) to decode nodes, exploiting their compute slack while respecting stringent SLOs for TTFT and per-token latency (TBT).

Kairos: Architecture and Scheduling Algorithm

Kairos introduces a proactive scheduling layer that opportunistically deflects prefill requests to decode nodes if doing so is predicted to lower tail TTFT without violating TBT SLOs for ongoing decode tasks. The system estimates, per request, the expected TTFT for both serving paths (predefined prefill vs. deflected decode) by leveraging analytical models of step latency. The deflection decision involves:

1. Estimating prefill node TTFT considering queue state and execution time.
2. Enumerating TBT-safe chunk schedules for each decode node; each schedule adaptively selects chunk sizes per step to maximize throughput while avoiding TBT violations.
3. Deflecting only if the projected TTFT on the decode route is below an operator-set multiplicative bound ($\alpha$) compared to the prefill route and also respects global SLOs.

A critical enabler is the realization that injecting small, state-dependent prefill chunks into decode batches causes only minimal step latency inflation, enabling TBT adherence even under high utilization (Figure 1).

Figure 2: Kairos architecture.

The Kairos dispatcher applies this decision algorithm for each request, using per-node state reported at a fixed interval. The per-request computation is lightweight ($<$1 ms), and no model changes are required, supporting practical deployment atop vLLM and similar platforms.

Analytical Step-Latency Modeling

Kairos achieves high-accuracy prediction of mixed-batch execution latency ($<$10% MAPE) using regression-based models trained during a brief deployment-time profiling phase. These models take as input the batch size, current KV occupancy, and candidate chunk size. Separate predictors are maintained for prefill, decode, and mixed batches. These predictors enable the construction of TBT-safe chunk schedules on-the-fly, crucial for maximizing decode node utilization without SLO violation.

The adaptive chunking mechanism addresses the finding that latency inflation is minimal for initial (small) chunks and grows for later ones as cached tokens accumulate. Thus, the chunk sequence is constructed greedily based on forecasted safe headroom.

Empirical Evaluation

Kairos is evaluated on production-derived traces, notably the “Bursty” workload with heavy-tailed prompt and completion length distributions. The evaluation compares Kairos to baseline disaggregated schedulers (DistServe-style) and TaiChi (which supports both aggregation and disaggregation).

Key results:

• TTFT Reduction: Kairos lowers P95 TTFT by up to 81% on realistic bursts, sustaining SLO-compliant operation at higher cluster RPS than other systems (Figure 3).
• SLO Attainment: Kairos achieves up to 79% higher SLO attainment rate compared to state-of-the-art baselines by dynamically exploiting decode-side slack.
• Throughput: Kairos matches or exceeds token throughput of other approaches, especially at high load, while strictly controlling SLO violations (Figure 4).
• TBT Safety: By construction, Kairos maintains zero TBT violations across tested workloads, demonstrating the correctness of analytical scheduling (Figure 3).

  Figure 3: Kairos maintains P95 TTFT and P95 TBT within SLOs at higher RPS compared to baselines, showing state-aware deflection effectiveness.

  

  Figure 4: Kairos achieves comparable or superior throughput versus baselines due to its opportunistic exploitation of decode compute slack.

The empirical microbenchmarks verify that naive static deflection policies are substantially less effective, either under-utilizing decode slack or violating SLOs. Only per-request, state-aware scheduling matches the Oracle's SLO compliance and TTFT minimization.

Design Implications and Future Directions

Kairos’s per-request adaptive scheduling fundamentally challenges the sufficiency of static deflection or coarse-grained resource rebalancing. Its analytical approach is both lightweight and effective, but its accuracy hinges on the validity of the step-latency models and timely synchronization of node state.

Practical implications:

• The elimination of inter-node KV transfer for deflected requests not only reduces TTFT, but also simplifies the path for multi-turn and prefix-cached workloads when properly integrated.
• The dispatcher-centric architecture may become a bottleneck at very high RPS, but sharding or distributing the dispatcher is feasible.
• Kairos integrates seamlessly atop established inference stacks such as vLLM, with no requirement for model changes or intrusive architectural modifications.

Limitations and future work:

• Analytical model errors, stale node state, and the inability to exploit prefix cache residency in deflection decisions may leave some efficiency untapped.
• More frequent state synchronization, cache-aware scheduling, and probabilistic TBT bounding could further optimize throughput/latency trade-offs.
• Incorporation of prefix-awareness, better multi-model/multi-instance scheduling, and fine-grained expert placement for MoE architectures remain open avenues.

Position in Broader Systems Literature

Kairos leverages and generalizes classical ideas—dynamic batching, analytical modeling, and opportunistic routing—within the new operational context of phase-disaggregated LLM inference. Unlike static-phase split systems (e.g., Splitwise, DistServe), or coarse instance ratio tuning (TaiChi), Kairos employs stateful, per-request logic using online estimators to make high-fidelity decisions. It is thus situated as a flexible, extensible framework for load balancing and tail latency optimization in next-generation LLM serving deployments.

Conclusion

Kairos demonstrates that combining proactive, per-request scheduling with lightweight analytical modeling enables substantial improvements in both tail latency and SLO attainment for disaggregated LLM serving. Its dynamic exploitation of decode-side slack, robust TBT safety, and practical overheads mark a significant advancement in the state of LLM inference system design. Integration with richer workload characteristics, prefix/residency awareness, and distributed dispatcher architectures are promising directions to further close the gap to theoretical serving upper bounds.

Reference: "Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving" (2607.02043)