Fixed Prompt Schedule in ML Inference

• Fixed prompt scheduling is a method that pre-defines resource allocation and routing in ML inference, ensuring predictability in service-level objectives.
• It leverages offline combinatorial optimization, employing integer and linear programming to assign GPUs, model approximations, and cache placements.
• Empirical evaluations demonstrate notable improvements in throughput and latency, with significant reductions in SLO violations compared to dynamic scheduling.

A fixed prompt schedule is a resource allocation and routing methodology in prompt-based machine learning inference systems, where the mapping from prompts or prompt classes to serving resources (e.g., GPUs, model approximations, cache replicas) is determined in advance—either offline via optimization or as a semi-static, periodically updated plan. Fixed schedules stand in contrast to fully dynamic, online scheduling policies. This technique is principally motivated by the need to deliver consistent service-level objectives (SLOs) under constrained compute and memory resources, where predictability and load balancing must be jointly optimized with task-specific concerns such as model approximation for generative tasks or key-value (KV) cache placement for LLMs. Fixed prompt scheduling underpins recent advances in both text-to-image and text-to-text serving pipelines, particularly in contexts where workload characteristics (prompt distribution, sharing frequency, or quality/latency curves) can be profiled or predicted in advance (Agarwal et al., 29 Jan 2025, Srivatsa et al., 2024).

1. Conceptual Framework and System Context

Fixed prompt scheduling emerges as a response to pronounced heterogeneity in model inference costs and quality impacts driven by prompt characteristics. In diffusion-based text-to-image generation systems, prompt sensitivity to model approximation levels motivates the use of a skip parameter $K$, which controls the number of denoising steps skipped to trade off latency for output fidelity. The fixed schedule determines the number of GPU-hosted replicas assigned to each $K$ level (model approximation), and what portion of the workload each serves, under resource budget and throughput constraints. All model weights are preloaded to avoid run-time model swapping overhead (Agarwal et al., 29 Jan 2025).

In distributed LLM serving, such as Preble (Srivatsa et al., 2024), the fixed schedule may describe the assignment of common prompt prefixes (KV cache state) to particular GPUs ahead of time, based on observed prompt frequency, to maximize reuse and minimize redundant prefill computation. This approach allows for pre-positioning frequently shared prefixes and treating them as “frozen” in the serving system’s routing logic, reducing the complexity of dynamic scheduling.

2. Formal Optimization Formulations

Fixed prompt schedules are typically constructed using offline combinatorial optimization. For text-to-image systems, the macro-placement and fraction assignment is handled by an integer program adapted from the Proteus framework, while the micro-level (per-query) redirection is done using a linear program (LP):

$$\min_{x_{i,j}\ge0}\quad \sum_{i=1}^L\sum_{j=1}^L x_{i,j}\,D(K_j,K_i)$$

subject to

$$\sum_{j=1}^L x_{i,j} = H(K_i)\quad\forall i,\quad \sum_{i=1}^L x_{i,j} = F(K_j)\quad\forall j$$

Here, $H(K_i)$ denotes the fraction of incoming queries with optimal skip $K_i$, $F(K_j)$ is the planned serving fraction at approximation $K_j$, and $D(K_j,K_i)$ quantifies the quality degradation for prompt type $i$ served at level $j$ (Agarwal et al., 29 Jan 2025).

For prefix caching in LLMs, a mixed-integer linear program (ILP) can be formulated with the goal of maximizing prefix reuse benefit minus a convex penalty on load imbalance:

$$\max_{x,y,r,L}\; \sum_{u,p,g} \Delta_{\mathrm{reuse}(p)} r_{u,p,g} - \lambda \sum_{g} \ell(L_g)$$

with constraints on prefix–request mapping, reuse implying placement, load accounting, and per-GPU cache memory limits (Srivatsa et al., 2024). This formalization makes clear the multi-resource nature of prompt scheduling—balancing quality, throughput, and compute/memory occupancy.

3. Construction and Implementation

In practical systems, fixed prompt schedules are built via trace analysis—a held-out sample of historical queries is analyzed to estimate (a) per-prompt optimal serving parameters (skip $K$ or prefix assignment), (b) prompt popularity or sharing frequency, and (c) serving resource constraints (model memory footprint, throughput per approximation, or cache capacity).

The optimization output specifies:

• Allocation of serving replicas per model variant or cache “bucket.”
• Target routing fractions $F(K)$ or explicit placement tables $\{p_i \to g_j\}$.
• Redirection plans $P(K'|K)$ for handling mismatches between optimal and available assignment (as in the text-to-image system’s LP).
• In the LLM case, a static table may be created mapping top-$k$ prefixes to dedicated GPUs via round robin, after which dynamic scheduling only handles the remainder set (Srivatsa et al., 2024).

This process can be done periodically (e.g., every few minutes) to adapt to shifting workload statistics but remains substantially less frequent and computationally demanding than fully dynamic, per-query optimization.

4. Scheduling Algorithmic Patterns

Fixed prompt scheduling is enacted in production via two primary mechanisms:

1. Offline periodical re-optimization (macro-time): The system periodically solves an integer program to update replica allocations and serving fractions, using recent query logs to track changes in prompt workload characteristics. Output includes the assignment of model variants or cache placements to specific GPUs and planned routing shares.
2. Online micro-level mapping (micro-time): For each incoming query, the system chooses the serving parameter (e.g., skip $K$ or cached prefix host) either by direct table lookup from the fixed plan or by sampling with redirection probability (if precise matching is under-constrained), as specified by solution of the micro-level LP or prefix assignment logic. Workload imbalance is mitigated using route-and-batch heuristics based on current queue length or per-bucket batching (Agarwal et al., 29 Jan 2025, Srivatsa et al., 2024).

5. Empirical Evaluation and Comparative Impact

Empirical studies of fixed prompt scheduling methodologies have demonstrated substantial benefits over naive, profile-agnostic or dynamic baselines. In text-to-image pipelines, prompt-aware fixed scheduling delivers up to 10 percentage points improvement in CLIP-scored image quality, 40% higher throughput, and a tenfold reduction in SLO violation rate (e.g., requests exceeding 3s latency) compared to static accuracy scaling (Agarwal et al., 29 Jan 2025).

Strategy	Quality (%)	SLO Violation (%)	Relative Throughput
Clipper-HA	~100	25	baseline
Clipper-HT	85	5	+30%
NIRVANA	94	20	–
Proteus	<90	25–30	stable
Prompt-aware	>90	<5	+40%

For distributed LLM serving, incorporation of fixed prefix schedules enables Preble to achieve 1.5×–14.5× reduction in average latency and 2×–10× drop in p99 tail latency over prior round-robin or purely online schemes. System performance correlates with the degree of prompt sharing and the prompt-to-decode workload ratio (Srivatsa et al., 2024).

6. Limitations and Practical Considerations

While fixed prompt schedules maximize predictability and enable global resource optimization, they require reasonably stationary workload patterns or accurate forecasting; rapid shifts in prompt distribution may induce suboptimality until the next schedule update. For LLM serving, prefix popularity skew is a prerequisite; otherwise, fixed assignments yield little benefit. Feature ablation studies suggest that omitting semantic features (e.g., CLIP embedding similarity) in the prompt classifier degrades performance by approximately 4%, highlighting the sensitivity of fixed schedules to accurate profiling (Agarwal et al., 29 Jan 2025). Using too coarse-grained approximation (e.g., only two skip levels) also reduces output quality compared to finer-grained schedulability.

A plausible implication is that hybrid approaches—combining periodic fixed scheduling with lightweight online adjustment—offer the best tradeoff for nonstationary or bursty applications.

7. Related Architectures and Research Directions

Fixed prompt schedules are foundational in contemporary scalable inference systems, notably those featured in the Prompt-Aware Scheduling system for text-to-image generation (Agarwal et al., 29 Jan 2025) and Preble for efficient distributed LLM serving (Srivatsa et al., 2024). Both leverage trace-driven, profile-informed static placement to optimize for latency, throughput, and output fidelity under real-world SLOs and hardware budgets.

Ongoing research explores automating schedule periodicity, integrating richer prompt feature spaces, and combining fixed scheduling with decentralized, demand-aware adaptation for highly dynamic or heterogeneous cloud environments. The interplay of prompt-centric approximation, cache locality, and hardware-efficient scheduling continues to be an active area, intersecting systems, ML, and optimization communities.