Extended Sparse Server (ESS) Overview

• ESS is an offload-centric cache management architecture that decouples GPU memory constraints from long-context LLM inference.
• It uses a layered design with an Offload–Prefetch Engine, LRU-Based Swap Engine, and Overlap Scheduler to manage cached latent data.
• Simulation results indicate throughput improvements from 69.4% to 123% by effectively increasing batch-size capacity via offloading.

The Extended Sparse Server (ESS) is an offload-centric cache management architecture designed to address the Decode-stage memory bottlenecks in long-context LLM inference, specifically for the DeepSeek-V3.2-Exp model employing sparse attention mechanisms. ESS introduces a layered set of algorithms and cache management strategies that selectively offload the linearly expanding latent-cache to CPU (host) memory, thereby decoupling batch-size scaling from the constraints of GPU memory during Decode-stage inference. By freeing GPU memory buffers crucial for batch-size expansion, ESS enables significant throughput gains in large-context serving scenarios while maintaining model accuracy (Chen et al., 11 Dec 2025).

1. System Architecture

ESS integrates into the Decode stage of the partially-disaggregated (PD) DeepSeek-V3.2-Exp inference pipeline. It is comprised of three principal modules: the Offload–Prefetch Engine, the LRU-Based Swap Engine, and the Overlap Scheduler.

• Offload–Prefetch Engine: Orchestrates fine-grained, entry-wise transfers of latent-cache between CPU and GPU memories using Unified Virtual Addressing (UVA) and a custom operator, FlashTrans. FlashTrans achieves high bandwidth for transfers of small, scattered blocks—crucial for real-time latency requirements.
• LRU-Based Swap Engine: Maintains a fixed-size "Sparse Memory Pool" (Editor's term) on the GPU. It retains the hottest latent-cache entries according to a standard Least Recently Used (LRU) policy, evicting the coldest when required.
• Overlap Scheduler: Dynamically switches, on a per-layer basis, between Dual-Attention (DA) Overlap and Dual-Batch-Attention (DBA) Overlap, maximizing the overlap between host-device data movement and GPU computation.

At each decoding step $t$ and for Transformer layer $l$, the Indexer module emits a Top-K latent-cache index set $K^l_t$ (default $K = 2048$). ESS intercepts this set and ensures on-demand prefetching of required entries into the GPU's Sparse Memory Pool $I^l$, while newly computed latent vectors are written back (device-to-host, D2H) after all layers complete. By offloading approximately $(1-\alpha)$ fraction of the cache to host memory, ESS releases GPU memory exactly where it is needed to enhance batch-size.

2. Cache Management and Swap Policies

ESS employs both proactive and reactive cache management algorithms:

• LRU-Warmup Preheating: Prior to Decode, the LRU structure is pre-populated by replaying Top-K index streams from the last 32 Prefill windows for every layer, reducing cold-start cache misses during early decode steps and ensuring the Sparse Pool contains frequently reused entries.
• On-Demand Cache Access: On each Decode step, if an entry required by $K^l_t$ is missing from the Sparse Memory Pool $I^l$, FlashTrans immediately prefetches it from CPU to GPU (host to device, H2D). If the pool exceeds capacity, the LRU engine evicts the lowest-recency entry before admitting the new one. Recentness is updated after every access. Newly generated vectors are written from GPU to host once a decode step completes.

Pseudocode for the essential steps:

procedure DecodeStep(t): for each layer l in 1…L: K ← Indexer(l, t) for each idx in K: if idx ∉ I^l: FlashTrans.prefetch(idx) if |I^l| ≥ PoolSize: evict ← LRU.evict() I^l.remove(evict) I^l.insert(idx) LRU.touch(idx) Attention(l) using I^l FlashTrans.drainWrites() end procedure

• Overlap Strategy Selection: Offline profiling is conducted to assess cache-miss patterns for each layer and context length. ESS selects DA-Overlap for layers with low miss rates, as it presents lower data movement costs, and DBA-Overlap for sections where miss rates spike (notably, in very long contexts), optimizing for computational overlap over data movement.

3. Mathematical Formulation of Resource Utilization

ESS's operational efficiency and profit emerge from its explicit decoupling of batch-size $B$ scaling from GPU memory growth as context length $L$ increases. Core formulations include:

• Latent-cache scaling: $S(L) = c \cdot L \quad \bigl( O(L) \bigr)$, where $c$ is a per-token cache constant.
• GPU/CPU memory usage (no offload): $$M_G^{\text{base}} = M_{\text{model}} + S(L) \times B$$
• With $\alpha$ fraction offloaded via ESS:
  • $$M_G^{\text{ESS}} = M_{\text{model}} + \alpha S(L) B$$
  • $M_C = (1-\alpha) S(L) B$
  • Freed GPU memory: $\Delta M_G = (1-\alpha) S(L) B$
• Throughput model: $T(B,L) = \frac{L \times B}{T_{\text{decode}}(B,L)}$
• Throughput improvement at key context lengths:
  • At 32K tokens: $\approx 69.4\%$ improvement
  • At 128K tokens: $\approx 123\%$ improvement

A plausible implication is that ESS shifts the limiting resource from GPU memory toward CPU memory and PCIe bandwidth, particularly as context lengths or batch sizes approach their pre-offload ceiling.

4. Performance Evaluation

High-fidelity simulation experiments configured with PCIe 5.0, FlashMLA, MTP=2, two-batch overlap enabled, and 4-node PD-disaggregated DeepSeek-V3.2-Exp demonstrate:

Context Length	Baseline Batch-Size (B)	Baseline Throughput (tokens/s)	ESS Batch-Size (B)	ESS Throughput (tokens/s)	Throughput Uplift
32K	52	9,647	160	16,348	69.4%
128K	13	3,669	54	8,170	123%

These figures account for FlashTrans overhead, cache misses, LRU operations, and overlap schemes. The metrics establish that with a Sparse Memory Ratio of 0.21 at 32K and 0.10 at 128K, substantial batch-size increases are realized, directly translating into throughput gains. The results validate the claim that ESS delivers precisely 69.4% and 123% throughput improvements at stated configurations.

5. System Trade-offs and Deployment Guidelines

ESS's deployment introduces several trade-offs:

• Latency Overhead: Fine-grained offload creates additional H2D/D2H traffic. FlashTrans (UVA-based) ameliorates this, boosting small-block bandwidth from 0.8 GB/s to approximately 37 GB/s, and compute–communication overlap further reduces perceptible transfer latency.
• Cache Miss versus GPU Pool Size: Smaller Sparse Memory Ratios free GPU RAM but raise miss rates. A recommended practice is to allocate at least ~6.4K latent entries (≈20% of the cache) to the GPU pool to control miss-induced traffic and avoid thrashing.
• Overlap Strategy Selection: DA-Overlap should be employed where misses per layer are low (fewer than 256 batches), whereas DBA-Overlap is advised for high-miss layers or extremely long contexts. These thresholds are to be established through a one-time offline profiling per deployment.
• Production Integration: ESS operates independently of training/inference frameworks (SGLang or similar), provided that UVA support, FlashTrans operator insertion at attention-call sites, and collection of per-layer cache-hit statistics are possible. The acceptance ratio ($\alpha$) should be tuned within $0.1$–$0.3$ for optimal GPU/CPU utilization.

A plausible implication is that in production environments with abundant CPU memory and fast host-device links, ESS provides a scalable method to "virtually" expand GPU memory capacity for decoding workloads with rapid context growth.

6. Significance and Practical Considerations

ESS substantiates a practical and scalable methodology for extending GPU memory for long-context LLM inference, enabling deployment of larger batches and improving throughput without accuracy loss. By redefining latent-cache as a first-class, offloadable resource, ESS overcomes the linear-memory barrier associated with transformer-based models under large context windows, previously a principal deployment bottleneck. This approach alters the cost structure in real-world LLM serving by enabling higher throughput without costly GPU upgrades, assuming sufficient CPU memory and high-bandwidth interconnect.

ESS's offload-aware design, achieved through a combination of fine-grained host-device transfers, adaptive overlap computation, and LRU-based hot-cache maintenance, ensures that the necessary cache information remains available to the attention mechanism with minimal latency penalty, even as working set sizes surpass on-device memory.

In summary, ESS demonstrates that with engineered device/host memory cooperation, throughput ceilings imposed by context-length scaling can be broken—achieving up to 123% acceleration for very long-context scenarios—while remaining model-agnostic and compatible with standard machine learning deployment frameworks (Chen et al., 11 Dec 2025).