Shared-KV Consolidator for Scalable LLM Inference

• Shared-KV Consolidators are high-performance systems that manage, compress, and share key-value caches across GPUs and nodes to accelerate transformer inference.
• They employ techniques like zero-copy DMA, CXL shared memory, and prefix-aware caching to minimize latency and boost throughput.
• Advanced synchronization, hash structures, and NUMA-aware allocation ensure memory consistency and efficient multi-node cache sharing in distributed ML workloads.

A Shared-KV Consolidator is a high-performance system for managing, compressing, and sharing key-value (KV) caches in large-scale deep learning deployments, particularly in rack-scale, multi-node, or multi-instance contexts. The consolidator enables efficient reuse, transfer, and management of KV tensors across computational workers or nodes, addressing bottlenecks in GPU memory, interconnect bandwidth, and time-to-first-token (TTFT) during transformer-based inference. Implementations such as TraCT leverage CXL shared memory and prefix-aware caching, allowing direct GPU access without intermediate networking layers, while other systems target disk, SSD, or distributed memory substrates for multi-instance sharing.

1. Architectural Foundations and Rack-Scale Design

Shared-KV Consolidators are architected to maximize prefix reuse and minimize data movement overhead in decomposed LLM inference pipelines. At rack scale, TraCT (Yoon et al., 20 Dec 2025) exemplifies this architecture through:

• Central CXL Type-3 shared memory, mapped via DAX into every server in the rack.
• Compute-heavy prefill workers and decode-phase latency-critical workers both access the same CXL region as byte-addressable memory.
• The consolidator's pipeline:
  1. Client requests prompt prefill.
  2. Prefill worker probes a prefix-index hash table in the CXL region.
  3. Cache-hit: GPU issues DMA read from shared CXL to retrieve required KV blocks.
  4. Cache-miss: GPU computes K/V, writes block to CXL via DMA, and updates the prefix-index.
  5. After all required blocks are present or generated, prefill signals the decode worker, which issues CXL→GPU DMA for all blocks and proceeds with autoregressive generation.
  6. Decoding attends fully over the shared KV cache; memory is freed and blocks eventually evicted by LRU logic.

All KV blocks are physically stored as aligned, contiguous arrays with per-block metadata footers. The system ensures true zero-copy DMA between GPU and CXL: no NIC, host DRAM, or software bounce buffer intervention.

2. Synchronization and Consistency Techniques

Non-coherent shared memory presents challenges in cross-node synchronization, metadata consistency, and cache-line state visibility. TraCT implements a two-tier synchronization mechanism:

• Tier 1: Per-node local locks (pthread_mutex analogues) restrict entry into global arbitration, bounding cross-node contention by number of nodes.

• Tier 2: Global CXL lock array, organized into slot states ({IDLE, WAITING, LOCKED}). Entrants mark their slot WAITING, spin until LOCKED, then release, synchronizing metadata writes via explicit clflush + mfence boundaries to prevent stale data propagation.

This software arbitration strategy works without global atomics, ensuring lock-safe metadata manipulation and predictable cross-node consistency for operations on the shared hash table, allocator bitmap, and control pages.

3. Prefix-Aware KV Caching and Hash Structures

Efficient prefix-oriented cache hit logic is foundational. Shared-KV Consolidators use:

• Prefix-preserving block hashing: For sequence $T_1...T_n$, block hashes are recursively defined as $h_i = H(h_{i-1}, T_i)$.
• Fixed-size block indices: Token blocks are mapped by $B$ (block size), with hashes stored in a static, load-factor bounded hash table alongside block metadata: $(h, offset, length, ref\_count, LRU\_links)$.
• Linear probing and eviction: On cache insertions, probe for available bucket, allocate CXL storage, DMA block transfer, and clflush to mark as READY; hits atomically update ref_count; LRU logic manages head/tail updating and eviction with cache-line-isolated metadata.

This structure enables robust prefix-aware deduplication, both for cache hits on context reuse and for eviction safety via reference counting in concurrent multi-node workloads.

4. Non-Coherent Memory Management and Allocation

Maintaining metadata-payload isolation is essential in non-coherent shared memory.

• Payloads: Large block tensors (keys, values) are accessed only via DMA, never touched by CPUs—obviating cross-cache visibility issues.
• Metadata: Clusters small control structures into cache-line-aligned control pages, always modified with clflush + mfence.
• Allocation: Global chunk allocator in CXL, bitmap-driven, with two-tier locks for mutation; nodes request whole chunks, sub-allocate blocks within their own DRAM for per-node local heaps. The CXL object store publishes shared root pointers for all global indices.

5. Performance Modeling and Quantitative Impact

Formalism for TTFT and throughput in consolidators:

Let

• $N$ = model size (layers × heads × head_dim)
• $L$ = context length (tokens)
• $\alpha$ = fraction of blocks served from cache
• $S_{KV}$ = full KV cache size $O(N \cdot L)$
• $B_{cxl}, B_{rdma}$ = bandwidth of CXL vs RDMA
• $\ell_{nic}$ = NIC hop latency.

Formulas:

RDMA-based TTFT:

$$T_{TTFT}^{RDMA} \approx T_{compute} + (1-\alpha)\left(\frac{S_{KV}}{B_{rdma}} + 2 \ell_{nic}\right) + \alpha\left(\frac{S_{block}}{B_{rdma}} + 2 \ell_{nic}\right)$$

CXL-based TTFT:

$$T_{TTFT}^{CXL} \approx T_{compute} + (1-\alpha)\frac{S_{KV}}{B_{cxl}} + \alpha\frac{S_{block}}{B_{cxl}}$$

TTFT speedup ratio:

$$\frac{T_{TTFT}^{RDMA}}{T_{TTFT}^{CXL}} \approx \frac{(1-\alpha)\frac{S_{KV}}{B_{rdma}} + \alpha \frac{S_{blk}}{B_{rdma}} + 2\ell_{nic}} {(1-\alpha)\frac{S_{KV}}{B_{cxl}} + \alpha \frac{S_{blk}}{B_{cxl}}}$$

Peak throughput scaling:

$$\text{Speedup} = \frac{B_{cxl}}{B_{rdma}} \cdot \left[ 1 + \frac{2\,\ell_{nic} B_{rdma}}{S_{KV}} \right]^{-1}$$

Empirical results: On real hardware (Niagara 2.0 CXL Type-3, 10 GB/s), TraCT reduces average TTFT by up to 9.8×, P99 prefill latency by 6.2×, and increases peak end-to-end throughput by 1.6× compared to RDMA/DRAM-based baselines; GPU compute occupancy and power both drop noticeably (Yoon et al., 20 Dec 2025).

6. Implementation Best Practices and Principles

Rigorous system design for Shared-KV Consolidation requires:

• Zero-copy DMA for all GPU-CXL transfers via page-locked host memory.
• Offset-only addressing: Store offsets, not virtual pointers, in shared structures for full locality.
• Cache-line isolation on hot metadata (buckets, locks, indices).
• Two-tier locking and explicit flushes for cross-node synchronization.
• Static hash tables/indices to avoid dynamic heap churn.
• NUMA awareness: Pin lock-manager and metadata threads to NUMA node attached to CXL root complex.
• Eviction safety via live ref_counting and proper delayed LRU removals.

Adherence to these principles results in a robust, scalable consolidator design capable of maintaining sub-microsecond metadata operation latencies, high-bandwidth rack-wide storage, and predictable, fault-tolerant cache sharing behavior.

7. Comparison with Alternative Shared-KV Management Mechanisms

TraCT is distinguished by tightly integrated CXL shared memory as the KV-transport and cache substrate. By comparison,

• Disk-based systems (e.g., Shared RAG-DCache) service multi-instance LLM inference through a disk-resident cache and proactive prefetching, but incur higher storage/retrieval latencies (Lee et al., 16 Apr 2025).
• Near-data processing (e.g. Co-KV) splits host and SSD compaction workloads for LSM-tree key-value stores, using collaborative offloading for throughput and write amplification reduction (Sun et al., 2018).
• Advanced transformers employ cross-layer fusion (e.g., FusedKV, FusedKV-Lite) or SVD-based compression (xKV, CommonKV) to consolidate or compress KV caches within/between layers, reducing memory further at the model representation level (Lin et al., 3 Dec 2025, Chang et al., 24 Mar 2025, Wang et al., 22 Aug 2025).
• Semantic sharing (e.g., SemShareKV) aligns token-level KV reuse between prompts via fuzzy matching and RoPE, targeting semantic redundancy rather than strictly shared prefix (Zhao et al., 29 Sep 2025).
• MoE/Multi-agent frameworks (PiKV, mixSGA, MoSKA, KVCOMM) deploy sharded, scheduled, or cross-context KV consolidation for distributed, heterogeneous compute environments (Liu et al., 2 Aug 2025, Rhee et al., 8 Nov 2025, Song et al., 16 Jun 2025, Ye et al., 14 Oct 2025).

Hence, Shared-KV Consolidators—when implemented on rack-scale CXL—yield the optimal TTFT reduction and throughput gains for high-concurrency, multi-GPU transformer deployments; they form the gold standard for tightly-coupled GPU/shared-memory LLM serving in contemporary distributed ML infrastructures.