What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

Abstract: LLMs, and the agents built on them, spend an ever-growing share of their compute and memory on remembering: caching attention keys and values, carrying long prompts, maintaining recurrent state, and storing what happened in previous turns and sessions. Because none of this memory is free, four largely separate research communities have each learned to compact it. They evict or quantize the KV cache, prune or distill prompts, bound architectural state, and consolidate agent memory. We argue that these are instances of one problem: a rate--distortion decision about what context-derived information to retain versus discard, at what fidelity, under a resource budget, so as to preserve downstream task utility. We make this lens precise with a single compaction objective and a layer-agnostic lower bound, use it to build a seven-axis taxonomy that classifies methods from across the stack uniformly, and use it to transfer mechanisms between layers that have never been connected, from serving-stack KV management to agent long-term memory. Two patterns hold across the survey. At every layer the signal that decides what to keep is attention magnitude or recency, and it fails in the same way everywhere, by discarding, before the query is known and with no way to undo it, information the query later needs. And while compression is measured carefully on single-turn long context, the repeated compaction that agents actually perform is almost never measured, and no benchmark holds one budget axis across all the layers at once. We turn both observations into a benchmark proposal, a small reference experiment, and a set of compaction-aware design principles, and we map the open problems.

• The paper introduces a unified rate–distortion framework to formalize memory compaction across LLM layers, enabling cross-method comparisons and error predictions.
• It demonstrates that query-conditioned and reversible compaction methods outperform irreversible, query-agnostic ones under limited memory budgets.
• Empirical results reveal that below critical information thresholds, all compaction methods incur predictable accuracy losses, informing efficient algorithm design.

A Unified Rate–Distortion Framework for Memory Compaction in LLMs and Agents

Overview and Motivation

The study "What to Keep, What to Forget: A Rate–Distortion View of Memory Compaction in LLMs and Agents" (2607.08032) establishes a rigorous, unifying perspective on memory compaction across all operational layers of LLMs and agentic systems. Instead of treating diverse techniques—such as KV-cache eviction, prompt pruning, recurrent state bounding, and agentic episodic memory consolidation—as independent, the authors argue that these are all manifestations of a single underlying optimization: selecting, under a memory budget, which information derived from context to retain (and at what fidelity) for maximal downstream utility.

The paper's thesis recasts compaction as a rate–distortion problem, formalizing it with a general objective and lower bound that applies agnostically to inference caches, prompts, architectural states, and agent memories. This intellectual move enables direct comparison of methods from previously disconnected communities, facilitates mechanism transfer across layers, and predicts universal failure modes and tradeoffs. Notably, the work introduces a comprehensive taxonomy, bridges previously siloed techniques, proposes a unified evaluation protocol (COMPACT-Bench), and empirically demonstrates its theoretical predictions.

Theoretical Framework: Rate–Distortion Objective and Lower Bound

At its technical core, the paper formulates memory compaction as optimizing:

$$\min_{\theta} \ \mathbb{E}_{(H,Q,Y)} [ \ell(U(C_\theta(H), Q), Y) ]$$

subject to a representation rate constraint, $rate(Z) \le B$, where $H$ is the historical context, $C_\theta$ is a compaction operator, $Z = C_\theta(H)$, $U$ is the usage operator (typically continued model computation), and $Y$ is the task target. This is shown to be equivalent to a rate–distortion problem: trading off the retained information about the history ($I(Z;H)$) against predictive information for the answer ($I(Z;Y|Q)$), aligning with the information bottleneck principle.

Crucially, this formalism leads to a layer-agnostic lower bound: $P_e \geq \frac{H(Y|Q) - B - 1}{\log|\mathcal{Y}|}$ for error probability $rate(Z) \le B$0 whenever the budget $rate(Z) \le B$1 falls below the task's information content $rate(Z) \le B$2. This immediately yields strong, falsifiable predictions:

• Below the information content threshold, all compaction methods (irrespective of architectural layer) must incur errors at a predictable rate.
• Compression is fundamentally task-dependent: information-dense tasks (e.g., exact, multi-hop retrieval) resist compaction, while redundancy-tolerant tasks allow aggressive reduction.
• Query-agnostic compaction (which does not condition on the query or future tasks) pays a strict penalty in effective budget proportional to the query entropy $rate(Z) \le B$3 versus query-conditioned methods.

Seven-Axis Taxonomy and Layer-Unified View

The formalism motivates a taxonomy spanning seven design axes: granularity, lifecycle stage, lossiness/fidelity, query/task adaptivity, learnability, compaction mechanism, and storage substrate. This schema is applied to classify and compare methods across four canonical layers:

1. KV-cache: Eviction, quantization, low-rank projections, and merging. Most methods are query-agnostic and irreversible, leading to brittle failures when queries require dropped tokens.
2. Prompt/context: Hard (token/sentence-level) or soft (dense-vector) compaction; importance is estimated via LM self-information, learned classifiers, or explicit mutual information. Query-conditioned and reversible methods demonstrably outperform agnostic, lossy ones on downstream tasks requiring precise recall.
3. Architectural/stateful: Models (e.g., RMT, Mamba, Infini-attention, RWKV) that bound state size by design. The lower bound shows that fixed state must lose information on sufficiently demanding tasks, explaining observed accuracy cliffs on retrieval-heavy or generative benchmarks.
4. Agent/semantic memory: Methods for curation, consolidation, and retrieval in agentic memory—often leveraging LLM-driven salience as the compaction signal. The most critical axis is reversibility: methods relying on lossy summarization suffer accumulating and irrecoverable errors, particularly under repeated compaction.

Key Empirical Evidence

The authors empirically validate the universality of their predictions using a unified bytes-per-token memory budget axis on a standard LLM and by measuring performance on both single-turn and repeated compaction tasks:

• Accuracy–Budget Frontier: As budget falls below the information threshold for a retrieval task, all tested compaction methods (KV-eviction, quantization, etc.) degrade in accuracy, with random-eviction acting as a lower-bound control.

(Figure 1)

Figure 1: The accuracy--budget frontier on natural-filler needle retrieval with Qwen2.5-1.5B, six KV-compaction methods on one bytes-per-token-of-history axis. Accuracy collapses once the budget falls below the task's information content, and the random-eviction control isolates the value each scorer adds.

• Repeated Compaction: When measuring fact recall over multiple compaction events, irreversible (summarize) methods exhibit superlinear error accumulation, while reversible (retrieval-based) memory maintains performance near the theoretical upper bound, even after many cycles.

(Figure 2)

Figure 2: Fact recall against the number of compaction events for an irreversible (summarize) and a reversible (retrieve) operator. Reversible memory stays near 0.95; irreversible memory loses roughly half its facts at every frequency. The two look identical under single-turn evaluation and diverge only when compaction repeats.

These quantitative results demonstrate that the theoretically justified properties—reversibility (P-rev) and query-conditioning (P-q)—are operationally predictive of robustness under compaction pressure.

Implications and Design Principles

The work distills multiple practical principles for future compaction approaches:

• At fixed budget, prefer reversible operators (retrieval-backed, archival memory) over irreversible ones (permanent eviction, lossy summarization).
• Condition compaction decisions on the query distribution wherever feasible; avoid query-agnostic policies unless forced.
• Adopt multi-fidelity storage, separating a recoverable episodic tier from a lossy semantic tier, with principled promotion and demotion mechanisms.
• Allocate budget non-uniformly, according to marginal utility or information-theoretic signals (e.g., per-head, per-layer, per-fact $rate(Z) \le B$4), rather than uniform ratio.

The paper also highlights persistent evaluation limitations—a lack of cross-layer, budget-normalized benchmarks—addressed in their reference COMPACT-Bench protocol, which enables systematic and comparable measurement across all compaction modalities.

Theoretical and Practical Open Problems

Strong theoretical claims are buttressed by recent lower bounds on attention-based computation and recurrent state, showing that exact retrieval is impossible without sufficient memory budget, and that information-dense segments (e.g., attention sinks) are structurally compressibility barriers.

However, the predictive quantity $rate(Z) \le B$5—the minimum sufficient information content for a given task/query—remains unmeasured in most practical setups. Closing this gap with a compression scaling law would allow for principled, rather than heuristic or empirical, budget assignments. Additionally, the composition of different compaction operators (quantization, low-rank, eviction, retrieval) lacks a systematic empirical or theoretical map.

In the context of agents, the issue of error compoundment, vulnerability to poisoning due to irreversible consolidation, and the need for asynchrony and RL-stable self-curation are flagged as open challenges.

Future Impact and Outlook

The unification under a rate–distortion lens is likely to have profound downstream implications across the AI stack:

• Algorithmic design: Tighter theoretical predictions for permissible compaction and failure points will inform both future LLM architecture and agentic system design.
• Practical deployment: Robust, reversible, query-conditioned, multi-fidelity compaction will be integral to scaling LLMs for complex, persistent, and interactive workloads under realistic resource constraints.
• Benchmark and methodology: Cross-layer, budget-aligned evaluation will become the standard for reporting and comparison; without this, claimed advances may be illusory or incommensurate.
• Security and governance: Versionable and auditable compaction is an emergent concern, especially for persistent, agent-based AI systems.

Conclusion

By grounding memory compaction in a precise, cross-cutting rate–distortion formalism, this work provides both an explanatory and prescriptive framework for understanding, benchmarking, and evolving resource-efficient memory mechanisms in modern LLMs and multi-turn agentic systems. The explicit mapping of axes, failure modes, and empirical behaviors provides a rigorous foundation for future research, system design, and theoretical analysis, leaving as its central open problems the predictive measurement of information content and the principled coordination of multiple compaction strategies.