DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings

Abstract: LLM agents increasingly act as personal assistants that must remember a user's profile over months: who they are (attributes), what they routinely do (habits), and what they prefer (preferences), and keep it updated as jobs, routines, and tastes drift. Existing benchmarks evaluate this "memory" ability through short, simplified interactions, missing three core properties of real behavior: the profile is heterogeneous, with attributes, habits, and preferences evolving on different timelines; changes are driven by external context such as seasons and life events; and evidence is rarely stated explicitly, instead scattered across many small actions in different apps that a memory system must infer from. We introduce DynamicMem, a synthetic benchmark that constructs 15 months of activity per user, providing long-term multi-app data that real users' privacy keeps out of reach. It provides user-consistent trajectories averaging 2.2M tokens and 1,772 grounded events per user across 16 applications such as e-commerce, fitness, and social platforms. The profile evolves over this period and is never given explicitly: each attribute, habit, or preference must be inferred from small signals scattered across apps. We evaluate at five quarterly checkpoints to track how systems scale as history grows. Benchmarking five representative systems exposes problems a single accuracy score hides: (i) profile reconstruction degrades with history length while service-task accuracy stays flat, despite both drawing on the same memory; (ii) no system both keeps facts that stay true and replaces facts that change, with errors clustering on preferences and on naming the exact referent; and (iii) over 93% of failures trace to what the memory retrieves, not to the model writing the answer, so the largest room for improvement lies in memory itself. Code: https://wenyaxie023.github.io/DynamicMem/

• The paper introduces DynamicMem, a benchmark that simulates 15-month user logs to evaluate long-horizon memory in LLM agents.
• It details a rigorous evaluation protocol with field-level metrics for state completion and personalized service across evolving profiles.
• Findings highlight trade-offs between memory retention and agility, with preferences showing significant degradation over extended histories.

DynamicMem: Benchmarking Long-Horizon Personalization Memory in LLM Agents

Motivation and Benchmark Design

The ability of LLM agents to serve as effective personal assistants fundamentally depends on their capacity to construct, maintain, and continually update rich user profiles from long behavioral histories. DynamicMem addresses the lack of benchmarks that (i) reflect the heterogeneous, causally driven and distributed nature of real-world user state evolution, and (ii) rigorously evaluate cross-application, long-term memory tracking at scale. Existing evaluations typically rely on short, homogeneous interaction logs, static profiles, and single-snapshot metrics, failing to probe core challenges in retention, update, and evidence integration over multi-application, months-long trajectories.

DynamicMem generates synthetic yet realistic 15-month user activity logs (average 2.2M tokens and 1,772 events per user) across 16 applications. User profiles evolve along three axes: attributes (discrete entities), habits (medium-term routines), and preferences (long-term inclinations), each changing at distinct temporal scales in response to explicit external drivers (e.g., seasonal shifts, regulatory changes, life events). Rather than explicit user-state statements, evidence is scattered across small, causally-linked actions across apps, requiring inference from distributed behavioral traces. The evaluation protocol probes systems at five quarterly checkpoints, exposing how performance scales as historical context grows, and enables per-category and per-transition-type analysis.

Figure 1: DynamicMem constructs evolving user profiles, generates intent-driven event chains, and grounds them into state-consistent multi-app interaction logs for long-horizon memory evaluation.

Profile Construction and Trajectory Synthesis

DynamicMem’s data pipeline comprises three tightly integrated stages: (1) construction of a multi-timescale, causally-grounded user profile; (2) intent-conditioned event chain generation; and (3) state-consistent grounding of events into application logs. Each user is anchored with a stable base profile and six life domains, with profile attributes, habits, and preferences updated at each quarterly window by sampling typed state deltas explicitly attributed to background world events.

Event chains, which span multiple apps (e.g., Gmail, LinkedIn, WhatsApp), operationalize state changes as sequences of actions motivated by intent and context. State consistency is strictly enforced: each app is a mutable environment, all entities maintain persistent identity across references, and downstream entries reference prior log artifacts as needed. Human validation and in-prompt LLM-based checking ensure coherence in both in-domain dynamics and cross-domain attribute alignment.

Evaluation Tasks and Protocol

Benchmarking focuses on two central memory tasks: State Completion (reconstructing the current user profile state of a given type) and Personalized Service (leveraging the inferred state in concrete service actions such as filtering, auto-filling, or reminder generation). These tasks are evaluated for each profile facet (attributes, habits, preferences) independently. Importantly, both tasks utilize only the memory constructed from checkpoint-bounded app logs, but differ in the nature of the retrieval and reasoning required.

Highly granular metrics are used. Scoring is performed at the field level, combining a core-correctness binary and a detail-quality trinary (0/1/2), aggregating into fine-grained summary statistics. LLMs are used as judges, and reliability is validated via manual audit.

Quantitative Results and Failure Analysis

Systematic evaluation across five representative memory architectures (Vanilla RAG, HippoRAG2, MemoryOS, A-Mem, SimpleMem) reveals several nontrivial phenomena:

1. State Completion degrades with increased history, while Personalized Service does not. From C1 to C5, State Completion declines for all systems (e.g., -4.4 for RAG, -14.2 for MemoryOS), in contrast to marginal improvement or flatness for Personalized Service (up to +4.9 for A-Mem).

   Figure 3: State Completion and Personalized Service scores across quarterly checkpoints for all systems; see degradation in State Completion and stability/improvement in Personalized Service.

2. Evidence retrieval recall for gold facts drops for both tasks, but Personalized Service queries increasingly benefit from the denser trajectory (as measured by rising top-5 query–chunk cosine similarity), whereas State Completion depends on singleton logs whose retrievability decays.

   Figure 2: Retrieval recall on gold evidence and mean query-chunk alignment; Personalized Service queries gain from longer histories, State Completion queries degrade.

3. Profile facet decomposition shows most long-horizon degradation is due to Preferences, not Habits or Attributes. The implicit, distributed nature of behavioral preferences results in increased interference as more logs accumulate, while Habits (which are repeatedly demonstrated) stay robust.

   Figure 4: Breakdown of State Completion by category; preferences are most affected by increased horizon.

4. No single design simultaneously solves long-term retention and rapid update. Systems trade off between memory longevity and agility to change. Retrieval- and structure-based memories either overcommit to stable facts (blocking update) or overmerge new values (forgetting rare but important facts).

   Figure 5: Retention and update scores dissected by state family; systems excel at one regime but not both.

5. For Personalized Service, proactive routine selection (matching context to the correct habit) is universally poor, indicating the main bottleneck is not memory recall per se but scenario-to-habit selection.
6. Error Attribution: Over 93% of failures are due to the memory system delivering insufficient (irrelevant, identity-missing, detail-missing, or conflated) evidence—non-answer-model bottlenecks. Structured memories shift the predominant failure from details to identity, while hierarchical schemes (e.g., MemoryOS) concentrate retrieval failures earlier in history.

   Figure 6: Distribution of failure types for each system and task; identity misses dominate in structured memories.

Implications for Long-Horizon Memory Systems

DynamicMem highlights that long-horizon memory is not a solved problem. The temporal scaling of distributed inference and robust update/retention operations remain open, especially for implicit attributes. Moreover, the dissociation between memory-reliant question-answering and memory-requiring action (service) reveals meaningful distinctions between “knowing” vs. “acting on” state, motivating architectural innovations.

For memory architectures, major open directions include:

• Explicit separation of retention vs. update policies, potentially typed by information category.
• Causal attribution linking evidence to context, to mitigate interference and support targeted updates.
• Scenario-conditioned retrieval for action grounding, transcending stateless chunk-based selection.

Practically, these findings inform the deployment of LLM-based assistive agents in real-world settings. Deployment without robust, interpretable, and updatable user modeling will risk either errorful persistence (ignoring profile drift) or loss of critical long-term facts, reducing real-world usability, especially as context windows and adoption grow.

Conclusion

DynamicMem advances the state of memory evaluation for LLM agents by simulating the heterogeneous, causally-evolving, and behaviorally-grounded user state trajectories found in real-world applications. Its rigorous, multi-faceted protocol surfaces failure modes and challenges masked by aggregate or short-context tests, and provides a robust testbed for novel memory system designs targeting long-horizon personalization. Future directions include neural methods that reason over structured event graphs, richer scenario-to-intent mapping for service tasks, and integration with privacy-preserving or user-controlled memory boundaries.