LongMemEval-S Memory Benchmark

• The paper introduces LongMemEval-S, a benchmark evaluating LLMs through 500 multi-session dialogues with histories averaging 115,000 tokens per instance.
• It rigorously tests long-term memory by challenging models on knowledge updates, multi-session reasoning, and temporal logic tasks.
• The benchmark underscores trade-offs between accuracy and efficiency, driving advancements in retrieval-augmented and dynamic memory consolidation systems.

LongMemEval-S (Long-Term Memory Evaluation—Standard Split) is a benchmark suite designed to rigorously evaluate the long-horizon memory and retrieval capabilities of LLM-based conversational agents. It targets tasks that require robust recall, integration, and temporal reasoning over lengthy, multi-session user–assistant dialogues. Developed as part of the LongMemEval initiative by Wu et al. (2024), LongMemEval-S encapsulates the most challenging scenario of context scale, featuring 500 synthetic conversations and 500 manually curated questions embedded in histories averaging approximately 115,000 tokens per instance. The benchmark has become a critical standard for assessing models and memory architectures that operate far beyond the training-time context window, especially for tasks like fact recall, multi-session composition, and dynamic knowledge tracking.

1. Benchmark Design and Objectives

LongMemEval-S is constructed to diagnose and stress-test how conversational systems manage scalable, long-term, and contextually complex memory. The benchmark simulates a realistic setting for an AI assistant: accumulating and querying online memory across multi-session dialogues, personalized to individual user personas. Each conversation aggregates a large number of timestamped exchanges, requiring systems to resolve information extraction, knowledge updating, multi-session reasoning, and temporal logic—all within histories that vastly exceed the context window of conventional LLMs (Wu et al., 2024, Li et al., 6 Jan 2026, Ma et al., 6 May 2025).

Task categories covered in LongMemEval-S:

• Knowledge Update (KU): Detect and apply corrections or superseding information, e.g., updated addresses or preference changes.
• Multi-Session Reasoning (MS): Synthesize or reconcile information distributed across multiple sessions.
• Single-Session Information Extraction: Retrieve (i) facts stated by the user (SSU), (ii) facts referenced by the assistant (SSA), (iii) session-limited user preferences (SSP).
• Temporal Reasoning (TR): Perform reasoning over explicit dates, durations, or tolerate off-by-one discrepancies (e.g., "How many days since my birthday?").

The underlying motivation is to challenge memory-augmented systems on "needle-in-haystack" scenarios via tasks that mimic long-term user interactions seen in personal assistants and enterprise conversational agents (Wu et al., 2024, Rasmussen et al., 20 Jan 2025).

2. Dataset Structure and Composition

The dataset comprises 500 synthetic conversation streams, each representing a unique user persona (sampled from a template pool covering occupation, hobbies, values, etc.). Each conversation involves multiple sessions (typically 50–60) with extensive multi-turn user-assistant exchanges. Questions are distributed by design, ensuring broad coverage across memory types and temporal axes (Li et al., 6 Jan 2026).

Task Type	Number of Questions
Knowledge Update	78
Multi-Session	133
SS Assistant	56
SS Preference	30
SS User	70
Temporal Reasoning	133
Total	500

During benchmarking, each question is embedded at a strategic point within an exceedingly long dialogue, and evidence for answers may require integrating knowledge from multiple disparate parts of the conversational history. Distractor sessions, derived from external datasets (e.g., ShareGPT, UltraChat), are interleaved to prevent trivial retrieval (Wu et al., 2024, Li et al., 6 Jan 2026).

3. Evaluation Protocol and Metrics

All evaluated systems are assessed using a uniform LLM-as-Judge (LLJ) pipeline. The protocol involves generating a single-turn answer string for each query, which is then automatically assessed for semantic correctness by a prompt-instructed LLM (typically GPT-4o or GPT-4o-mini), with ">97% human–LLJ agreement" reported (Li et al., 6 Jan 2026).

The primary evaluation metric is accuracy: $$\mathrm{Accuracy} = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\{\hat y_i = y_i\}$$ where $N$ is the number of questions, and $\hat y_i$, $y_i$ denote model prediction and ground-truth, respectively.

For systems that expose evidence or memory retrievals (e.g., retriever–reader architectures), the following metrics are sometimes additionally reported on auxiliary slices or ablations:

• Recall@k: Fraction of gold evidence retrieved in the top-$k$ results.
• nDCG@k: Normalized Discounted Cumulative Gain for the ranking of retrieved evidence.

Latency and efficiency are measured as context size (tokens per query) and recall latency (median and tail, P50/P95), enabling direct comparison of memory consolidation and retrieval trade-offs (Patel et al., 17 Nov 2025, Li et al., 6 Jan 2026).

4. Memory System Architectures and Baselines

LongMemEval-S serves as the principal arena for comparing memory architectures, including both retrieval-augmented generation (RAG)-style stacks and memory-consolidation frameworks. The following approaches have been directly evaluated:

• Full-Context LLM (baseline): The entire 115k-token dialogue is provided to the LLM for each question. This baseline often exhibits pronounced degradation in accuracy (<40%) and incurs prohibitive compute latency (Patel et al., 17 Nov 2025, Rasmussen et al., 20 Jan 2025).
• Fact Card Memory Layers (e.g., ENGRAM-R): Compact, typed fact snippets (Fact Cards) with explicit citation control are retrieved and presented as context, demonstrating marked gains in efficiency and accuracy (+30.1% on MS, +13.5% on TR) (Patel et al., 17 Nov 2025).
• Temporal-Hierarchical Frameworks (e.g., TiMem): Memory is structured in a temporal memory tree, enabling consolidation from fine-grained episodic details up to high-level persona summaries, and balancing recall precision with efficiency. TiMem achieves 76.88% ± 0.30 overall accuracy and reduces memory context by ≈22.8% compared to strong RAG baselines (Li et al., 6 Jan 2026).
• Temporal Knowledge Graphs (e.g., Zep/Graphiti): Episodes and facts are organized in bi-temporal, fact-entity graphs with dynamic timestamp-based validity windows. This enables accurate support for Knowledge Update and Temporal Reasoning queries while reducing average context to ≈1,600 tokens (vs. 115k) and latency by 90% (Rasmussen et al., 20 Jan 2025).

Other notable baselines include A-MEM (agentic self-organizing memory), Mem0 (semantic clustering), MemoryBank (preference-weighted), MemoryOS, and MemOS (multi-tier/page-swap memory) (Li et al., 6 Jan 2026).

5. Empirical Results Across Systems

Selected results illustrate both the magnitude of the long-context memory challenge and the progress enabled by structured memory frameworks.

Memory System	Answer Model	Accuracy (%)	Context Tokens	Latency (s, P50/P95)
Full-Context	GPT-4o-mini	55.4	115,000	31.3 / 8.76
Zep/Graphiti	GPT-4o-mini	63.8	1,600	3.20 / 1.31
TiMem	GPT-4o-mini	76.88 ± 0.30	1,270.62	1.76 / 4.48
ENGRAM-R	gpt-oss-20b	59.8	54,301 (COG)	1.88 / 5.54
MemOS	GPT-4o-mini	68.68 ± 0.16	1,091.51	1.64 / 2.70

When focusing on the two hardest categories, ENGRAM-R demonstrates a 30.1% absolute accuracy gain for multi-session reasoning and a 13.5% gain for temporal reasoning, while cutting input tokens by ≈96% and median latency by ≈81% (Patel et al., 17 Nov 2025). TiMem establishes state-of-the-art (<1.3k memory tokens/query; 76.88% accuracy), outperforming baselines by up to 12.9 percentage points (Li et al., 6 Jan 2026).

Sustained improvements are achieved on temporal reasoning and multi-session logic, but retrieval and recall remain challenging, particularly with assistant-centric queries or under evidence budget constraints (Patel et al., 17 Nov 2025, Rasmussen et al., 20 Jan 2025).

6. Analytical Findings, Ablations, and Limitations

Analysis across evaluated systems surfaces several recurring findings:

• Signal-to-noise challenge: Full-context LLMs struggle to isolate relevant segments in ultra-long input, suffering from context overload and “dilution” of evidence signal (Patel et al., 17 Nov 2025).
• Temporal organization: Explicit modeling of fact validity intervals and temporal hierarchy (memory trees, knowledge graphs) proves critical for knowledge update and time-based inference (Li et al., 6 Jan 2026, Rasmussen et al., 20 Jan 2025).
• Recall and consolidation: Fact granularity, session decomposition, semantic-guided consolidation, and context gating are essential for bridging precision (avoiding distractors) and recall (capturing all salient evidence) (Wu et al., 2024, Li et al., 6 Jan 2026).
• LLM-as-Judge reliability: LLM-based meta-evaluation is robust with >97% human alignment, but subject to residual bias—especially salient for high-stakes applications (Patel et al., 17 Nov 2025, Li et al., 6 Jan 2026).
• Resource-efficiency trade-offs: Retrieval budgets ($k$) present an explicit trade-off; setting $k$ too low may omit critical evidence, while too high increases compute (Patel et al., 17 Nov 2025).

Category-level breakdowns indicate that accurate updating, multi-session synthesis, and single-session user recalls are the main axes of progress for leading systems, while temporal-reasoning and assistant-centric queries still invite further refinement (Rasmussen et al., 20 Jan 2025, Li et al., 6 Jan 2026). Manifold analyses of memory representations via distance metrics confirm that semantic consolidation mitigates noise and enables stable persona tracking in the embedding space (Li et al., 6 Jan 2026).

7. Broader Impact and Future Directions

LongMemEval-S is now a de facto benchmark for research and development of memory architectures for LLM-based agents in both academic and industrial contexts. Its structure directly supports innovations in:

• Retrieval-augmented LLMs: Guiding retrieval and reading policy design under ultra-long context constraints.
• Temporal reasoning: Informing fact validity modeling, date arithmetic, and temporal slotting in agent memories.
• Personalization and long-horizon learning: Stress-testing approaches to persona stabilization, evolving preference recall, and user identity continuity.

A plausible implication is that further advances will require not only architectural improvements (e.g., adaptive memory budget allocation, distributed memory consolidation) but also more challenging variants (LongMemEval-M), human-in-the-loop judging, and richer evaluation of abstention/uncertainty alongside accuracy (Wu et al., 2024, Li et al., 6 Jan 2026). The emergence of LongMemEval-S as a shared yardstick will continue to sharpen competitive benchmarks and drive the development of lifelong conversational reasoning in AI systems.