LoCoMo & PerLTQA: Agent Memory Benchmarks

Updated 9 February 2026

The paper introduces LoCoMo and PerLTQA as benchmarks that quantitatively measure retrieval and reasoning over lifelong agent memory using metrics like BLEU, F1, and ROUGE.
LoCoMo simulates multi-session dialogues with extensive agent interactions while PerLTQA aggregates personalized semantic and episodic memories for structured QA analysis.
Comparative evaluations, including xMemory, demonstrate that hierarchical memory structuring enhances retrieval efficiency and multi-hop reasoning in long-context settings.

LoCoMo and PerLTQA are advanced benchmarks designed to rigorously evaluate the long-term memory, retrieval, and synthesis capabilities of LLMs within agent-like, conversational, and personal knowledge settings. These resources underpin recent progress and critical analyses in agent memory architectures, particularly focusing on the granularity, longevity, and reasoning over multi-session and lifelong memory streams.

1. Benchmark Definitions and Core Objectives

LoCoMo (Long-Conversation Memory) targets very long-term conversational memory through simulated, multi-agent interactions spanning up to 35 sessions and 9,000 tokens per dialogue. Each agent possesses a persona, temporal event graph, short- and long-term memory modules, and multimodal capacity via image sharing and captioning (Maharana et al., 2024). The conversations are constructed using an LLM–human pipeline, ensuring both high throughput and manual verification for narrative and factual consistency.

PerLTQA (Personal Long-Term QA) is constructed to evaluate QA grounded in lifelong personal memory, explicitly distinguishing between "semantic" memory—profiles, world knowledge, relationship graphs—and "episodic" memory, represented through rich event timelines and historical dialogues (Du et al., 2024). PerLTQA adopts a taxonomy rooted in cognitive science to assess how well agent models leverage structured and unstructured personal memory across a wide spectrum of question types.

Both benchmarks are aligned with three principal objectives:

Quantitatively measure retrieval and reasoning over long, temporally-structured memories.
Distinguish memory types and their impact on reasoning performance.
Provide robust evaluation tasks for both retrieval-augmented and generative memory models.

2. Dataset Construction and Statistics

LoCoMo's pipeline simulates two generative agents (backed by gpt-3.5-turbo) interacting over in-world timelines of 6–12 months, each agent initialized with:

Persona profiles (expanded via prompt-based LLM synthesis).
Temporal event graphs with up to 25 causally-linked events and timestamps.
Short-term memory: session-wise summaries.
Long-term memory: atomic, turn-indexed observations.
Multimodal capabilities: image captioning (BLIP-2) and web-searched image insertion with grounded textual reactions.

All dialogues, numbering 50, undergo human review for event grounding and long-range coherence. Summative statistics include:

Average turns per conversation: ≈304.9
Average sessions per conversation: ≈19.3
Tokens per conversation: ≈9,209.2
Images per conversation: ≈32.3

PerLTQA aggregates personalized memory for 141 character profiles, spanning 10 professional categories and 299 specialties. It divides its memory bank into:

Semantic: profiles (PRO), and social relationships (SR) among 1,339 pairwise links.
Episodic: 4,501 detailed event narratives (avg. 313 words each), and 3,409 dialogues (25,256 utterances).
QA pairs: 8,593 annotated item pairs, each linked to memory "anchors" (≈2.8 anchors per QA).

This schema enables detailed, anchor-aware evaluation of memory classification, retrieval, and answer generation.

3. Evaluation Protocols and Tasks

Both resources employ multi-stage evaluation frameworks:

LoCoMo defines:
- Question Answering (QA): Five question types (single-hop, multi-hop, temporal, open-domain, adversarial), evaluated by token-level F1 and Recall@k (for retrieval tasks).
- Event Summarization: Extraction of concise event sequences from full-dialog histories; scored by FactScore (atomic fact overlap) and ROUGE-1/2/L.
- Multimodal Dialogue Generation: Next-turn generation including both text and images, evaluated by BLEU, ROUGE-L, and MM-Relevance.
PerLTQA implements a three-stage memory integration and QA pipeline:
- Memory Classification: Question type is classified as "semantic" or "episodic," using a fine-tuned BERT classifier or LLM prompt-based inference; micro-averaged F1 and accuracy are reported.
- Memory Retrieval: Top-k memory items selected via BM25 (sparse), DPR (dense supervised), or Contriever (unsupervised dense). Final ranking fuses classifier and retriever scores.
- Memory Synthesis: LLMs generate answers using concatenated, re-ranked memory snippets, following instructional prompts. Coherency (by gpt-3.5-turbo), correctness, and mean average precision (MAP) over gold memory anchors are measured.

4. Baseline Architectures and System Comparisons

Across both datasets, extensive baselines highlight the limitations and opportunities in current retrieval-augmented generation (RAG) memory systems:

System	Memory Structure	Main Retrieval Mechanism
Naive RAG	Flat-chunked	Top-k embedding similarity
A-Mem	Zettelkasten-style notes	Structured note-based retrieval
MemoryOS	Hierarchical w/ lifecycle	Multi-level chunk + pruning
LightMem	RAG + LLMLingua-2 pruning	Compressed chunk selection
Nemori	Cognitive episode struct.	Self-organizing episodic retrieval
xMemory	Theme+semantics hierarchy	Top-down decoupling aggregation

xMemory adopts a multi-level memory hierarchy: themes → semantics → episodes → messages. Retrieval operates by (1) query-aware representative selection (RepSel) at sem/theme level, and (2) uncertainty-gated episode/message expansion (UncSion), yielding compact, non-redundant contexts for answer synthesis (Hu et al., 2 Feb 2026).

5. Quantitative and Qualitative Findings

Key comparative findings are as follows:

On LoCoMo (averaged across question categories), xMemory improves answer accuracy and efficiency compared to all baselines:
- Qwen3-8B: BLEU 34.48 (Nemori: 28.51), F1 43.98 (Nemori: 40.45), tokens/query 4,711 (−39% vs. best baseline).
- Llama-3.1-8B-Instruct: BLEU 24.73 (Nemori: 22.21), F1 34.77 (30.99), tokens/query 5,540 (−63%).
- GPT-5 nano: BLEU 38.71 (36.65), F1 50.00 (48.17), tokens/query 6,581 (−28%) (Hu et al., 2 Feb 2026).
On PerLTQA:
- Qwen3-8B: BLEU 36.24 (MemoryOS: 35.14), F1 47.08 (42.35), ROUGE-L 42.50 (38.48), tokens/q 5,087 (−22%).
- Llama-3.1-8B-Instruct: BLEU 42.68 (Nemori: 41.01), F1 52.37 (49.62), tokens/q 6,066 (−47%).
- GPT-5 nano: BLEU 36.79 (33.44), F1 46.23 (41.79), ROUGE-L 41.25 (38.43), tokens/q 7,307 (−38%).

Additional qualitative findings highlight persistent weaknesses in reasoning over long-range temporal and causal dependencies. LoCoMo reveals that LLMs, even with retrieval or long context, display major performance drops (e.g., temporal questions F1 ≈ 20–25 vs. human F1 = 87.9). Multi-hop evidence gathering is limited, with xMemory doubling multi-hit retrieval rates compared to RAG, yet gaps to human coherence and factuality remain (Maharana et al., 2024).

In PerLTQA, best performance is achieved via combined semantic and episodic memory retrieval, while episodic-only settings reach much higher accuracy than semantic-only. Incorrect or absent retrieval frequently results in factual hallucinations (e.g., mistaken occupation answers), emphasizing the necessity of robust recovery of memory anchors (Du et al., 2024).

6. Advances in Agent Memory: From RAG to Hierarchical Decoupling

Standard RAG, designed for unstructured, heterogeneous text corpora, fails to address the redundancy, topic overlap, and sequential dependencies of lifelong agent memory. Fixed-k similarity retrieval often returns near-duplicate segments, and naive pruning can break causal chains necessary for multi-step reasoning (Hu et al., 2 Feb 2026).

xMemory addresses this by explicitly disentangling dialogue/history into latent themes and semantic units, hierarchically organizing memories, and adaptively retrieving representative, uncertainty-reducing units. Ablations demonstrate:

Vanilla hierarchy + similarity yields notable gains (+3.89 BLEU, +4.35 F1 vs. RAG) but with high token usage.
Query-aware selection and uncertainty-based expansion further improve accuracy (+1.42 F1) while reducing retrieval redundancy.
Retroactive split/merge of semantic groups (≈45% reassignment) increases both compactness and answer quality, balancing speed and faithfulness, as predicted by information-theoretic bounds (Fano's inequality).

This progression demonstrates that structurally-aware memory operations are superior for bounded, agent-centric memory tasks.

7. Implications and Future Research Directions

LoCoMo and PerLTQA catalyze a systematic exploration of both fundamental and applied memory architectures for LLM agents:

These resources expose large, persistent drawbacks in LLM temporal, causal, and multi-hop reasoning despite improved retrieval, context length, and memory structuring.
They motivate research into retroactively structured, hierarchically-aggregated memory, combining decoupling at storage with uncertainty-driven aggregation at retrieval.
Future work outlined in PerLTQA targets expansion into real-world personal data, richer and multimodal memory modalities (e.g., sensory, procedural, image, audio), dynamic in-situ updating, and differentiable end-to-end retrieval/generation architectures.
Scaling both models and datasets, and exploring collaborative, multi-agent memory networks, remain open challenges (Du et al., 2024, Hu et al., 2 Feb 2026).

In summary, LoCoMo and PerLTQA represent state-of-the-art resources and methodologies for benchmarking, analyzing, and advancing agent memory in long-horizon, dialogue-rich, and personal knowledge-intensive domains. Their integration into new system designs—anchored by robust hierarchies and fine-grained retrieval—sets a standard for future research into agentic LLMs and lifelong reasoning (Du et al., 2024, Maharana et al., 2024, Hu et al., 2 Feb 2026).