PM-Bench: Evaluating Prospective Memory in LLM Agents

Abstract: A significant challenge in agentic AI is prospective memory: the ability to execute an intention at a specific future cue or state while other activities are ongoing. We introduce PM-Bench, a text-based benchmark for measuring prospective memory capabilities in modern LLM agents. Inspired by the Virtual Week paradigm from cognitive science, PM-Bench evaluates how well LLM agents maintain user intentions, execute delayed intentions, and monitor latent environment changes. Over the course of a simulated seven-day week, agents must continue an ongoing activity while deciding whether any deferred task is due. We compare eight state-of-the-art LLMs on PM-Bench under eight different agent configurations. PM-Bench proves challenging across all settings: the best method, a GPT-5.4 agent, reaches only 65.1\% F1 score under our evaluation. Furthermore, no single strategy for improving prospective memory dominates across models. We release PM-Bench as a controlled testbed for diagnosing these failures and developing training or inference-time interventions that support reliable prospective behavior.

• The paper introduces PM-Bench, a novel benchmark for assessing prospective memory and deferred intention execution in LLM agents.
• It employs a controlled seven-day simulation with time- and event-triggered tasks to examine agents’ monitoring, recall, and action selection.
• Empirical results reveal tradeoffs between precision and recall across different agent scaffolds, underscoring unsolved challenges in proactive monitoring.

PM-Bench: A Controlled Benchmark for Prospective Memory in LLM Agents

Motivation and Problem Definition

The PM-Bench benchmark addresses a consequential issue for agentic LLMs: prospective memory, i.e., the ability of an agent to reliably execute intentions at the correct future cue or time while engaging in ongoing activities. Unlike retrospective recall, prospective memory requires not only remembering what should be done but also tracking dynamic environmental cues, reschedulings, and latent changes—skills critical for real-world autonomy across domains such as aviation, healthcare, and software assistance.

Prior benchmarks for long-horizon LLMs concentrate on memory retrieval, context engineering, and multi-session dialogue, but they do not rigorously isolate the challenge of intention execution at the right future moment. PM-Bench, inspired by the Virtual Week paradigm in cognitive science, operationalizes this domain for LLM agents and introduces a synthetic, scalable, and controlled suite of tasks requiring deferred intention maintenance, event- and time-triggered execution, and proactive monitoring.

Figure 1: Overview of PM-Bench's seven-day schedule and task structure, showing deferred intentions intertwined with ongoing activity, exposing agentic memory and monitoring challenges.

Benchmark Design and Protocol

Scenario Construction

PM-Bench simulates a seven-day week with daily irregular schedules. Each day comprises 10–13 steps, where at each step the agent must select from ongoing activity options and decide whether any prospective actions (including deferred and updated tasks) are presently due. Tasks are defined by natural-language labels, trigger conditions (event-based or time-based), and executable actions. Agent-visible objects include narrative vignettes, activity choices, action menus of anonymized handles, and queryable state channels (e.g., clock, email, appointment portal).

The benchmark enforces non-leakage of information and reshuffles action handles to prevent memorization artifacts. Tasks can be regular routines, non-regular one-off intentions, cross-day deferred obligations, and state-channel-triggered cues, with randomized distractors (lures) to increase selection pressure.

Figure 2: PM-Bench's evaluation interface highlights narrative, activity choices, state channels, and anonymized action handles for task execution.

Monitoring and Solvability

PM-Bench embeds 11 query-based external state channels, making cues observable only upon explicit querying. Thus, success in PM-Bench demands not merely memory maintenance but also selective and proactive monitoring—a major driver of agentic failure. The benchmark’s generation pipeline uses deterministic sampling with strict validation checks for cue realizability, time alignment, update integrity, and dependency consistency. Solvability is audited via oracle execution; failures can be conclusively attributed to agent behavior.

Agent Configurations and Evaluation Framework

Scaffolds and Agent Setups

PM-Bench tests eight agentic setups across eight cutting-edge LLM backbones (GPT-5.4, GPT-5.3-Codex, Llama-3.3-70B, Mistral-Large, Mistral-Small, Qwen3 variants):

• Single-agent Baseline: No explicit memory scaffolding or prompting.
• Todo-ledger: Persistent in-context ledger for pending intentions.
• Heartbeat-Augmented: Periodic nudges for proactive channel querying, optionally agent-controlled.
• Hierarchical Multi-agent: Coordinator and specialists for monitoring and action proposal.
• Majority/Unanimous Voting: Action aggregation replay for evidence-based task selection.

  Figure 3: The four major evaluated agent setups with memory, monitoring, and action decomposition interventions.

Metrics

The principal metric is set-level F1, balancing precision and recall for prospective actions at each step:

$\mathrm{Set\mbox{-}F1} = \frac{2TP}{2TP + FP + FN}$

where $TP$ are correctly selected due actions, $FP$ are false positives, and $FN$ are missed due actions. Auxiliary metrics include cross-day hit rates, update-sensitive execution, monitoring-required hits, false alarms, and proactive query counts.

Empirical Results and Findings

No Universal Best Scaffold

PM-Bench reveals that no single agentic scaffold dominates all metrics. The optional-heartbeat method achieves the highest macro Set-F1 score ($65.1\%$), while the todo-ledger variant minimizes false positives and maximizes precision. Aggressive monitoring (e.g., hierarchical agents or frequent heartbeats) increases recall but at the cost of increased false positives, collapsing overall F1.

Notably, the best-performing method is model-dependent: different LLM backbones prefer different scaffolds, suggesting heterogeneous weaknesses in memory, monitoring, and action control (see Table 1 in the paper).

Monitoring and Proactive Behavior Remain Unsolved

Tasks requiring hidden-channel monitoring are the hardest by a substantial margin; clock-based monitoring achieves $>50\%$ hit rates only for the strongest setups, but non-clock proactive tasks rarely exceed $16.7\%$. Even scaffolds issuing orders-of-magnitude more queries do not guarantee high monitoring hit rates. The challenge is not simply frequency of monitoring—it is the strategic timing and correct action selection based on acquired evidence.

Figure 2: The PM-Bench interface exposes both channel and narrative cues, requiring agents to decide when to actively query for hidden signals.

Cross-Day and Update Tasks

Long-horizon tasks, including cross-day deferred intentions and tasks requiring dynamic update handling, remain unsolved. The best hit rates for these tasks do not exceed $50\%$, underscoring persistent limitations in maintaining commitments over extended trajectories with intervening distractions and events.

Implications and Prospects

PM-Bench demonstrates that prospective memory in LLM agents is orthogonal to context window size or simple memory retrieval. The observed precision-recall tradeoff implies that scaffold design must be adaptively matched to backbone-specific weaknesses. Proactive monitoring, especially for latent environmental changes, is a major unsolved axis that is critical for safe autonomy.

Practically, PM-Bench offers controlled diagnostics for intervention development: e.g., inference-time scaffolding, intent ledgering, proactive reminders, or hierarchical signal tracking. Theoretically, it introduces evaluation regimes that may drive research on memory-centric architectures, strategic monitoring modules, and agent behavior controllability.

As LLM-driven assistants are deployed in safety-critical settings, benchmarking prospective memory failures is imperative; PM-Bench provides an actionable standard for measuring and mitigating these risks without over-inflated claims of readiness.

Conclusion

PM-Bench offers a rigorous testbed for agentic LLM evaluation along the axis of prospective memory, advancing benchmarks beyond recall-centric paradigms. Current frontier LLMs exhibit significant gaps in prospective task execution, proactive monitoring, and long-horizon intention maintenance. These limitations, now quantifiable via PM-Bench, spotlight urgent research challenges that must be solved for robust, safe, and reliable agentic AI.

(2607.12385)