Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants

Abstract: Proactive agents that anticipate user needs and autonomously execute tasks hold great promise as digital assistants, yet the lack of realistic user simulation frameworks hinders their development. Existing approaches model apps as flat tool-calling APIs, failing to capture the stateful and sequential nature of user interaction in digital environments and making realistic user simulation infeasible. We introduce Proactive Agent Research Environment (Pare), a framework for building and evaluating proactive agents in digital environments. Pare models applications as finite state machines with stateful navigation and state-dependent action space for the user simulator, enabling active user simulation. Building on this foundation, we present Pare-Bench, a benchmark of 143 diverse tasks spanning communication, productivity, scheduling, and lifestyle apps, designed to test context observation, goal inference, intervention timing, and multi-app orchestration.

• The paper introduces PARE, a formal evaluation framework that assesses proactive assistants via state-driven user simulations in multi-app environments.
• It models interactions as a Stackelberg POMDP, capturing realistic mobile UI constraints and enforcing controlled, user-approved interventions.
• Evaluation on 143 scenarios shows that large LLMs achieve around 40% success, highlighting persistent challenges in proactive digital assistant deployment.

Proactive Agent Research Environment: A Formal Evaluation Framework for Proactive Assistants

Introduction

The paper "Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants" (2604.00842) addresses a key problem in evaluating LLM (LM)–powered digital assistants: the inadequacy of current evaluation paradigms for proactive assistants. Most existing research platforms only support reactive agents responding to explicit human requests, relying on passively simulated users. To overcome this, the paper introduces the Proactive Agent Research Environment (PARE), a simulation framework that enables robust, ecologically valid evaluation of proactive agents via LM-based user simulators interacting in a state- and event-driven, multi-app digital environment.

By formalizing user and assistant interaction as an asymmetric Stackelberg POMDP, PARE captures essential properties of mobile user interfaces: restricted, screen-by-screen user navigation (modeled as per-app FSMs), as opposed to unrestricted backend API access for the assistant. This foundation allows researchers to study complete agent–user interaction loops, including goal inference, proposal, verification, and execution, while modeling environment events, system notifications, and tool failures. The framework includes a large-scale, procedurally generated benchmark (PARE-Bench) with 143 multi-app scenarios, supporting systematic evaluation of various LLM-based proactive assistant architectures.

Figure 1: The PARE framework comprises an event-based environment with FSM-modeled app transitions; users navigate stateful interfaces while assistants interact via flat backend APIs.

Related Work

The paper thoroughly situates PARE in the context of recent agent benchmarks and user simulators. Frameworks such as Generative Agents [park2023generativeagentsinteractivesimulacra], $\tau$-bench [yao2024taubenchbenchmarktoolagentuserinteraction], and ToolSandbox [luToolSandboxStatefulConversational2025] have demonstrated that LLMs can simulate plausible human actions in interactive environments. However, these works have pivotal limitations: either restricting users to passive roles, omitting grounded state-altering action execution, or failing to capture realistic navigation constraints and event-driven flows seen in mobile devices.

Furthermore, previous work on LLM-based proactive agents (e.g., ProactiveAgent [luProactiveAgentShifting2024], ContextAgent [yangContextAgentContextAwareProactive2025]) evaluates models only via offline datasets or through abstract textual simulation, lacking state-aware user interaction and unable to measure real goal completion outcomes. PARE addresses these gaps by enforcing environment dynamics that mirror actual software constraints, supporting both agent and user actions that are tightly coupled to the actual app states and event streams.

Framework Architecture

PARE extends the Agent Research Environment (ARE) [froger2025arescalingagentenvironments] by incorporating active user simulators as first-class agents. A pivotal feature is the asymmetric action and observation space: the user is forced to operate via explicit FSM transitions that correspond to real application screens and allow only limited action sets at each step, while the assistant observes all user actions and notifications and can execute flat APIs for any app (Figure 2).

Figure 2: Comparison between non-FSM agents with direct API access and FSM-based user simulation requiring realistic navigation and screen transitions.

This design is instantiated across a portfolio of mobile apps (e.g., Email, Messaging, Calendar, Shopping), each with detailed FSMs specifying screen-level state and legal user transitions. For example, sending a message as a user requires navigating through several UI states (open app → search → compose), while the assistant can immediately issue the API call.

User and assistant communicate and negotiate tasks through a shared message interface (PASAgentUserInterface), enforcing a strict protocol: the assistant can only execute state-altering actions with explicit user acceptance, respecting autonomy and providing controlled evaluation of intervention quality.

Benchmark and Scenario Generation

To enable systematic evaluation at scale, the authors introduce PARE-Bench, a benchmark of 143 scenarios spanning communication, scheduling, productivity, and lifestyle tasks. Each scenario is auto-generated in four pipeline stages: scenario description, application state/data seeding, event flow specification, and validation. A scenario is only admitted after passing execution-based verification, which ensures it is both syntactically and semantically fit (Figure 3).

Figure 3: The LLM-based scenario generation pipeline for PARE: description, state seeding, event flow construction, and validation with oracle-based feedback and deduplication.

Each scenario specifies initial and terminal states, a timed sequence of exogenous events (e.g., new emails, notifications), and an oracle for automated verification of goal completion. PARE supports the injection of tool failures and notification noise for robustness analysis, allowing for controlled ablations and stress testing of agents.

User Simulator and Proactive Assistant Agents

The user simulator is a strict, context-sensitive ReAct-based agent, receiving the current app, state, and available actions each turn. It must navigate FSM-defined screens, process truncated notifications (mirroring mobile OS behaviors), and respond to assistant proposals with high selectivity—as specified in rigorous prompting.

The proactive assistant is implemented as a two-phase observer–executor: it alternately monitors user actions/environment events, proposes concrete plans when sufficiently confident, and, upon user acceptance, transitions to an executor subagent with full API access for multi-app task completion. Critically, no action is taken without user approval. All action/opinion cycles are formalized with a thought–action–observation pattern.

Figure 4: Sequence diagram showing a proactive scenario: assistant observes event → infers constraint → proposes plan to user → executes only after user confirmation.

Experimental Evaluation

Experimental Configuration

Seven LLMs are evaluated: four frontier models (Claude 4.5 Sonnet, GPT-5, Gemini 3 Pro/Flash) and three compact open-weight models (Qwen 3 4B, Llama 3.2 3B, Gemma 3 4B). The main metrics are:

• Task Success Rate ($R_\textrm{Succeed}$): Fraction of user goals completed.
• Proposal Acceptance Rate ($R_\textrm{Accept}$): Fraction of assistant proposals accepted.
• Proposal Rate: Proposals per turn (lower is better).
• Read Actions: Number of information-gathering queries per scenario.

Frontier models achieve markedly higher success (Gemini 3 Flash: 42.1%, Claude: 42.0%)—but still only complete about 40% of proactive tasks, indicating substantial headroom for future work. Compact models underperform and exhibit elevated inconsistency and passivity.

Robustness to Failures and Noise

The framework supports stress tests under tool failures and event noise:

Figure 5: Success, proposal, and acceptance rates under increasing tool failure probability; larger models are robust, while small models degrade rapidly.

Figure 6: Evaluation with increasing notification noise; top models maintain proposal quality, whereas smaller models’ success drops sharply.

Across these perturbations, only Claude maintains stable success under 40% tool failure and high noise; smaller models both misreact and overreact (high false positives or missed opportunities). Model ablations show that large model quality differences manifest as substantial proposal selectivity/precision, not merely in generation fluency.

Theoretical Model

PARE formalizes user–agent joint action as a Stackelberg POMDP: the user (leader) acts first, modifying the observable global state, followed by the assistant (follower), proposing plans or gathering information with full observability over user action but not vice versa. Success is thus a function of user/assistant policy composition, proposal timing, and user selectivity—a configuration uniquely captured in PARE and missing in prior paradigms.

Key Findings

1. Action Asymmetry: Modeling the user as a state-constrained actor (FSM per app, truncated observation) while granting the assistant direct access to backend APIs enables realistic evaluation of intervention and user experience, capturing distinctions missed in pure API call–based environments.
2. Intervention Quality and Timing: Proposal timing is critical. Large models succeed via higher acceptance and fewer, more precise, context-aware interventions. Weaker models either intervene prematurely or fail to act at all.
3. User–Model Interplay: Systematic user simulation ablation shows that PARE preserves ranking stability across varying user model personalities (strict, permissive), but absolute success rates are sensitive to user idiosyncrasies, emphasizing the need for careful protocol specification in benchmarking.
4. Robustness: Only state-of-the-art LLMs maintain performance under significant simulated error/noise; open compact models (Qwen 4B, Gemma 3/4B) frequently fail to distinguish relevant signals, confirming an open research challenge in small-model proactive assistant deployment.
5. Observe–Execute Decomposition: Decoupling proactive task proposal (observation/decision) from execution is essential for effective, privacy-preserving assistant deployment, allowing on-device models to perform continuous monitoring with cloud execution only upon explicit user consent.

Implications and Future Directions

PARE marks a significant methodological advance for agent-oriented evaluation. Practically, it enables the development of privacy-preserving, autonomous, yet user-aligned digital assistants, supporting edge deployment and systematic safety auditing. The API-level FSM abstraction maintains a natural privacy boundary (agents "observe" only actions, not full screen content).

Theoretically, its POMDP formulation opens research directions in RL-based policy optimization for Stackelberg settings, goal inference from state/action trajectories, realistic modeling of mixed-initiative human–agent workflows, and systematic analysis of the interplay between agent intervention, user trust, and long-term adoption.

A current limitation is the absence of direct visual grounding; extending the framework toward UI screenshot–driven multimodal interaction (e.g., [zhang2023appagentmultimodalagentssmartphone, hong2024cogagentvisuallanguagemodel]) would bridge the gap to true end-user deployment. Further, deeper modeling of user diversity and theory-of-mind (e.g., [zhou2026tomsweusermentalmodeling]) would increase generalizability.

Conclusion

This work sets a rigorous new bar for proactive assistant evaluation by introducing an environment and benchmark that operationalize realistic user and agent capabilities, contextually rich scenarios, robust execution and evaluation pipelines, and extensive diagnostic metrics. The persistent success gap between large and small models, and the finding that execution capability (not intent inference) is the principal bottleneck for smaller models, indicates substantial open challenges and opportunities in agent architecture design, user–agent theory, and scalable, privacy-aware deployment of next-generation digital assistants.