UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks

Abstract: The rapid development of LLMs and multimodal LLMs has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures. To address these limitations, we introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. UniClawBench is built around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Based on these capabilities, we design 400 bilingual real-world tasks. Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints. Furthermore, we design a closed-loop evaluation strategy comprising an executor agent, a hidden supervisor agent, and a user agent to simulate realistic multi-turn human feedback without leaking grading criteria. To disentangle base model capabilities from framework-level design choices, we evaluate state-of-the-art models under multiple agent frameworks. Through comprehensive comparisons across both models and frameworks, we show how base model capabilities and agent framework designs jointly shape performance in real-world environments. To facilitate future research, we make our benchmark and code publicly available at https://github.com/HKU-MMLab/UniClawBench.

• The paper introduces a novel benchmark that decomposes agent capabilities into five distinct dimensions for detailed evaluation.
• It employs a three-role closed-loop evaluation system (executor, supervisor, user simulator) to enhance multi-turn error recovery in real-world settings.
• Experimental results highlight challenges in long-context, multimodal, and cross-platform tasks, emphasizing the critical role of framework architecture.

Capability-Driven Benchmarking of Proactive Agents: An Analysis of UniClawBench

Motivation and Benchmark Design

UniClawBench ("UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks" (2607.08768)) introduces a rigorous capability-driven evaluation framework tailored for proactive AI agents operating in live, heterogeneous environments. Prior benchmarks predominantly leveraged static sandboxes and scenario-based taxonomies, obfuscating failure modes and lacking diagnostic resolution on individual model competencies. UniClawBench addresses these deficiencies by decomposing agent capabilities into five orthogonal dimensions—Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination—and constructing 400 bilingual tasks, each with explicit capability bottlenecks and extensive coverage of application domains, input/output modalities, and real-world workflows.

Key architectural innovations include:

• Three-role closed-loop evaluation: Distinct executor, supervisor, and user simulator agents enable multi-turn interaction and error recovery, while maintaining strict isolation of hidden rubrics and ground truth, preventing information leakage into agent-facing prompts.
• Dynamic execution environment: Each task is instantiated in a Docker container with live browsers, native apps, and injected skills, eliminating dependence on static answers or pre-recorded artifacts.
• Fine-grained checkpoint scoring: Stepwise evaluation traces intermediate progress, capturing partial completions and supporting recovery cycles in alignment with real-world user-agent interaction patterns.

Capability Taxonomy

The benchmark categorizes tasks based on primary capability bottlenecks, providing diagnostic separation:

• Skill Usage: Proficiency in tool/API invocation and artifact generation, requiring explicit interface consultation.
• Exploration: Open-ended search and audit tasks, stressing evidence gathering, negative proof, and source validation amidst noisy or misleading information.
• Long-Context Reasoning: Synthesis operations over extended evidence chains, consistency maintenance, and memory persistence.
• Multimodal Understanding: Extraction and reasoning grounded in images, videos, audio; correct output requires non-textual evidence, not just inferred content.
• Cross-Platform Coordination: State synchronization and artifact orchestration across browsers, local filesystems, desktop apps, and databases.

Tasks are manually curated to maximize realism and diversity, assigning each scenario to its dominant capability and ensuring auxiliary operations arise only if needed, minimizing confounding factors.

Evaluation Protocol

Instead of static answer matching, UniClawBench employs a closed-loop, checkpoint-based protocol:

• Supervisor agent: Applies hidden rubric, scoring each checkpoint rigorously only against explicit requirements; rationale, missing artifact lists, and guidance tags remain internal.
• User simulator: Receives only coarse progress signals; synthesizes feedback on visible trajectory, emulating real user turns without privileged information.
• Executor agent: Operates within Docker, accessing only permitted skills, sources, and tools; evolves the task state across up to two follow-up cycles.

Metrics include Pass Rate (PR, percentage of successfully completed tasks) and Average Score (AS, supervisor-assigned normalized completion).

Experimental Results

Cross-Model Evaluation

The evaluation (10 SOTA models under OpenClaw framework) reveals:

• Best-performing models (Claude Opus-4.8, GPT-5.4) achieve PR < 50%—the benchmark is exceptionally difficult and exposes substantial gaps between sandboxed and real-world execution.
• Open-source models (Qwen-3.5-Plus, Kimi-2.6) are competitive, at times surpassing closed-source alternatives (Gemini-3.1-Pro).
• There is a pronounced intermediate-progress failure pattern: high AS but low PR, indicating agents reliably reach partial completions yet fail on extended trajectory reliability.
• Bottlenecks are strongest in Long-Context, Multimodal, and Cross-Platform tasks; local tool operation and information seeking (Skill Usage, Exploration) are relatively tractable.

Cross-Framework Evaluation

Benchmarking three models across multiple frameworks (OpenClaw, Nanobot, EDICT) demonstrates:

• Framework architecture exerts greater influence than model choice—structural context management and orchestration are critical.
• OpenClaw (centralized context, minimal information loss) yields highest PR.
• EDICT (multi-agent orchestration) suffers coordination friction, with sub-agent context loss and token inefficiency, especially on long-context tasks.
• Nanobot (ultra-lightweight design) achieves superior token efficiency but at the expense of holistic reasoning and evidence chain continuity; partial progress is common, full completion rare.

Token usage correlates tightly with task complexity and capability weaknesses, peaking for long-context and multimodal dimensions.

Multi-Turn Interaction

Agentic performance improves across cycles, affirming the necessity of multi-turn protocols for error recovery—strict single-turn benchmarks are insufficient for realistic agent evaluation.

Evaluation Reliability

Correlation with human expert judgments remains high: supervisor decisions attain 92% agreement (binary pass/fail), and checkpoint AS correlates strongly with averaged human scores (Pearson r = 0.71, Spearman p = 0.68).

Implications for Agent Research and Development

UniClawBench shifts standard practice toward capability-driven agent diagnostics and unified framework-model co-design:

• Diagnostic separation: Explicit attribution of failure to model vs. framework architecture enables targeted research, whether in memory augmentation, tool-use precision, cross-modal reasoning, or stateful coordination.
• Framework-centric evaluation: Agent orchestration, context preservation, and closed-loop feedback are as pivotal as core model design; future agent wrappers must optimize for trajectory retention, progressive state, and artifact management.
• Multimodal and long-context resilience: Persistent bottlenecks in these dimensions underscore the need for tailored augmentation strategies, e.g., hierarchical memory modules, context chunking, advanced multi-modal fusion.
• Benchmark-driven training and adaptation: Although UniClawBench advances evaluation fidelity, it also risks inducing overfitting to benchmark archetypes; generalization and transfer to unseen workflows require vigilant attention to avoid shortcut learning.

Limitations and Future Directions

• Task set size and curation: 400 manually crafted tasks, while diverse, may be insufficient for broad coverage; scaling to larger, more varied real-world workflows is necessary.
• Dynamic execution instability: Live-environment artifacts may capture shifting web content, data drift, or infrastructure errors, challenging reproducibility.
• LLM-eval bias: Reliance on LLM supervisors and user simulators may introduce subtle grading artifacts or drift; hybrid approaches with human-in-the-loop verification may enhance reliability.

Future developments should prioritize:

• Expansion of task coverage and scenario realism.
• Enhanced orchestration frameworks for robust context management.
• Research into adaptive, error-resilient memory structures and multimodal perception.
• Continued diagnostic benchmarking to drive advances in proactive, "digital coworker" agents.

Conclusion

UniClawBench constitutes a significant advancement in capability-driven benchmarking of proactive agents. Its architectural separation of model and framework contributions, fine-grained closed-loop evaluation, and realistic task curation illuminate nuanced failure modes and reliability gaps previously unaddressed. The results robustly demonstrate that agent frameworks, not just underlying models, are the primary determinants of success in live, real-world task execution, particularly concerning cross-modal, extended-context, and multi-platform workflows. Adopting such diagnostic methodologies is essential for the principled development of next-generation, persistent, proactive assistants.