EcomBench: Towards Holistic Evaluation of Foundation Agents in E-commerce

Abstract: Foundation agents have rapidly advanced in their ability to reason and interact with real environments, making the evaluation of their core capabilities increasingly important. While many benchmarks have been developed to assess agent performance, most concentrate on academic settings or artificially designed scenarios while overlooking the challenges that arise in real applications. To address this issue, we focus on a highly practical real-world setting, the e-commerce domain, which involves a large volume of diverse user interactions, dynamic market conditions, and tasks directly tied to real decision-making processes. To this end, we introduce EcomBench, a holistic E-commerce Benchmark designed to evaluate agent performance in realistic e-commerce environments. EcomBench is built from genuine user demands embedded in leading global e-commerce ecosystems and is carefully curated and annotated through human experts to ensure clarity, accuracy, and domain relevance. It covers multiple task categories within e-commerce scenarios and defines three difficulty levels that evaluate agents on key capabilities such as deep information retrieval, multi-step reasoning, and cross-source knowledge integration. By grounding evaluation in real e-commerce contexts, EcomBench provides a rigorous and dynamic testbed for measuring the practical capabilities of agents in modern e-commerce.

• The paper presents EcomBench—a benchmark derived from authentic, expert-curated e-commerce tasks—to thoroughly evaluate foundation agents.
• It employs a level-based task stratification to highlight performance gaps in long-horizon reasoning and multi-tool integration in complex scenarios.
• Quantitative results reveal that while agents perform above 90% on simple tasks, their accuracy drops to below 46% on challenging, high-difficulty queries.

EcomBench: Holistic Evaluation of Foundation Agents in Realistic E-commerce Contexts

Motivation and Benchmark Design

EcomBench addresses critical deficiencies in agent evaluation by introducing a domain-grounded testbed for assessing foundation agents within the complex ecosystem of e-commerce. Existing agentic benchmarks are predominantly academic or synthetic, offering limited relevance for real-world deployment in domains characterized by dynamic market conditions, multifaceted user interactions, and operational intricacies. EcomBench is constructed on four design principles:

• Authenticity: All tasks are derived from genuine user demands harvested from major e-commerce platforms, ensuring the representation of real decision-making requirements and current market trends.
• Professionalism: Subject-matter experts curate and validate tasks via a human-in-the-loop pipeline, guaranteeing both technical depth and domain verifiability.
• Comprehensiveness: The benchmark encapsulates seven principal categories, including policy consulting, cost/pricing, logistics, strategy, product selection, opportunity discovery, and inventory management, spanning both multiple-choice and open-ended formats.
• Dynamism: EcomBench undergoes quarterly updates to align with prevailing regulatory, market, and user trends, and to maintain hardness as agent capabilities progress.

Task difficulty is rigorously stratified through a tool-hierarchy mechanism. Level-3 (high-difficulty) tasks are identified by leveraging LLMs equipped with specialized tools, ensuring coverage of long-horizon reasoning, multi-source knowledge integration, and adaptive tool usage.

Dataset Construction and Methodology

The benchmark pipeline begins with extraction of raw user intents from global e-commerce ecosystems. An initial filtering pass by LLMs discards ill-defined or overly subjective requests, producing seed questions with verifiable answers. Domain experts then refine, reformulate, and peer-review each question to enforce clarity and answer consistency. Preference for human-generated tasks over LLM-synthesized content circumvents the synthetic bias prevalent in previous datasets and enhances downstream agent alignment with authentic operational scenarios.

Difficulty stratification proceeds via rejection sampling on LLM-assisted tool runs. Only those queries requiring complex reasoning chains and multi-tool action sequences survive to the hardest tier. This process scales with the expansion of agent toolsets and directly addresses the rising need for agents capable of beyond-trivial information seeking and business logic formulation.

Benchmark Analysis: Task Taxonomy and Complexity

EcomBench encompasses seven distinct categories, ensuring a diverse operational spectrum. The majority of the dataset comprises Level-3 tasks (50%), which disproportionately stress contemporary agentic architectures by demanding precisely integrated, long-horizon reasoning and robust tool orchestration.

Quantitative Evaluation of Foundation Agents

A comprehensive empirical evaluation of thirteen leading foundation agents was carried out. Each model's response was adjudicated both automatically (via LLM judge model) and by expert inspection for semantic equivalence. Metrics reflect binary correctness against ground-truth answers.

Comparison on EcomBench (Figure 1) demonstrates marked performance differentials across models:

Figure 1: Performance comparison of foundation agents across the entire EcomBench suite, highlighting significant performance gaps and prevailing weaknesses on hard tasks.

The progressive decline as task hardness increases is evident: Level-1 accuracy for state-of-the-art models (e.g., ChatGPT-5.1, Gemini DeepResearch) exceeds 90%. However, Level-3 performance deteriorates sharply; leading models only reach 46%, and most agents fall below 35% (Figure 2). This substantiates the claim that current agent mastery is confined to simple cases, and generalization remains weak for operationally complex task compositions.

Figure 2: Accuracy of foundation agents segregated by EcomBench difficulty level, revealing precipitous drops on multi-step and tool-integrated questions.

Domain-wise evaluation exhibits persistent gaps in category specialization. Finance-Related and Strategy-Related domains induce pronounced performance variance, with agents exhibiting distinct strengths per category but failing to achieve uniform competence (Figure 3). For example, SuperGrok outperforms others on finance tasks, while Gemini DeepResearch excels at strategy, indicating incomplete generalization in domain-specific reasoning.

Figure 3: Comparative performance of leading agents across major e-commerce task categories, evidencing domain specialization and lack of robust generalist capabilities.

Theoretical and Practical Implications

EcomBench demonstrates that progress in agentic reasoning within general LLMs does not translate into operational robustness for high-stakes, real-world e-commerce challenges. The observable accuracy gap and domain-specific shortcomings highlight several implications:

• Agentic Infrastructure Limits: Even with retrieval or tool augmentation, current agents lack long-horizon compositional reasoning and consistent toolchain integration.
• Benchmark Evolution: Dynamic updating of EcomBench ensures that evaluation keeps pace with rapidly improving agent architectures and shifts in regulatory and market environments. This closes the feedback loop, preventing benchmark obsolescence and data contamination.
• Specialization vs. Generalization: The empirical evidence suggests that specialization via domain-adaptive training can outperform broad agentic methods in specific task categories. Research into multi-domain transfer, tool synergy, and adaptive memory mechanisms may be required for true generalist agent deployment.
• Dataset Construction Bottlenecks: Reliance on human experts for question design, validation, and maintenance imposes a practical constraint on scalability. Future efforts may introduce more sophisticated LLM-human collaborative pipelines or formalized question synthesis via structured information retrieval models.

Future Prospects

Expansion of EcomBench will target interactive simulation environments, predictive tasks (e.g., trend analysis, forecasting), and broader analytical workloads, further increasing the discriminative capacity and operational relevance of the benchmark. Expected developments in RL-based agent training, emergent tool ecosystems, and foundation model architecture may gradually bridge the observed performance gap. However, the requirement for continuous benchmark evolution and domain-aligned evaluation frameworks is paramount for tracking genuine agentic intelligence in commerce.

Conclusion

EcomBench represents a robust, multifaceted challenge for evaluating foundation agents in realistic e-commerce environments. Grounded in authentic user requirements, curated by experts, and dynamically maintained, it exposes critical limitations of current agentic frameworks—especially in reasoning, tool usage, and domain adaptability. Ongoing expansion and iterative update cycles will ensure EcomBench remains an essential resource for systematic measurement and advancement of agentic AI in commerce (2512.08868).