AgenticDataBench: A Comprehensive Benchmark for Data Agents

Abstract: Data science aims to derive actionable insights from heterogeneous raw data, unlocking the value of the massive amounts of data generated in modern society. Automating this process is essential to reducing labor-intensive efforts for data scientists and enabling scalable data-driven applications. Recently, LLM-based data agents have emerged as a promising solution to automate data science workflows. However, the field lacks comprehensive benchmarks to rigorously evaluate these agents across diverse scenarios with fine-grained granularity. To address this gap, we propose AgenticDataBench, a comprehensive benchmark featuring realistic tasks spanning diverse domains with fine-grained ground-truth labels. This enables evaluations to capture the diversity and complexity of data science workflows and the detailed performance of agents. First, to cover diverse domains, we collect real datasets and tasks from 15 vertical domains, including 5 real-world B2B use cases from a leading fintech company. Second, to remove redundancy in real-world tasks and generate high-quality tasks for domains lacking real data, we introduce data science skills, recurring data-centric operational patterns, and quantify benchmark coverage by the number of skills included. Representative skills are extracted from large-scale task solutions on Stack Overflow using skill-aligned hierarchical clustering. Third, for real-world business tasks, we select task-solution pairs that maximize diversity in skill composition, ensuring broad coverage of practical scenarios. Fourth, to generate realistic tasks for devise domains without real tasks, we propose a systematic LLM-based task generation approach to create workflows and tasks based on these skills. Finally, we evaluate state-of-the-art data agents using our annotated benchmark and open-sourced testbed, providing detailed skill-level insights.

• The paper introduces a comprehensive benchmark that systematically quantifies data science skills using a hierarchical set of 433 skills derived from real-world sources.
• It employs an LLM-based evaluation pipeline spanning 15 diverse domains to analyze performance trade-offs, precision, and computational costs.
• The evaluation reveals significant variances in agent performance, emphasizing the need for skill-driven memory, debugging, and adaptive execution strategies.

AgenticDataBench: A Comprehensive Skill-Level Benchmark for Data Science Agents

Motivation and Benchmark Design

The automation of data science workflows through LLM-based agents introduces substantial potential for increasing productivity in data-driven domains. However, robust evaluation across diverse real-world scenarios and with fine-grained failure analysis remains a significant gap. "AgenticDataBench: A Comprehensive Benchmark for Data Agents" (2607.01647) addresses this by constructing a benchmark that systematically emphasizes diversity, complexity, and skill traceability. The benchmark’s instances are derived from realistic tasks—with granular ground-truth labels—spanning 15 vertical domains, including data from industry production environments and public datasets.

The central innovation is the formalization and extraction of a comprehensive, hierarchical set of 433 data science skills. These skills, mined from Stack Overflow and curated business solutions, are used to annotate benchmark tasks. The benchmark curation process systematically samples, de-duplicates, and composes task instances to maximize both domain and skill coverage, supporting a detailed breakdown of agent capabilities and limitations.

Pipeline and Dataset Construction

The workflow for constructing the benchmark (Figure 1; see also Figure 2 for pipeline overview) integrates data collection, skill discovery, task generation, and fine-grained evaluation:

Figure 1: The systematic process for constructing AgenticDataBench, including hierarchical skill extraction and skill-coverage-driven task selection/generation.

Figure 2: The test pipeline includes Benchmark Instances, Data Agent, Executor, and Leaderboard to enable reproducible, fine-grained agent scoring.

Skill extraction uses LLM-based decomposition and embedding-driven hierarchical clustering to produce a multi-level taxonomy that captures both broad data operations and fine-grained actionable patterns. This is crucial for benchmarking realistic workflow decomposition and agent debugging.

To reduce redundancy, tasks from business domains are curated with a greedy coverage-maximization strategy. For public datasets lacking task diversity, an LLM-based task generator composes workflows by sampling frequency-weighted skill paths from the skill graph, guided by detailed profiling of data formats and cross-file relationships.

Figure 3: Visualization of the 433 extracted data science skills, highlighting the granularity and coverage of operational patterns.

Skill categories are distributed across seven axes (Figure 4): Data Format Handling, Preprocessing, Manipulation, Analysis, Modeling, Visualization, and Cross-Stage Skills. The benchmark instances span significant heterogeneity in data source, size, and domain, with substantial per-task footprint (average 493.4MB of data and 23.5 distinct skill applications per task).

Figure 4: Distribution of skill categories across diverse domains, indicating the distinct operational demands per use case.

Evaluation Protocol and Agent Analysis

AgenticDataBench integrates a multi-faceted evaluation pipeline capable of holistic performance and skill-level error attribution. Each benchmark instance includes: a task description, associated datasets, an executable ground-truth solution, explicit skill requirements, and a domain-specific evaluation function (supporting modalities such as table, JSON, chart, and modeling-based scoring).

State-of-the-art data agents—including generic and data-science-specialized implementations—are evaluated by integrating multiple LLMs (Qwen3.5, Kimi-K2.5, Claude 4.6) and harnesses (DA-Agent, Smolagents, Claude Code, CodeX). The leaderboard ranks agents by overall task accuracy and skill-level consistency.

Numerically, the highest-scoring setup (CodeX + Kimi-K2.5) achieves the best aggregate accuracy, but at substantial token and computational cost—general-purpose harnesses systematically outperform lightweight baselines in accuracy but not in efficiency. Importantly, cost-performance trade-offs vary, with Smolagents (Qwen3.5) and DA-Agent (Qwen3.5) achieving favorable efficiency despite slightly lower accuracy (Figure 5).

Figure 5: Trade-off between mean agent score and token cost, demonstrating divergent cost-accuracy profiles across harnesses and LLMs.

Skill-level analysis highlights that agent strengths are not uniform across domains or skill categories (Figure 6). For example, Data Format Handling and Cross-Stage Skills are consistently easier for all agents; Data Analysis and Data Modeling skills, especially those requiring nuanced reasoning or cross-file integration, remain major error sources.

Figure 6: Skill score distributions by category and agent system.

Deeper skill-wise breakdowns reveal domain-specific deficiencies: failures in “Data Alignment & Merging” and “Text Processing & Cleaning” are frequent, particularly when typical pretraining does not generalize well to highly heterogeneous or non-relational data formats. The strengths and weaknesses of leading agents differ substantially (Figure 7), highlighting coupling between harness design and LLM capabilities.

Figure 7: Skill-level performance profiles of top agents, indicating divergent strengths and weaknesses in both general and domain-specific skill applications.

Failure analysis (Figure 8) further distinguishes error sources, attributing a major share to either Data Analysis or global/step-level timeouts. Notably, extending execution budgets rarely ameliorates core deficiencies, as failures often result from reasoning or harness-LLM misalignment rather than insufficient per-task resources.

Figure 8: Distribution of failure modes across agent systems, with major contributions from Data Analysis and harness–LLM execution mismatches.

Implications and Future Directions

AgenticDataBench defines a new state-of-the-art in agent evaluation for automated data science, emphasizing systematic, reproducible, and skill-level diagnosis. Its insights have several key implications:

• Agent architectures require explicit integration of skill-driven memory, debugging, and adaptive execution strategies to address persistent failures in reasoning-intensive and cross-modal task compositions. The current best performance still underachieves on complex, multi-skill, or high-dimensional modeling tasks.
• LLM-harness compatibility is critical: agent performance is non-monotonic with respect to LLM capabilities and is sensitive to how the harness interfaces with execution, memory, and feedback. Highly capable LLMs may be penalized if the agent harness is not tightly aligned to their output or tool usage conventions.
• Skill-level benchmarks should become standard for evaluating agentic AI in specialized domains, providing transparent diagnostics that guide model, harness, and dataset co-evolution.

Looking ahead, the challenge persists in improving agent robustness under data heterogeneity, reducing reliance on ad hoc code generation, and developing systems that can dynamically acquire and refine new skills. Further, future developments in agentic AI should incorporate meta-learning, explicit error recovery, and learnable orchestration layers, particularly as skill hierarchies and cross-domain transfer become more central [DBLP:conf/iclr/00010GA25, DBLP:conf/iclr/Yu0GBGA24].

Conclusion

AgenticDataBench (2607.01647) sets a new benchmark paradigm for evaluating autonomous data science agents, centering evaluation on skill-level analysis and domain diversity. The integrative approach—combining hierarchical skill extraction, skill-optimized task sampling, and a comprehensive analysis pipeline—enables detailed empirical study of agent capabilities. While current systems remain far from the optimum in dynamic, cross-modal data science automation, this benchmark provides both the granularity and diversity necessary to drive future advances in agentic AI and skill-centered evaluation methodologies.