Can Agentic AI Match the Performance of Human Data Scientists?

Abstract: Data science plays a critical role in transforming complex data into actionable insights across numerous domains. Recent developments in LLMs have significantly automated data science workflows, but a fundamental question persists: Can these agentic AI systems truly match the performance of human data scientists who routinely leverage domain-specific knowledge? We explore this question by designing a prediction task where a crucial latent variable is hidden in relevant image data instead of tabular features. As a result, agentic AI that generates generic codes for modeling tabular data cannot perform well, while human experts could identify the important hidden variable using domain knowledge. We demonstrate this idea with a synthetic dataset for property insurance. Our experiments show that agentic AI that relies on generic analytics workflow falls short of methods that use domain-specific insights. This highlights a key limitation of the current agentic AI for data science and underscores the need for future research to develop agentic AI systems that can better recognize and incorporate domain knowledge.

• The paper demonstrates that human data scientists using multimodal inputs achieve significantly higher predictive performance than agentic AI relying solely on tabular data.
• The methodology employs a synthetic insurance dataset with a hidden latent variable, Roof Health, encoded through detailed aerial roof images to test cross-modal inference.
• The findings reveal that current agentic AI pipelines are bottlenecked by their inability to extract embedded domain knowledge, underscoring the need for advanced multimodal reasoning.

Evaluation of Agentic AI Versus Human Data Scientists: Predictive Modeling with Hidden Latent Variables

Introduction

This paper addresses whether current agentic AI systems, exemplified by LLM-driven analytics agents, can match the predictive performance of expert human data scientists—especially in environments where crucial domain knowledge is embedded in modalities outside the primary tabular data. The authors construct a synthetic property insurance dataset in which a critical latent variable (Roof Health) is omitted from tabular features and only visually encoded within associated roof images. This design exposes the limitations of agentic AI strategies that rely on generic analytics code ignoring such domain-specific cues, and quantifies the empirical performance gap relative to human workflows that exploit cross-modal knowledge integration.

Synthetic Dataset Design and Data Generative Process

The authors generate data instances in three stages: (1) structured policy features are sampled using standard stochastic models (log-normal for house value, Beta for house age and risk, Bernoulli for wall type, empirical distribution for credit score); (2) a key latent variable, Roof Health, is deterministically computed from a nonlinear combination of released features, then categorized as Good, Fair, or Bad at adaptive quantile thresholds; (3) for each policy, a synthetic 1024×1024 aerial roof image is created via detailed text-to-image prompts to encode the designated Roof Health state through style, color, surface, edge, and debris descriptors.

Figure 1: Synthetic roof images faithfully encode the latent variable RoofHealth (Good, Fair, Bad), which is inaccessible from tabular data but recoverable from domain-specific visual cues.

The outcome target, next-year total loss $Y_p$, is simulated using a compound model with a negative binomial for frequency conditioned on both structured features and the latent roof health, and a gamma model for severity, again modulated by roof state. Importantly, the only viable pathway for recovering the true risk determinant (Roof Health) is via image modality reasoning—a scenario where human practitioners systematically outperform naive analytical pipelines.

Figure 2: Data generation schema for property insurance, where critical latent variables (dotted boxes) are omitted from features and modulate both claim frequency and severity.

Empirical Evaluation: Agentic AI Pipelines Versus Human Data Scientists

To operationalize the core comparison, the authors establish three experimental tiers: (1) a standard agentic AI workflow where models are trained solely on tabular data via generic code generation (simulating current LLM agentic pipelines as in (Li et al., 2024) and (Jiang et al., 18 Feb 2025)); (2) human data scientist approaches leveraging multimodal data, including image embeddings extraction (CLIP-vectorization with direct or clustered usage), VLM-based Roof Health annotation (using gpt-4o-mini), and perfect image-based labeling (oracle-level human expertise); (3) a true oracle, granted access to the underlying generative equations and latent state.

Key performance is quantified using the normalized Gini coefficient ($G_{\mathrm{norm}}$), a robust, distribution-agnostic ranking metric standard in insurance analytics.

Figure 3: Comparison of data scientist and agentic AI approaches—a 2.2× normalized Gini performance gap is observable when domain knowledge is requisite for loss prediction.

The results show that:

• Agentic AI (tabular-only): $G_{\mathrm{norm}} = 0.3823$ (minimal signal capture, fails to exploit essential information in images);
• Data Scientist Approaches:
  • CLIP clustering: $G_{\mathrm{norm}} = 0.5042$ (modest gain if mapping is poor),
  • Full CLIP embeddings: $G_{\mathrm{norm}} = 0.7719$ (substantially improved),
  • Vision-language-model (gpt-4o-mini) roof health extraction: $G_{\mathrm{norm}} = 0.7271$ (high correlation, efficient cross-modal reasoning),
  • Perfect roof health labeling: $G_{\mathrm{norm}} = 0.8310$ (near theoretical optimum);
• Oracle: $G_{\mathrm{norm}} = 0.8379$ (Bayes-optimal, upper bound).

The results underscore the strong dependence of achievable predictive accuracy on the model’s ability to recover non-tabular, latent semantic variables—a faculty fundamentally tied to human domain interpretability and currently not emulable by generic agentic AI code pipelines.

Implications for Agentic AI Design and the Future of Automated Data Science

These findings formally demonstrate a crucial limitation of extant agentic AI systems for end-to-end data science automation: they are inherently bottlenecked by their inability to reason over, identify, and incorporate hidden or latent factors accessible through domain knowledge or cross-modal inference. The empirical gap between agentic AI and optimal human approaches, exceeding 0.44 in normalized Gini, is significant in practical insurance and risk assessment settings.

Notably, modest improvements from multimodal embeddings and recent VLMs suggest partial addressal but not closure of the performance gap. The optimal (oracle) limit is only attainable with complete access to both generative mechanisms and latent state, a scenario unreachable by simple code synthesis or typical LLM agents constrained to tabular workflows.

The results advocate for several avenues for advancing agentic AI in complex, real-world data workflows:

• Multi-hop, multimodal reasoning components, effectively fusing structured and unstructured (visual/textual) information.
• Active domain knowledge elicitation, hypothesis testing, and cross-modal verification steps within agentic pipelines.
• Systematic benchmarks and datasets with hidden critical variables to robustly quantify gains from enhanced reasoning and domain cognition.

Conclusion

The study offers a rigorous, synthetic, but realistic analysis of where current agentic AI falls short compared to expert human data scientists. Strong numerical evidence supports the claim that reliance on generic code generation is inadequate in scenarios requiring identification and integration of latent, domain-encoded variables. This finding motivates architectural advances in agentic AI toward multimodal reasoning, active information seeking, and explicit domain knowledge incorporation as prerequisites for approaching human-level data science performance.