An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery

Abstract: LLM coding agents increasingly perform open-ended data modeling and analysis. These agents are stochastic and adaptive, and therefore their autonomous model discovery behavior cannot be adequately characterized by a single benchmark run. In this work, we propose an experimental design and analysis framework for systematically evaluating this discovery process, quantifying its variability, and identifying important factors. The proposed framework treats these agents as stochastic model-discovery operators, which map task-specific discovery data and an optimization target to a fitted model. Specifically, we investigate two such operators, Codex and Claude Code, under controlled experimental factors including agent's reasoning effort, task, optimization metric, and composition of training data. For each agent-task-metric combination, regression models and inference are conducted for multiple responses such as output quality, dollar cost, wall-clock time, and process complexity. Furthermore, we develop a utility-aligned canonical decomposition to characterize the dominant direction of the reasoning-effort effect and to assess whether that direction aligns with a performance-cost utility direction. The proposed framework is demonstrated on a testbed of networked word-forming games with insightful findings on reasoning effort with respect to cost and process complexity.

• The paper demonstrates that increased reasoning effort substantially raises operational costs and process complexity, often without yielding proportional gains in model quality.
• The paper employs a complete factorial design and introduces the Utility-Aligned Canonical Decomposition (UACD) to rigorously analyze multivariate responses.
• The paper reveals that while agents self-assess accurately on predictive tasks, they overestimate performance on generative tasks, highlighting the need for external validation.

Experimental Design for Evaluating Agentic AI in Model Discovery

Introduction

This paper presents a comprehensive experimental framework that systematizes the evaluation of agentic AI—specifically LLM-powered coding agents—in autonomous model discovery tasks. Recognizing the adaptive, stochastic, and highly variable behavior of such agents, the authors establish a factorial design paradigm that enables robust, multivariate analyses of agent performance, cost, and process complexity across controlled settings. The empirical focus leverages networked group-anagram game data, requiring both predictive and generative modeling, and involves two commercially available LLM agents, Codex (GPT-5.5) and Claude Code (Claude Opus 4.7).

Figure 1: Illustration of the experiment and analysis pipeline.

Experimental Framework and Methodology

The proposed framework treats LLM coding agents as "model-discovery operators," evaluating their ability to autonomously construct, evaluate, and select statistical or agent-based models for complex temporal, networked data. The experimental design is a complete factorial over six factors: agent, task type, target metric, reasoning effort requested, fold of cross-validation, and data regime. The main focus is on how the requested reasoning effort (as natively parameterized by each agent) manifests in the joint space of model quality, operational cost, and modeling process complexity.

Two primary tasks are evaluated:

• Task 1 (Predictive): Next-action prediction for individual players, with weighted area under the ROC curve (wAUC) and a rare-event discrimination metric (MRI-RO) as targets.
• Task 2 (Generative): Construction of agent-based simulators to reproduce whole session trajectories, assessed via a multi-component KL divergence ($\mathrm{KL}_6$) and a Wasserstein distance on word-formation edit paths ($D_{\mathrm{LD}}$).

Each agent-task-metric combination constitutes a separate analysis stratum.

Multivariate Analysis and Canonical Decomposition

The analysis utilizes multivariate linear models (MLM) to capture the impact of experimental controls, particularly the linear and quadratic effects of reasoning effort within each agent-task-metric stratum. The eight-dimensional response vector comprises: primary and non-primary model quality metrics, monetary and wall-clock costs (both log-transformed), and multiple process-tracing features (number of models/features examined, logged modeling decisions, and script length).

A key methodological innovation is the introduction of the Utility-Aligned Canonical Decomposition (UACD), designed to characterize the dominant direction of effort-induced changes in the multivariate outcome space. This allows assessment not only of whether reasoning effort yields statistically robust effects but also if those effects align with a pre-specified utility direction prioritizing performance over cost.

Figure 2: Simplified rank-one illustration of UACD. Left: trivial case with identity covariance, right: general case with structured error covariance.

Empirical Findings

Task Specialization and Metric Optimization

The analysis demonstrates strong agent specialization to assigned metrics: agents consistently perform better on metrics they explicitly optimize. For neural predictive metrics (wAUC), and rare-event discrimination (MRI-RO), optimizing directly for these measures drives substantial cross-metric differentiation but does not necessarily yield broader generalization.

Figure 3: Performance by primary metric across the reasoning-effort ladder.

Resource Usage Scaling with Reasoning Effort

The most robust, monotonic effect of increased reasoning effort is a substantial rise in process costs and resource use, measured by both token consumption and codebase expansion. For instance, median fresh-token consumption for Codex and Claude Code more than doubles across the reasoning-effort ladder.

Figure 4: Fresh-token use and submitted-code length across the reasoning-effort ladder, by agent.

This resource scaling consistently fails to translate into commensurate gains in held-out model quality or utility.

Figure 5: Left: strong correlation between token use and script length; right: weak or null relationship between token use and standardized performance.

Cost–Performance Decoupling

Analysis of the cost–performance plane reveals that Codex reliably incurs lower operational costs (by a factor of approximately 7), while reaching similar levels of primary-metric performance as Claude Code. There is no systematic within-agent association between higher cost and better model quality, excepting a single condition (claude-code on t-MRI).

Figure 6: Primary metric against cost in US dollars (log axis), by condition.

Self-Assessment Calibration

Agents provide accurately calibrated self-reported validation scores for the predictive task, but systematically overestimate their performance on generative simulation tasks, leading to unreliable self-assessment in these settings.

Figure 7: Agent self-assessment against the evaluated metric, one panel per condition.

Full versus Partial Data Availability

Full data access conferred an advantage only for the generative multi-summary KL metric; otherwise, impact is negligible—highlighting the interaction between data regime and metric type.

Figure 8: Primary performance under full versus partial observation, by task and metric.

Statistical Inference on Effort Effects

Univariate regression on the primary metric alone generally fails to detect significant effects of reasoning effort. However, projecting the response onto the pre-specified utility direction (equal weights for gains in performance minus costs) reliably yields significant negative slopes: higher requested effort produces lower overall utility in seven out of eight strata (Holm-adjusted $p < 0.05$).

Figure 9: Directed utility slope $u^{\top}\hat$ per stratum with $95\%$ interval and utility-alignment $\eta^{(1)}$ for the canonical direction.

The UACD reveals that nearly all the effort-induced multivariate effect is concentrated in a single direction—but this leading direction is negatively aligned with the pre-specified utility vector in all eight strata. The sign of this misalignment is consistent and statistically unlikely under the null ($p = 0.008$ by sign test).

Process Complexity and Search Width

Increased reasoning effort reliably expands the candidate model set considered (median increases from 3 to 6 across the ladder), and yields longer scripts and more intricate modeling processes. However, broader search does not systematically lead to better discovery, echoing the findings from utility regression and canonical alignment.

Figure 10: Number of candidate models considered, one panel per reasoning-effort level.

Model Selection Behavior

Task 1 submissions concentrate almost entirely on gradient-boosted trees and are agent-dependent in implementation; Task 2 (simulative) submissions span a wider family of approaches.

Figure 11: Composition of submitted model families, by task, split by agent.

Implications and Future Directions

Practical Implications

• Reasoning effort control should be viewed as a resource-use parameter, not a reliable pathway to quality gains. In these experiments, increased effort consistently drove up resource expenditure and process complexity, with negligible or negative returns in evaluated utility.
• Metric selection is crucial: Agents optimize for explicit objectives, potentially at the expense of secondary performance goals.
• Self-assessment remains unreliable for generative tasks, implying external evaluation is non-negotiable for trustworthy deployment.
• Agent and provider choices strongly affect cost-efficiency, often decoupled from predictive metric quality.

Theoretical and Methodological Insights

• The UACD offers a generalizable tool for multivariate experimental analysis of agentic AI, especially in contexts where no single performance indicator suffices.
• The results accentuate the necessity of full joint modeling of performance, cost, and process in LLM-augmented scientific workflows.
• The work underscores the limits of single-run benchmarking for stochastic, adaptive AI: rigorous designed experiments are crucial for reliable agentic system evaluation.

Future Work

The authors note several limitations, including lack of per-configuration replication, analysis restrictions to two commercial agents and one domain, and imputed (not metered) costs for Codex. Extending this analysis to a wider range of agents, domains, and cost validation, as well as deeper trace analysis to diagnose reasoning effort allocation, are identified as priorities.

Conclusion

This work establishes an experimental design and analysis framework that systematically evaluates stochastic, agentic LLM coding agents in autonomous data modeling. The framework yields detailed operational guidance for balancing agentic configuration controls, particularly reasoning effort, emphasizing that increases to effort lead primarily to increased cost and process complexity, not higher model quality or utility. The methodological advances, notably the UACD, provide a foundation for future empirical and theoretical study of agentic AI discovery systems, informing both deployment practices and the development of next-generation, cost-aware, multivariate evaluation benchmarks.

---

Reference: "An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery" (2607.06413)