Box's Apprentice: AI Iterative Science

• Box's Apprentice is a benchmark that simulates the iterative Boxian cycle of proposing models, designing experiments, revising theories, and communicating results.
• It integrates experimental design with model discovery by applying Bayesian optimal design and standardized prediction error to evaluate both experiment quality and model performance.
• Empirical findings reveal that current LLMs, including GPT-4o augmented with explicit modeling, struggle with strategic data collection and tend toward overly simple models.

“Box’s Apprentice” denotes, in the framing of BoxingGym, a possible AI system that does a simplified version of science in the spirit of George Box: it proposes a model, designs informative experiments, revises the model, and communicates what it learned. The phrase is used explicitly to describe a benchmark target rather than a finished scientific agent. In that sense, the benchmark tests whether a system treats models as useful but revisable tools rather than final truth, and whether it can couple experimental design with model discovery in an interactive loop (Gandhi et al., 2 Jan 2025).

1. Conceptual framing

The core motivation for BoxingGym is that prior work usually evaluates experimental design and model discovery separately, while real science tightly couples them. Scientists use theories to decide what to measure, then use the resulting data to update those theories. The benchmark is introduced on the claim that there was no benchmark that systematically tests this full loop for LLM-based agents. BoxingGym therefore operationalizes a Boxian cycle of model-building, perturbation or experimentation, and revision (Gandhi et al., 2 Jan 2025).

The benchmark is organized around four desiderata. First, it supports active experimentation without real lab costs, so agents can interactively run experiments in a simulated world. Second, it allows flexible theory representations, because scientific theories may be symbolic, probabilistic, or verbal, and the benchmark therefore uses a language interface and generative models. Third, it provides integrated evaluation of experimentation and theory-building, measuring both experiment quality and model quality. Fourth, it supports goal-directed science, since environments can be instantiated with a user-specified inquiry goal. This suggests that “Box’s Apprentice” is best understood as a capability target for autonomous scientific agents rather than as a single fixed architecture.

2. Benchmark realization in BoxingGym

Each BoxingGym environment is implemented as a generative probabilistic model defining a joint distribution over experimental outcome $y$, experimental design $d$, and unobserved parameters $\theta$, with prior $p(\theta)$ and simulator $p(y \mid \theta, d)$. Running an experiment means choosing $d$ and sampling from the predictive distribution

$$y \sim p(y \mid d) = E_{p(\theta)}[p(y \mid \theta, d)].$$

This construction makes interactive experimentation tractable and supports quantitative Bayesian optimal experimental design style evaluation (Gandhi et al., 2 Jan 2025).

The benchmark contains 10 environments spanning psychology, ecology, medicine, education, and human behavior. A distinctive aspect is that some environments are human-behavior settings with natural-language observations, not just numeric simulators.

Environment	Domain	Stated structure or goal
Location finding	Signal localization	Hidden signal sources in 2D; infer source locations or predict signal
Hyperbolic temporal discounting	Human choice	Choice between immediate and delayed reward with latents $k,\alpha$
Death process	Population infection	Choose observation times for a binomial observation model
Item Response Theory (IRT)	Education	Predict correctness for student-question pairs using 2PL IRT
Animal growth curves (Dugongs)	Biology	Predict dugong length from age with a Bayesian growth model
Population growth dynamics (Peregrines)	Ecology	Predict peregrine falcon population via cubic Poisson regression
Mastectomy survival analysis	Medicine	Predict whether a patient is alive after surgery
Predator-prey dynamics	Ecology	Predict populations over time under Lotka-Volterra dynamics
Emotions from outcome	Human behavior	Predict 8 Likert-scale emotions, then translate to natural language
Moral Machines	Human moral decisions	Logistic regression over dilemma attributes, translated to free-form language

Several environments are specified by explicit generative equations. In location finding, the signal model is

$$\mu(\theta,\xi)=b+\sum_{k=1}^K\frac{\alpha_k}{m+\mid\mid \theta_k-\xi \mid\mid^2},$$

observed with Gaussian noise. In hyperbolic temporal discounting, the priors are

$$\log k \sim N(-4.25, 1.5), \alpha \sim HalfNormal(0,2),$$

and the choice probability is

$d$0

where $d$1 and $d$2. In Dugongs, the growth model is

$d$3

In Peregrines, the population model is

$d$4

3. Experimental-design evaluation

BoxingGym evaluates experiment quality using Expected Information Gain (EIG) from Bayesian optimal experimental design: $d$5 EIG is interpreted as the expected reduction in uncertainty about latent parameters $d$6 after performing design $d$7. Because EIG is doubly intractable, the benchmark approximates it using a nested Monte Carlo estimator. The printed formula in the paper is typeset incorrectly, but the intended estimator is the standard nested Monte Carlo estimator for EIG (Gandhi et al., 2 Jan 2025).

For actual benchmarking, the paper uses Expected Information Regret (EI Regret). This compares the EIG of the agent’s chosen experiment to the maximum EIG among 100 random experiments. Lower regret therefore corresponds to more informative experiment selection. The benchmark also assesses whether agents can use collected data by measuring prediction quality before and after experimentation.

Prediction quality is normalized relative to a prior predictive baseline through standardized prediction error. The paper describes computing a discrepancy $d$8, then standardizing against a prior predictive mean $d$9 and variance $\theta$0. The printed formula is malformed, but the intended idea is to compare agent error to the baseline error under the prior predictive mean. Negative standardized error indicates doing better than the baseline; positive standardized error indicates doing worse.

4. Model discovery as explanation and prediction

The benchmark’s most novel evaluation is communication-based model discovery. The procedure is fixed. First, a scientist agent interacts with the environment for 10 experiments. Second, it produces a natural language explanation of what it discovered. Third, a novice agent, with no direct environment interaction, receives only that explanation and must make predictions. The explanation is token-limited, explicitly rewarding parsimony, and the evaluation is intended to be representation-agnostic: the scientist may have an explicit statistical model internally, but must ultimately explain the phenomenon in language (Gandhi et al., 2 Jan 2025).

This procedure turns model quality into a test of whether the model can be compressed into a concise, predictive explanation. The paper’s underlying criterion is that a good scientific theory is a concise and predictive explanation. Besides explanation transfer, BoxingGym also uses “standard model evaluation metrics such as prediction errors,” with standardized prediction error as the main reported metric in the tables. The framework distinguishes three conditions: scientist error before experiments as a negative control, scientist error after experiments as a weak positive control, and novice error after explanation as the discovery or communication score. A good explanation should make novice performance better than the negative control and ideally close to the positive control.

A plausible implication is that BoxingGym does not equate model discovery with parameter recovery alone. It instead evaluates whether an agent can discover a model that remains predictive after being translated into a concise natural-language account.

5. Empirical profile of the benchmarked agents

The paper benchmarks two systems: GPT-4o as a pure language agent, and Box’s Apprentice, defined as GPT-4o augmented with the ability to write and fit PyMC generative models and use them in experiment selection and explanation. The main empirical finding is that current LLMs like GPT-4o struggle with both experimental design and model discovery, and that adding explicit statistical modeling does not reliably improve performance (Gandhi et al., 2 Jan 2025).

The results are mixed rather than uniformly negative. GPT-4o improves in some domains: Peregrines changes from $\theta$1, and Predator-Prey from $\theta$2. However, the paper reports that experiments can also worsen predictions because the model “over-corrects” from limited data. Hyperbolic Discounting is given as an example where choice prediction worsens after experimentation.

Box’s Apprentice can sometimes infer useful explicit models. In IRT, it proposes a logistic model with student ability and question difficulty. In Location Finding, it discovers an inverse-distance style law. In Death Process, it outperforms GPT-4o strongly on prediction after experiments: Box’s Apprentice: $\theta$3, GPT-4o: $\theta$4. Yet these gains are not systematic. In Peregrines, Box’s Apprentice obtains $\theta$5 versus $\theta$6 for GPT-4o, and Hyperbolic Discounting again worsens after experiments. The paper identifies a major failure mode: with few datapoints, Box’s Apprentice is biased toward overly simple functional forms; in Peregrines, it uses a simple linear trend for a nonlinear population process. It also often fails in strategic data collection: even when it proposes a near-correct model family, it does not gather the right low-data observations to fit it well, and appendix EI regret results show high regret relative to random baselines.

The communication results are similarly partial. Explanations from both agents usually help the novice somewhat, but novice performance generally remains worse than the scientist with direct interaction. A notable exception occurs in IRT for GPT-4o, where communicating the explanation helped the novice outperform the scientist: scientist after experiments: $\theta$7, novice after explanation: $\theta$8. The authors interpret this as communication forcing the model to generalize. The paper also compares a prior condition with domain context to a no-prior condition with abstract descriptions, because some of GPT-4o’s apparent success seems to come from stored prior knowledge, not learned discovery from interaction itself.

Coverage is incomplete. Box’s Apprentice was not run on some domains because GPT-4o could not reliably generate valid PyMC programs, notably Predator-Prey and Emotions.

6. Scope, limitations, and neighboring uses of “apprentice”

BoxingGym is aligned with the broader goal of automated science: not merely answering questions from static data, but building models, perturbing systems with experiments, revising models, and explaining results. The benchmark’s name and framing signal that the goal is not to find “true” models, but useful, predictive, revisable ones. Its significance is therefore methodological: it offers a concrete benchmark for measuring progress toward a future scientific apprentice to George Box’s philosophy (Gandhi et al., 2 Jan 2025).

The paper is equally explicit about limitations. The environments use pre-defined experimental paradigms rather than requiring invention of the experimental method itself. There are no explicit cost/time constraints on experimentation. Domain breadth is limited. Some human-behavior environments rely on LLM-mediated simulation. Evaluation assumes a known generative model, so EIG is meaningful partly because the benchmark world is model-generated. For these reasons, BoxingGym is presented as a simplified proxy for scientific discovery, not a full solution.

In the broader literature, the term “apprentice” is used in several distinct senses. In a computational creativity system for movie-title food puns, the apprentice is an NMT based sequence-to-sequence model learning under a genetic algorithm master in a socially hierarchical relation shaped by authoritative, authoritarian, permissive, and rejecting-neglecting parenting styles; there, apprenticeship is modeled explicitly as socialization rather than Boxian scientific revision (Hämäläinen et al., 2019). In low-precision deep learning, “Apprentice” refers to a quantized student network trained under a full-precision teacher by knowledge distillation, with Scheme-A, Scheme-B, and Scheme-C spanning joint training, pretrained-teacher training, and post-quantization fine-tuning; there, apprenticeship denotes teacher-student transfer for deployment efficiency rather than experimental design or theory revision (Mishra et al., 2017). In humanoid rearrangement, a paper on Humanoid Hanoi uses the query’s phrasing to discuss long-horizon box rearrangement through reusable skills executed by a shared, task-agnostic whole-body controller; that work is about skill composition for box pickup, transport, and stacking, not about George Box or scientific model discovery (Kim et al., 14 Feb 2026).

These neighboring usages clarify the specificity of the BoxingGym phrase. “Box’s Apprentice” in the Boxian sense refers neither to generic teacher-student learning nor to a socially subordinate helper model. It refers to an AI system evaluated on whether it can engage in an iterative cycle of proposing models, designing informative experiments, revising those models from data, and communicating concise predictive explanations.