Bilevel Autoresearch: Meta-Autoresearching Itself

Abstract: If autoresearch is itself a form of research, then autoresearch can be applied to research itself. We take this idea literally: we use an autoresearch loop to optimize the autoresearch loop. Every existing autoresearch system -- from Karpathy's single-track loop to AutoResearchClaw's multi-batch extension and EvoScientist's persistent memory -- was improved by a human who read the code, identified a bottleneck, and wrote new code. We ask whether an LLM can do the same, autonomously. We present Bilevel Autoresearch, a bilevel framework where an outer loop meta-optimizes the inner autoresearch loop by generating and injecting new search mechanisms as Python code at runtime. The inner loop optimizes the task; the outer loop optimizes how the inner loop searches. Both loops use the same LLM -- no stronger model is needed at the meta level. On Karpathy's GPT pretraining benchmark, the meta-autoresearch outer loop achieves a 5x improvement over the standard inner loop alone (-0.045 vs. -0.009 val_bpb), while parameter-level adjustment without mechanism change yields no reliable gain. The outer loop autonomously discovers mechanisms from combinatorial optimization, multi-armed bandits, and design of experiments -- without human specification of which domains to explore. These mechanisms succeed by breaking the inner loop's deterministic search patterns, forcing exploration of directions the LLM's priors systematically avoid. The core principle is simple: if autoresearch can meta-autoresearch itself, it can, in principle, meta-autoresearch anything with a measurable objective.

• The paper introduces an outer meta-loop that uses LLM-generated Python code to dynamically optimize the search mechanism, achieving up to a 5× validation loss improvement.
• It employs a three-level architecture where the inner loop handles standard autoresearch tasks while Level-2 autonomously revises the strategy to overcome LLM biases.
• Empirical evaluations demonstrate dramatic gains in hyperparameter tuning and architectural self-improvement, underscoring the feasibility of fully autonomous code-driven optimization.

Bilevel Autoresearch: Autonomous Meta-Optimization via LLM Code Generation

Framework and Motivation

Bilevel Autoresearch formalizes the autoresearch loop as a bilevel optimization problem where an LLM-driven meta-loop autonomously improves the search mechanism itself. Classical autoresearch systems (e.g., "autoresearch," AutoResearchClaw, EvoScientist) utilize LLMs to iteratively propose changes (hyperparameters or architecture), execute training, and evaluate outcomes, but their inner search strategies are static and human-engineered. This paper introduces an outer loop that reads the inner loop's code, identifies performance bottlenecks, then generates and injects new search mechanism code at runtime, leveraging the same LLM model for both levels. The paradigm shift is that structural improvements formerly requiring a human are now realized through LLM code generation.

Figure 1: The inner loop optimizes task output; the outer meta-loop optimizes the search mechanism by generating and injecting new Python code at runtime.

Architectural Design and Algorithmic Flow

The system comprises three algorithmic levels: Level~1 executes the standard autoresearch cycle—propose, train, evaluate, accept or discard. Level~1.5 (outer strategy adjustment) periodically updates search parameters, freezes/unfreezes specific hyperparameters, and guides the inner loop toward under-explored regions. Level~2 (mechanism research) performs autonomous architecture modification through structured LLM dialogues that result in new Python modules, which are dynamically injected into the running system.

Figure 2: Bilevel Autoresearch architecture: Level~1 is the baseline loop; Level~1.5 adjusts search every five iterations; Level~2, every two cycles, generates new search mechanisms via a four-round research session.

Each Level~2 session consists of exploration (survey adjacent optimization domains), critique (select promising mechanisms based on observed search failures), specification (define integration interfaces and points), and code generation (produce valid Python modules). Rigorous validation and dynamic import protocols ensure that faulty code is reverted and does not affect system integrity.

Figure 3: Level~2 research session: four sequential LLM calls generate and validate a Python module modifying the inner loop's search behavior.

Empirical Evaluation and Results

A four-group ablation protocol was used to isolate contributions from each architectural level, all evaluated on Karpathy's GPT pretraining benchmark (50M parameters, RTX~5090, 300s training budget). Key findings include:

• Group~A (Level~1 only): $-0.009 \pm 0.002$ mean improvement—consistent, but small and deterministic.
• Group~B (Level~1 + 1.5): $-0.006 \pm 0.006$—outer strategy adjustment increased diversity but yielded no reliable gain.
• Group~C (Level~1 + 1.5 + 2): $-0.045 \pm 0.030$—a 5× improvement over Group~A and 7.5× over Group~B; two repeats showed substantial gains.
• Group~D (Level~1 + 2): $-0.034 \pm 0.031$—similar improvements, demonstrating Level~2’s sufficiency even without Level~1.5.

  Figure 4: Running-minimum validation loss per iteration; Groups~C and~D show sharp drops after Level~2 mechanisms steer search toward effective batch-size reduction.

The observed improvement is attributed to Level~2 discovering mechanisms that enforced exploration of parameter dimensions systematically avoided by the LLM’s inductive bias. Specifically, discovering that reducing TOTAL_BATCH_SIZE (from $2^{19}$ to $2^{17}$-$2^{18}$) on Blackwell compute under a fixed time budget led to dramatic gains. Default LLM search paths repeatedly attempted batch size increases, missing the optimal decrease direction—only Level~2’s combinatorial and orthogonal exploration mechanisms enabled this discovery.

Level~2 Mechanism Inventory and Behavioral Analysis

Level~2 generated a diverse mechanism inventory from distinct domains (Tabu Search Manager, Multi-Scale Bandit Proposer, Systematic Orthogonal Exploration). All implementations were produced in one-shot LLM calls and five of six passed validation; one was reverted due to missing external dependencies. Tabu Search Manager, for example, prevented proposal repetition by maintaining a tabu region history, breaking deterministic cycles inherent to Level~1. Other mechanisms forced bandit-style parameter exploration or enforced orthogonal search directions.

Inner loop search behavior was near-deterministic without Level~2—LLMs repeatedly proposed the same hyperparameters. With Level~1.5, exploration widened but remained structurally bounded. Level~2 altered the proposal logic, ultimately enabling robust search over previously neglected parameter spaces (e.g., batch size reduction).

Practical and Theoretical Implications

Practically, the bilevel architecture enables fully autonomous improvement of autoresearch frameworks, eliminating the bottleneck of human-driven architectural tuning. The approach scales to any system where objectives are measurable and code structures are injective, generalizing beyond hyperparameter optimization to algorithm configuration or architecture search. Theoretically, this establishes that meta-optimization of discrete code artifacts—rather than just real-valued parameters—is feasible and effective using only the baseline LLM model.

Key limitations include small sample sizes ($n=3$ per group), single benchmark evaluation, and potential fragility from dynamic code injection and prompt-induced domain bias. Scalability to broader tasks, robust dependency handling, and prompt generalization remain open research trajectories.

Speculation on Future Developments

Future work should focus on statistical robustness ($n \geq 10$), multi-benchmark evaluation, improved promptless mechanism discovery, and introspective code validation pipelines. Richer interface specifications and generic test harnesses could further enhance mechanism quality. The approach could catalyze AI-driven tool chains that execute self-improvement cycles in software engineering, model design, and scientific discovery, ultimately evolving systems that meta-research any objective-driven process.

Conclusion

Bilevel Autoresearch substantiates that an LLM can meta-optimize its own research loop, generating and dynamically integrating new search strategies without human intervention. Empirical results show 5× validation improvement over the inner loop baseline, with mechanism-level modification being essential for escaping LLM-induced search biases. The method sets a precedent for autonomous architectural self-improvement, suggesting that measurable, code-driven research systems can evolve their logic through recursive LLM-driven code generation.