Language-Based Bayesian Optimization Research Assistant (BORA)

Abstract: Many important scientific problems involve multivariate optimization coupled with slow and laborious experimental measurements. These complex, high-dimensional searches can be defined by non-convex optimization landscapes that resemble needle-in-a-haystack surfaces, leading to entrapment in local minima. Contextualizing optimizers with human domain knowledge is a powerful approach to guide searches to localized fruitful regions. However, this approach is susceptible to human confirmation bias and it is also challenging for domain experts to keep track of the rapidly expanding scientific literature. Here, we propose the use of LLMs for contextualizing Bayesian optimization (BO) via a hybrid optimization framework that intelligently and economically blends stochastic inference with domain knowledge-based insights from the LLM, which is used to suggest new, better-performing areas of the search space for exploration. Our method fosters user engagement by offering real-time commentary on the optimization progress, explaining the reasoning behind the search strategies. We validate the effectiveness of our approach on synthetic benchmarks with up to 15 independent variables and demonstrate the ability of LLMs to reason in four real-world experimental tasks where context-aware suggestions boost optimization performance substantially.

• The paper presents a novel framework that fuses traditional Bayesian optimization with LLM-driven insights for effective high-dimensional search.
• It employs a Gaussian Process surrogate model and dynamic action strategies to adaptively incorporate contextual domain knowledge.
• Experimental results on synthetic and real-world benchmarks demonstrate substantial efficiency gains in optimization tasks.

Language-Based Bayesian Optimization Research Assistant (BORA)

Introduction

The paper "Language-Based Bayesian Optimization Research Assistant (BORA)" (2501.16224) presents a novel framework designed to enhance Bayesian Optimization (BO) for complex scientific problems. BORA integrates LLMs to provide a context-aware optimization approach, addressing the challenges posed by multivariate, high-dimensional searches that are typical in experimental science. The framework adopts an innovative hybrid optimization strategy that dynamically leverages both stochastic inference and domain knowledge insights derived from LLMs to guide the exploration of search spaces effectively.

Figure 1: The BORA framework. Icons from~\protect\cite{c:flaticon_icons.

Methodology

BORA combines the strengths of conventional BO, known for its effective design space exploration, with LLMs' capacity to contextualize domain knowledge in optimization tasks. This synergy allows BORA to offer substantial improvements over traditional BO by providing real-time commentary and hypothesis-driven insights to steer the optimization process.

Framework Design

• Surrogate Model: At the core of BORA is a Gaussian Process (GP), serving as a surrogate model to approximate the unknown objective function. The GP is iteratively updated to capture new data and refine predictions.
• Actions and Adaptation: BORA operates through multiple actions: continuing with vanilla BO, LLM-driven suggestions, and LLM-informed BO point selection. These actions adjust dynamically based on the estimated uncertainty and performance plateauing, representing a policy guided by heuristic rules.
• Incorporation of LLMs: LLMs provide contextual insights and suggest hypotheses that are tested within the optimization loop, enhancing exploration when the search space complexity or size makes conventional BO start inefficiently.

Experiments

Extensive experiments validate BORA's capabilities across synthetic and real-world benchmarks, including functions with up to 15 variables and tasks in chemical materials design and solar energy optimization.

Figure 2: Visualization of BORA maximizing Branin 2D (which contains three global maxima) under a budget of 22 optimization steps (numbered points).

Results

• Synthetic Functions: BORA consistently outperformed baselines such as vanilla BO, TuRBO, and LLM-only strategies across high-dimensional benchmarks like Levy and Ackley functions.
• Real-World Applications: In tasks such as Hydrogen Production and Sugar Beet Production, BORA demonstrated substantial efficiency gains, with significant improvements in search exploration and convergence speed.

  Figure 3: BORA vs Baselines on six experiments. Solid lines show average values while shaded areas indicate $\pm 0.25$ standard error.

Theoretical and Practical Implications

The introduction of LLMs in the BO framework represents a significant step towards more autonomous and adaptive optimization systems in scientific research. BORA's ability to leverage large-scale textual data and domain-specific insights extends the applicability of BO to complex real-world problems. The framework also establishes a foundation for future developments in incorporating machine learning insights directly into experimental and optimization workflows.

Conclusion

BORA exemplifies a pioneering approach by blending the exploratory strength of Bayesian methods with the reasoning capabilities of LLMs. This work underscores the potential for advanced AI systems to not only assist in experimental design and execution but also to drive substantive gains in efficiency and outcome quality across varied scientific domains. Future research may focus on refining BORA's meta-learning strategies and exploring its application in multi-fidelity and multi-objective optimization scenarios.