Empirical-MCTS: Adaptive Monte Carlo Tree Search

• Empirical-MCTS is a family of algorithms that integrates adaptive, empirical experience accumulation with traditional Monte Carlo Tree Search to enhance decision making.
• It employs a dual-loop architecture featuring local meta-prompt evolution and global memory optimization to blend immediate adaptations with enduring learning.
• Empirical-MCTS shows improved performance in benchmarks like AIME25 and MathArena Apex, outperforming stateless MCTS approaches in complex reasoning tasks.

Empirical-MCTS denotes a family of algorithms that expand the canonical Monte Carlo Tree Search (MCTS) paradigm by explicitly incorporating adaptive, empirical experience accumulation at inference time. These algorithms fuse structured tree-based exploration with ongoing, non-parametric adaptation based on success/failure episodes—contrasting sharply with classical, stateless MCTS procedures that discard all search traces after each task. Recent developments have operationalized Empirical-MCTS via dual update loops (local meta-level evolution and global memory distillation), enabling persistent improvement in decision quality across diverse problem instances and domains, particularly in reasoning-intensive settings for LLMs and complex games (Lu et al., 4 Feb 2026, Galván et al., 2021).

1. Background: Monte Carlo Tree Search and the Need for Experience Accumulation

Standard MCTS interleaves stochastic simulation, heuristic evaluation, and bandit-inspired sampling to construct an asymmetric search tree representing variable-depth trajectories from an initial state. At each node, actions are selected to maximize an exploration–exploitation tradeoff, most notably via the Upper Confidence Bounds for Trees (UCT) criterion: $U_i = \bar X_i + C_p\,\sqrt{\frac{\ln N}{n_i}},$ where $\bar X_i$ is the mean reward for child $i$, $N$ is the parent's visit count, $n_i$ the child's, and $C_p$ an exploration hyperparameter (Mirsoleimani et al., 2015). However, classical MCTS (including UCT) is episodic: after a rollout, all internal statistics (except immediate recommendations) are reset. This process precludes any empirical learning from prior rounds, limiting cross-instance generalization and preventing the agent from accumulating strategic “wisdom” (Lu et al., 4 Feb 2026).

Recent work identifies this statelessness as a key limitation for applications requiring adaptive reasoning over multiple problems or extended gameplay. In contrast to single-use inference, empirical problem solving—as exhibited by humans—derives substantial advantage from continually integrating lessons, distilled insights, and evolving heuristics (Lu et al., 4 Feb 2026).

2. Dual-Loop Architecture: Integrating Local and Global Experience

Empirical-MCTS augments the standard search loop by introducing two intertwined adaptation mechanisms:

1. Local Search Loop with Meta-Prompt Evolution (PE-EMP): Each expansion phase employs pairwise comparison between candidate and incumbent responses, using a reflexive meta-prompt optimizer. PE-EMP adapts "prompts" (search policies, domain constraints, or critic instructions) in real time by generating evaluation criteria, evolving new meta-prompts, and synthesizing context-aware guidance.
2. Global Memory Optimization Agent: Successful episodes yield distilled “experiences” (meta-prompts or solution strategies), which are stored, merged, or pruned in a dynamic global repository $\mathcal{D}$. This non-parametric library is updated using atomic operations (Add, Modify, Merge, Delete) and conditions subsequent searches via retrieval of top-$k$ prior experiences matching the task context.

The result is a dual experience loop—short-term meta-prompt evolution within a problem, and long-term memory optimization across problems—enabling continuous, stateless adaptation without internal model weight updates (Lu et al., 4 Feb 2026).

3. Formal Mechanisms: PE-EMP and Memory Optimization

Pairwise-Experience-Evolutionary Meta-Prompting (PE-EMP): Let $S_p = (\mathcal{P}_p, r_p)$ be a parent state and $S_c = (\mathcal{P}_c, r_c)$ its child. The judge module computes a vector-valued score, produces new distilled experience $\mathcal{E}_{new}$, and synthesizes an evolved meta-prompt $\mathcal{P}_{evolved}$. Transition probabilities use a Bradley-Terry model: $$Q_{\mathrm{local}}(s_c) = \frac{\exp(S_c)}{\exp(S_c) + \exp(S_p)}.$$ A hybrid reward, incorporating enhanced Borda count over the candidate set, is used for backpropagation: $$R = \alpha Q_{\mathrm{local}}(s_c) + (1-\alpha)\frac{r_{\mathrm{borda}}}{|\mathcal{T}|}.$$

Memory Optimization Agent: The agent’s repository $\mathcal{D}$ is updated via atomic operations driven by candidate experience utility: $$\pi_{mem}(\mathcal{E}_{new},\mathcal{E}_{exist}) = \{\text{Add}:\,e_{new}^*,\,\text{Modify}:\,\Delta e_i,\,\text{Merge}:\,\{e_i,e_j\}\to e_{ij},\,\text{Delete}:\,e_k\}.$$ Optimized $\mathcal{D}$ entries condition the policy prior for subsequent episodes.

function UpdateLibrary(D, ops): for op in ops: if op.type == 'Add': D.add(op.experience) elif op.type == 'Modify': D.update(op.id, op.delta) elif op.type == 'Merge': D.merge(op.ids, op.new_experience) elif op.type == 'Delete': D.remove(op.id) return D

4. Algorithmic Workflow and Pseudocode

The following pseudocode summarizes the Empirical-MCTS dual-loop workflow (Lu et al., 4 Feb 2026):

Input: query q, base prompt P₀, memory D, rollouts T Initialize tree T with root v₀(q,P₀) P_evolved ← P₀ E_prior ← Retrieve(D, q) for t = 1 to T: v_p ← Select(T, UCB) S_c ← Expand(v_p, P_evolved, E_prior) (s, E_new, P_evolved) ← Judge(q, E_prior, S_c, S_p) R ← HybridReward(s, EBC(T)) Backpropagate(v_p, R) if E_new contains high-value insights: pi_mem ← Optimizer(D, E_new) D ← UpdateLibrary(D, pi_mem) return BestResponse(T)

Local exploration proceeds via bandit-informed selection and expansion, while global memory is refined over episodes.

5. Empirical Results and Comparative Evaluation

Empirical-MCTS has demonstrated substantial gains over stateless MCTS and prior memory-based approaches on complex, multi-step reasoning benchmarks. On AIME25 (mathematical problem solving), the best reported accuracy with Empirical-MCTS (DeepSeek-V3.1 backbone) is 73.3%, exceeding repeated sampling (70.0%) and LLaMA-Berry (63.3%). On MathArena Apex (proof writing), Empirical-MCTS achieved 4.17%, outperforming all previous baselines, some of which achieved 0.00% (Lu et al., 4 Feb 2026).

Ablation studies indicate the sensitivity of performance to the inclusion of meta-prompt evolution and global memory: removing PE-EMP or dynamic memory regresses accuracy to near-repeated sampling levels. Qualitative evidence further shows that the memory repository grows rapidly during search and that retrieved meta-prompts evolve from generic advice to highly domain-specific constraints (e.g., "For cyclic quadrilaterals, verify Ptolemy’s inequality first").

Table 1. AIME25 and MathArena Apex Results

Method	AIME25 (%)	MathArena Apex (%)
Baseline	56.7	0.00
FLEX	66.6	–
Repeated Sampling	70.0	0.00
LLaMA-Berry	63.3	2.08
Empirical-MCTS	73.3	4.17

Cost-performance analysis reveals that Empirical-MCTS with lightweight models (e.g., Gemini 3 Flash) achieves strong results at a fraction of the cost of more parametric approaches such as GPT-5.2 (High).

6. Extensions: Evolutionary Selection in Games and Beyond

An independent line of work frames Empirical-MCTS as online, evolutionary adaptation of the UCT formula itself (Galván et al., 2021). Using a (μ=1, λ=4) evolution strategy, symbolic expressions for selection are mutated and selected dynamically based on rollout performance, evolving alternatives such as

$$f(s,a) = \overline{Q}(s,a) + \sqrt{\overline{Q}(s,a)}$$

that may discard explicit visit-count dependence yet maintain competitive or superior performance.

Empirical evaluation in the game of Carcassonne shows that ES-MCTS (the evolutionary variant) outperforms both tuned UCT, star-minimax, and random controllers under strict statistical significance (Wilcoxon p < 0.05), and that integrating this adaptation with backpropagation to the search tree is critical for success.

7. Limitations and Ongoing Directions

The performance of Empirical-MCTS depends on the veracity and utility of accumulated experiences. Weak internal verification or unchecked "hallucinated" strategies can degrade future recommendations. Memory and prompt evolution mechanisms introduce algorithmic overhead and managerial complexity compared to stateless approaches. Ongoing work explores robustness to noisy or adversarial memories, multi-session persistence, and hybrid parametric + non-parametric update regimes (Lu et al., 4 Feb 2026).

A plausible implication is that Empirical-MCTS constitutes a foundation for online, continually learning agents that bridge the gap between episodic decision making and lifelong adaptation, especially in domains where structured search and empirical generalization must coexist.