Evolution Strategies at the Hyperscale

Abstract: We introduce Evolution Guided General Optimization via Low-rank Learning (EGGROLL), an evolution strategies (ES) algorithm designed to scale backprop-free optimization to large population sizes for modern large neural network architectures with billions of parameters. ES is a set of powerful blackbox optimisation methods that can handle non-differentiable or noisy objectives with excellent scaling potential through parallelisation. Na{ï}ve ES becomes prohibitively expensive at scale due to the computational and memory costs associated with generating matrix perturbations $E\in\mathbb{R}^{m\times n}$ and the batched matrix multiplications needed to compute per-member forward passes. EGGROLL overcomes these bottlenecks by generating random matrices $A\in \mathbb{R}^{m\times r},\ B\in \mathbb{R}^{n\times r}$ with $r\ll \min(m,n)$ to form a low-rank matrix perturbation $A B^\top$ that are used in place of the full-rank perturbation $E$. As the overall update is an average across a population of $N$ workers, this still results in a high-rank update but with significant memory and computation savings, reducing the auxiliary storage from $mn$ to $r(m+n)$ per layer and the cost of a forward pass from $\mathcal{O}(mn)$ to $\mathcal{O}(r(m+n))$ when compared to full-rank ES. A theoretical analysis reveals our low-rank update converges to the full-rank update at a fast $\mathcal{O}\left(\frac{1}{r}\right)$ rate. Our experiments show that (1) EGGROLL does not compromise the performance of ES in tabula-rasa RL settings, despite being faster, (2) it is competitive with GRPO as a technique for improving LLM reasoning, and (3) EGGROLL enables stable pre-training of nonlinear recurrent LLMs that operate purely in integer datatypes.

• The paper introduces EGGROLL, which leverages low-rank matrix perturbations to efficiently train billion-parameter neural networks.
• It replaces full-rank perturbations with products of low-rank matrices, reducing computational complexity from O(mn) to O(r(m+n)).
• EGGROLL demonstrates competitive performance across reinforcement learning, language model fine-tuning, and integer-only RNN pretraining.

Evolution Strategies at the Hyperscale: An Examination of EGGROLL

Introduction

The paper "Evolution Strategies at the Hyperscale" (2511.16652) presents Evolution Guided General Optimization via Low-rank Learning (EGGROLL), a novel approach to scaling evolution strategies (ES) for optimizing neural networks with billions of parameters. Traditional ES methods face challenges such as computational inefficiency and excessive memory consumption due to full-rank matrix perturbations. EGGROLL addresses these limitations by leveraging low-rank approximations, allowing efficient parameter updates with significant reductions in computational cost and memory usage.

Methodology

EGGROLL exploits the computational advantages of low-rank perturbations by decomposing the parameter matrix into a product of smaller matrices, thereby reducing the representational complexity and memory footprint. Specifically, it substitutes full-rank perturbations with the product of two low-rank matrices, drastically reducing auxiliary storage and computation from $O(mn)$ to $O(r(m+n))$ per layer, where $r \ll \min(m, n)$. This transformation allows EGGROLL to achieve up to a hundredfold increase in training throughput for billion-parameter models, closely matching the efficiency of batch inference operations.

The algorithm facilitates large-scale parallelization, a crucial characteristic for optimizing vast neural architectures. EGGROLL uses a deterministic random number generator to construct noise perturbations on-the-fly, eliminating the need for storing perturbations in memory. A theoretical analysis demonstrates that EGGROLL's low-rank updates approach the accuracy of full-rank Gaussian updates at an $O(1/r)$ convergence rate, underscoring the adequacy of low-rank approximations in large-scale training environments.

Experimental Results

EGGROLL's performance is empirically validated across various domains including reinforcement learning (RL), LLM (LM) fine-tuning, and integer-only nonlinear recurrent neural networks (RNNs):

1. Reinforcement Learning: EGGROLL delivers competitive performance in tabula rasa and multi-agent RL settings, matching the efficacy of naive ES while substantially reducing training time. The use of low-rank updates allows scaling to larger population sizes, enhancing exploration without compromising computational resources.
2. LLM Fine-Tuning: In LM adaptation scenarios, EGGROLL competes favorably against established techniques like GRPO, particularly in optimizing reasoning tasks. The method proves capable of efficiently fine-tuning large recurrent LMs, demonstrating viability for memory-intensive and non-differentiable objective spaces.
3. Pure Integer Pretraining: The algorithm supports stable pretraining of integer-only nonlinear RNNs by enabling large population sizes and circumventing gradient vanishing/exploding issues associated with integer datatypes. This capability highlights EGGROLL's potential in hardware-constrained environments where integer arithmetic provides computational efficiencies.

Theoretical Implications

The theoretical foundations of EGGROLL underscore the validity of low-rank perturbation updates in large-scale optimization. The O($1/r$) convergence rate assures that low-rank approximations do not significantly detract from optimization quality, making EGGROLL an attractive choice for cases where computational savings are imperative. By preserving the potential for full-rank updates through the average of population-wide low-rank matrices, EGGROLL achieves a balance between efficiency and performance.

Conclusion

"Evolution Strategies at the Hyperscale" introduces a significant advancement in scaling ES for modern large-scale neural networks. EGGROLL's approach of utilizing low-rank matrix perturbations offers a promising path forward for efficiently training billion-parameter models across diverse areas including RL and LMs. While offering theoretical soundness and empirical robustness, EGGROLL also proposes new directions for integrating ES in hardware-constrained or non-differentiable component systems, paving the way for expansive applications in large-scale AI systems.

In summary, this work cements EGGROLL's role as a scalable, resource-efficient enhancement to traditional ES, broadening the applicability of evolution-based optimization in contemporary AI model training contexts. Future exploration could extend EGGROLL's methods to broader domains, particularly in neurosymbolic systems or end-to-end systems with intricate non-differentiable architectures.