Expressive Optimizers Overview

• Expressive optimizers are optimizers with rich parameter spaces that encode nuanced inductive biases and domain-specific update rules.
• They employ techniques like equality saturation, parameterized preconditioners, and evolutionary or meta-learned synthesis to tailor performance.
• Empirical outcomes show that these optimizers enhance convergence, generalization, and robustness in both program and neural network contexts.

An expressive optimizer is an optimizer with a sufficiently rich parameterization or configuration space that allows it to encode nuanced inductive biases, realize a broad set of algorithmic behaviors, represent domain-specific knowledge, or meta-learn problem-adaptive rules. The defining feature is its flexibility: expressive optimizers are capable of shaping not only the rate but also the qualitative properties of the solutions they reach, via manual design, combinatorial rule expression, meta-learning, symbolic synthesis, or evolutionary search. This concept spans program (compiler) optimization, deep neural network training, symbolic and neural black-box optimization, and hybrid meta-optimization frameworks. The sections below provide a detailed overview of the concept in modern computational and machine learning contexts.

1. Motivations for Expressivity in Optimization

Traditional optimizers, both in compilers and machine learning, typically apply general-purpose, phase-ordered transformations or pre-defined gradient-update rules that are agnostic to domain-specific structure, data, or task objectives. This restricts their capacity to realize end-user or domain-expert desiderata, whether those desiderata involve code transformations, statistical properties of learned models, or algorithmic behaviors. In compilers, generic optimization passes may miss domain-specific rewrites; in neural network training, common optimizers (e.g. SGD, Adam) encode implicit biases but are inflexible to injected domain knowledge or architectural requirements.

Expressive optimizers address these limitations by offering rich specification or learning spaces:

• In program optimization, expressive optimizers allow declarative specification of rewrite rules directly at the API level, decoupled from rigid compiler internals (Merckx et al., 24 Feb 2025).
• In machine learning, they enable optimizers to encode or meta-learn inductive biases, regularization schemes, or update dynamics tailored to architecture, data, and task (Pascanu et al., 16 Jul 2025).
• In black-box optimization, expressive optimizers can be synthesized via symbolic or neural meta-learning, capturing or surpassing existing algorithmic templates (Chen et al., 2024, Zheng et al., 2022).

A critical driver for expressivity is the observation that optimizer choice fundamentally shapes not only the convergence speed but also the qualitative nature and inductive bias of the solutions reached (Pascanu et al., 16 Jul 2025).

2. Techniques for Achieving Optimizer Expressivity

a. Program Optimizers via Rewrite Systems and Equality Saturation

Modern expressive program optimizers leverage e-graphs—a compact data structure representing equivalence classes of program expressions—and equality saturation, whereby all rewrite rules coexist in the e-graph until an optimal representation is extracted. Rewrite rules are specified as symbolic patterns with optional type constraints, supporting context-sensitive and domain-informed transformation (Merckx et al., 24 Feb 2025). The saturation process ensures completeness, making rewrite order irrelevant and enabling both high-level and low-level rules to interact without phase-ordering conflicts. When optimization involves side-effectful operations or control flow, a "CFG skeleton" is maintained to anchor only side-effecting statements, permitting "unsafe" rewrites under user control.

b. Inductive Biases and Preconditioners in Gradient-Based Learning

Expressivity in gradient-based optimization arises by parameterizing the update rule (or its preconditioning matrix) to explicitly or implicitly encode regularizers or solution preferences. Optimizers such as SGD, momentum, Adam/AdamW, and preconditioned methods like Shampoo or K-FAC are interpreted as optimizing not just empirical loss but loss plus an optimizer-induced implicit regularizer $R_O(\theta)$ (Pascanu et al., 16 Jul 2025). The space of reachable solutions is thereby reduced from the full set of loss minima to those preferred by the optimizer's update geometry.

Explicit parameterization of the preconditioner $P(\theta)$, including non-diagonal or structure-aware forms, enables the enforcement of properties such as sparsity, class orthogonality, or margin maximization (Pascanu et al., 16 Jul 2025). The optimizer design thus becomes a central lever for encoding solution structure.

c. Automated Design: Evolutionary, Symbolic, and Meta-learned Optimizers

Automated synthesis of expressive optimizers is enabled through several mechanisms:

• Evolutionary Search: Optimizers are encoded as genomes—a combinatorial recipe of primitive update terms (e.g., raw gradient, momentum, adaptive terms, Lion-style sign-based updates)—subjected to mutation, recombination, and selection. This framework discovers novel hybrids and hyperparameter settings exceeding hand-crafted rules (Marfinetz, 5 Dec 2025).
• Symbolic Meta-Learning: Update rules are generated in closed-form from compact operator grammars, with meta-learning (PPO, behavior cloning, or combination) tuning the generator toward task-adaptive performance. Symbolic optimizer representations capture and surpass classical strategies in BBO and permit transparent adaptation (Chen et al., 2024, Zheng et al., 2022).
• Learned (Neural) Optimizers: Hierarchical models (e.g., hypernetwork-based LSTM architectures) ingest rich state (multiscale gradient EMAs, loss, tensor metadata) and output parameter-specific updates. Scalably meta-trained, these optimizers demonstrate emergent, nontrivial adaptation and solution diversity (Metz et al., 2022).

d. Optimization of Optimizers

Recent theoretical work poses optimizer design as a convex optimization problem over the space of parameter update operators, admitting closed-form optimizers and optimal hyperparameter selection per problem instance. Recovery of established optimizers through this framework (e.g., SGD, Adam, momentum, Adagrad) demonstrates its expressivity and capacity for dynamic adaptation during training (Lee et al., 6 Dec 2025).

3. Formal Representations and Extraction in Program Optimization

Expressive program optimization via equality saturation relies on e-graphs, which compactly encode congruence over program expressions. Rewrite rules are expressed with pattern variables and type constraints to integrate with multiple dispatch and Julia’s dynamic IR semantics (Merckx et al., 24 Feb 2025). Extraction of the optimal program after saturation is formulated as an Integer Linear Program (ILP) minimizing the cost of selected nodes, subject to constraints for dominator coverage, non-dangling references, root coverage, and acyclicity within the e-graph. This acyclic extraction enables reuse of computations across dominator trees and supports robust handling of both pure and effectful computations.

A key mechanism for accommodating side effects is the CFG skeleton relaxation, where effectful statements reside in an external skeleton unless explicitly removed by an "unsafe" rewrite. This approach lets the optimizer transform high-level, effectful code as well as pure computation, extending optimizer expressivity to the full IR domain.

4. Symbolic and Evolutionary Discovery of Optimizers

Symbolic equation learning frameworks for optimizer synthesis (e.g., Symbol, Symbolic L2O) operate by constructing update rules from a small set of grammatical tokens, with structure determined via LSTM-based generators or symbolic regression. In Symbol, the symbolic equation generator operates on contextual state (fitness landscape features, optimization history) and produces closed-form, task-adaptive updates (Chen et al., 2024). Discovered optimizers blend or surpass existing methods (e.g., DE, PSO), and meta-learned switching among rules enhances performance and generalization.

Symbolic Learning to Optimize (L2O) further distills complex neural optimizers into concise, interpretable closed-form rules via programmatic symbolic regression, maintaining competitive accuracy while permitting transparency and low overhead (Zheng et al., 2022). These methods allow for fine-grained analysis of temporal perception fields and mapping complexity, linking optimizer structure directly to convergence and generalization.

Evolutionary search over optimizer genomes encodes linear combinations of primitive update terms and hyperparameters, allowing for the discovery of hybrid optimizers with task-specific scheduling, sign/adaptive hybridization, and nonstandard parameterization (e.g., warmup as bias-correction alternative, low-momentum regimes) (Marfinetz, 5 Dec 2025). The search space's expressivity is limited primarily by the allowed composition and operator set.

5. Expressivity, Inductive Bias, and Solution Sets

Optimizers define induced solution sets $\mathcal{F}_{A,O}$, a subset of critical points that can be reached from initialization $\theta_0$ under the optimizer's dynamics and state (Pascanu et al., 16 Jul 2025). Expressive optimizers, whether via user-defined rewrites, meta-learned update laws, or evolved primitives, provide a principled mechanism for introducing or amplifying inductive biases: from bias toward flat minima, sparsity, or low-rank representations in deep networks, to the selection of fused, domain-optimized kernels in compiled code.

Empirical studies confirm that optimizer-induced biases meaningfully alter solution quality in domains such as continual learning (robustness to catastrophic forgetting via non-diagonal preconditioners), model sparsification (magnitude-throttled preconditioning), and feature orthogonality (low-dimensional hidden representations). Methodological tools include representation effective rank, feature cosine similarity, and weight histograms, which diagnose optimizer-expressed properties in trained models (Pascanu et al., 16 Jul 2025).

6. Design Principles, Trade-Offs, and Overheads

Expressive optimizers introduce inherent trade-offs:

• Complexity vs. control: Increased expressive power complicates theoretical analysis and may introduce pathological behaviors (e.g., infinite rule expansion, brittle meta-learning).
• Computational overhead: Program optimizers based on e-graphs and ILP extraction incur significant—but bounded—compilation time, typically dominated by e-graph saturation and solve time ($\sim$80 ms for complex Julia function graphs) (Merckx et al., 24 Feb 2025). Learned optimizers demand heavy meta-training (up to 4,000 TPU-months for VeLO), but inference overhead is decoupled by design (Metz et al., 2022).
• Interpretability vs. scalability: Symbolic optimizers trade off scalability with LSTM-based or hypernetworked meta-trained optimizers but permit white-box analysis of update dynamics (Zheng et al., 2022).
• Robustness and adaptation: Mechanisms like dynamic hyperparameter adaptation, stateful preconditioning, and meta-policy switching enhance robustness across tasks and distribution shifts (Lee et al., 6 Dec 2025, Metz et al., 2022, Chen et al., 2024).

In program optimization, expressive rule sets must be managed to avoid unsound rewrites, infinite expansion, or semantic divergence; timeout or rule-hierarchy mechanisms cap computational and behavioral expressivity as needed (Merckx et al., 24 Feb 2025).

7. Impact, Empirical Outcomes, and Open Directions

Expressive optimizers demonstrably unlock new performance regimes:

• Program optimization: Domain-specific rewrites, e-graph saturation, and fused kernel extraction achieve substantial simplifications and performance gains in both high-level and system code, enabling entire transformation chains to collapse into minimal forms under domain-specific rules (Merckx et al., 24 Feb 2025).
• Neural network optimization: Empirically, meta-learned and evolved optimizers outperform tuned hand-designed baselines by 1–7% in vision and language tasks, with significant acceleration in convergence and increased parameter scaling tolerance (Metz et al., 2022, Marfinetz, 5 Dec 2025). Zero-shot generalization across problem dimensions, populations, and domains is achieved with symbolic meta-learned optimizers (Chen et al., 2024).
• Interpretability: Symbolic frameworks permit plug-and-play deployment, transparent inspection, and analytic mapping from optimizer structure to behavior, supporting theory-guided design and debugging (Zheng et al., 2022).
• Automated optimizer design: Convex-analytical frameworks unify and systematically generalize classic optimizer families, offering dynamic, closed-form hyperparameter tuning and data-adaptive specialization (Lee et al., 6 Dec 2025).

A key research direction is systematic quantification and exploitation of optimizer-induced biases, advancing from convergence-focused evaluation to direct control of generalization, robustness, compositionality, and other qualitative properties via the optimizer itself (Pascanu et al., 16 Jul 2025). Cross-pollination between symbolic, learned, and evolutionary strategies for optimizer synthesis promises further advances in both expressivity and interpretability.