Neural Optimization Framework BOOST

• BOOST is a neural optimization framework that leverages boosting-inspired functional preconditioning and per-neuron whitening to enhance training efficiency.
• It employs an optimal least squares solution and input whitening to achieve rapid, well-conditioned descent for neural networks.
• Empirical validations on architectures like MLPs, CNNs, and transformers show accelerated convergence and improved robustness compared to traditional optimizers.

The Neural Optimization Framework BOOST encompasses a set of principled methodologies and algorithmic frameworks aimed at improving neural network training or inference through boosting-inspired strategies, preconditioned optimization, and, in its most formal instantiations, sophisticated functional space descent. Although the term "BOOST" appears in various contexts, this article focuses on the detailed methodologies introduced in "Simple Linear Neuron Boosting" (Munoz, 3 Feb 2025), which formalizes BOOST as a global, per-neuron functional preconditioning framework for differentiable networks. This approach stands apart from classical boosting (ensemble-based) and instead centers on boosting-inspired optimization in the function space of neurons, yielding a mechanism for rapid, well-conditioned descent applicable to modern large-scale architectures.

1. Functional Gradient Descent and the BOOST Paradigm

BOOST in the formulation of (Munoz, 3 Feb 2025) originates from a functional perspective: rather than optimizing network parameters directly using their gradients, one seeks the optimal function increment at each neuron compatible with the linearity of the neuron in its inputs. Consider a network composed of $m$ neurons, each represented as $f_i(x; w_i) = w_i^\top x$, where $w_i$ denotes the trainable weights and $x$ is the input. The collective parameter vector is $\theta_B = \{w_1, \dots, w_m\}$. The expected loss is $L(\theta_B) = E_{(x, y)}[\ell_y(F(x; \theta_B))]$.

The key insight is to project the functional gradient (the backpropagated error with respect to the neuron's output) onto the space of admissible linear functions via an optimal least squares (OLS) problem for each neuron: $$\hat w_i = \arg\min_w \frac12\,E\left[ \left\| w^\top x_{i-1} - \lambda_i \right\|^2 \right],$$ where $\lambda_i = \partial L / \partial x_i$ is the backpropagated error and $x_{i-1}$ is the input to neuron $i$. The OLS solution leads to the normal equations: $$M_i\,\hat w_i = g_i,\qquad M_i = E[x_{i-1} x_{i-1}^\top],\qquad g_i = E[x_{i-1} \lambda_i^\top].$$ Consequently, the per-neuron update $\hat w_i = M_i^{-1} g_i$ is stacked into a global block-diagonal preconditioner $P$, yielding the preconditioned step

$\Delta \theta_B = -\eta P \nabla_{\theta_B} L.$

This paradigm reframes classic parameter-space descent as metric-corrected gradient descent in function space, precisely tailored to each neuron's statistics.

2. Preconditioning as Input Whitening

The BOOST update is not only an optimizer-specific modification but is mathematically equivalent to whitening each neuron's inputs with respect to its observed second moment. When a bias term is present, for $w_i^\top [x; 1]$, the corresponding covariance structure is

$$M_i = E\left[ \begin{pmatrix} x \ 1 \end{pmatrix} (x^\top, 1) \right] = \begin{pmatrix} \Sigma_i + \mu_i \mu_i^\top & \mu_i \ \mu_i^\top & 1 \end{pmatrix},$$

where $\mu_i = E[x]$ and $\Sigma_i = \text{Cov}[x]$. The inverse $M_i^{-1}$ can be factored as $W_i W_i^\top$, with

$$W_i = \begin{pmatrix} \Sigma_i^{-1/2} & 0 \ -\mu_i^\top \Sigma_i^{-1/2} & 1 \end{pmatrix}.$$

By reparameterizing the neuron in terms of whitened features $\phi = W_i [x; 1]$, the learning dynamics correspond to canonical (white) features, but this normalization is only present during optimization; at inference, only the reparameterized weights are required because the transformation is absorbed. This mechanism ensures improved conditioning and invariance to affine transformations of neuron inputs, without introducing runtime normalization layers.

3. Online, Matrix-Free Preconditioning via EMA and Conjugate Gradient

Direct computation of second-moment matrices $M_i$ is infeasible in streaming or large-scale settings. BOOST addresses this by using per-mini-batch exponential moving averages (EMA): $$\mu_i^{(t)} = (1-\alpha)\mu_i^{(t-1)} + \alpha \bar x_i^{(t)},\qquad \chi_i^{(t)} = (1-\alpha)\chi_i^{(t-1)} + \alpha \overline{(x_i^{(t)} \odot x_i^{(t)})}.$$ The diagonal of the covariance is approximated by $\chi_i - \mu_i \odot \mu_i$. Matrix-vector products (MVP) with $M_i$ are computed using forward and reverse-mode autodifferentiation (JVP+VJP), and the linear system $M_i \hat w_i = g_i$ is solved approximately using a few conjugate gradient steps, further preconditioned via diagonal/incomplete-Cholesky of the EMA statistics.

The batch-wise procedure is as follows:

1. Compute gradients via backpropagation and cache neuron inputs.
2. Update EMA of gradients; update $\mu_i, \chi_i$.
3. Construct preconditioners from current EMA.
4. Define MVP via JVP+VJP.
5. Solve for $\hat w_i$ with (preconditioned) conjugate gradient.
6. Assemble global direction from all neurons.
7. Update parameters using adaptive step size.

The EMA ensures that, for small enough $\alpha$, the estimated moments are statistically consistent, and a small number of CG steps suffices to align updates with the dominant curvatures of $M_i$.

4. Trust-Region Adaptive Step Size

BOOST employs an explicit metric-based trust region strategy for step size selection. The optimal step under a quadratic trust-region of size $\epsilon$ is: $$\arg\min_{\delta\theta} L(\theta) + \nabla L \cdot \delta\theta \quad \text{s.t.} \quad \frac12 \langle \delta\theta, \delta\theta \rangle_{M} = \epsilon,$$ which yields $\delta\theta \propto M^{-1} \nabla L$. The step size is set as

$$\eta = \sqrt{\frac{\epsilon}{(\hat\theta_B^\top \nabla L)}}$$

with clamping for numerical stability ($z \ge z_0 > 0$), to prevent explosion of $\eta$ in degenerate cases.

5. Applicability to Modern Neural Architectures

BOOST is designed generically so long as autodifferentiation (reverse and forward) is available. Notably:

• Convolutional layers, being linear in their weights, are accommodated by appropriate spatial covariance calculation.
• Transformer architectures: embedding layers and class token projections typically have identity input statistics ($M_i \approx I$), so BOOST defaults to vanilla gradient steps in those layers.
• Scale and affine parameters for normalization layers (LayerNorm, BatchNorm) can be absorbed into the whitening framework: BOOST normalizes pre-activations outside of inference, avoiding the introduction of additional normalization layers at test time.

6. Experimental Validation and Conditioning Benefits

BOOST demonstrates consistently improved convergence speed, both in epochs and wall-clock time, across a spectrum of architectures and tasks:

• Matrix factorization (ill-conditioned linear nets): BOOST converges in $\approx 10$ iterations, Adam requires $\approx 1000$; wall-clock speedup is $20\times$.
• MLP on MNIST: exceeds $98.5\%$ test accuracy in $5$ epochs, compared to Adam's $10$; exhibits invariance to pixel inversion, a regime where Adam fails due to sensitivity to affine feature shifts.
• Vision Transformer on CIFAR-10: $50$ epochs to convergence versus $80$ for Adam, with a $+1.2\%$ absolute test accuracy gain.
• UNet on VOC Segmentation: BOOST attains $75\%$ mIoU after $30$ epochs, Adam needs $50$; per-epoch time is $2.9$min for BOOST, $1.3$min for Adam; final accuracy is comparable.

A central finding is that per-neuron whitening (via OLS preconditioning) is the direct source of improved convergence rates and stability. BOOST's trust-region step size further guarantees robust descent even under challenging data statistics.

7. Theoretical and Practical Significance

BOOST, as established in (Munoz, 3 Feb 2025), differs fundamentally from ensemble boosting: it is not primarily about aggregation of multiple weak learners, but rather about function space descent that is tightly aligned with the data geometry at the level of each neuron. The algorithm's design naturally induces invariances (e.g., to input normalization, affine transformations) conventionally obtained via batch or layer normalization, but without modifying the model's structure at inference.

This approach unifies a spectrum of prior ideas: functional gradient boosting, second-order preconditioners, and online (matrix-free) adaptation, into a training algorithm applicable to a broad range of differentiable architectures with minimal modification. Its empirical advantages are pronounced for ill-conditioned, high-dimensional, or input-shift-affected tasks. BOOST represents a new class of functional optimization algorithms for neural networks, with performance and stability enhancements stemming from explicit per-neuron whitening and trust-region adaptation (Munoz, 3 Feb 2025).