Front-Greater Weighting in Ensemble Methods

• Front-Greater Weighting is a structured ensemble method that assigns monotonically decaying weights to base learners ordered by complexity.
• It leverages spectral and geometric properties in RKHS to optimally balance bias, variance, and approximation error in ensemble learning.
• By enforcing ℓ2-bounded and monotonic constraints, the method reduces residual variance and outperforms uniform averaging in risk minimization.

Front-Greater Weighting is a structured ensemble weighting principle designed to optimize generalization performance by assigning monotonically decaying weights to base learners ordered by complexity. This approach deviates from traditional uniform averaging and classical variance reduction in ensemble methods, instead leveraging spectral and geometric properties of the hypothesis space—particularly effective for ensembles comprising stable, regularized base estimators in reproducing kernel Hilbert spaces (RKHS) (Fokoué, 25 Dec 2025).

1. Formal Framework for Structured Ensemble Weighting

Let $\mathscr{H} \subset L^2(P_X)$ denote the hypothesis space, equipped with inner product $$\langle f, g \rangle_{L^2} = \mathbb{E}_{X \sim P_X}[f(X)g(X)]$$. Consider an ordered dictionary of base learners $$\mathcal{D}_M = \{h_1, h_2, \dots, h_M\} \subset \mathscr{H}$$ such that $h_1$ is the simplest (lowest complexity) and $h_M$ is the most complex. The ensemble predictor for weights $w = (w_1, \dots, w_M) \in \mathbb{R}^M$ is given by:

$\hat{f}_w(x) = \sum_{m=1}^M w_m h_m(x).$

The admissible weighting space is defined as:

$$\mathcal{W}_M = \Big\{ w \in \mathbb{R}^M : w_m \ge 0, \; \sum_{m=1}^M w_m = 1, \; w_1 \ge w_2 \ge \cdots \ge w_M, \; \sum_{m=1}^M w_m^2 \le C_w \Big\},$$

where (W1) ensures nonnegativity and normalization, (W2) imposes front-greater monotonicity, and (W3) $\ell_2$-boundedness controls residual variance.

2. Refined Bias–Variance–Approximation Decomposition

Assume the regression target $f^\star \in \mathscr{H}$, $Y = f^\star(X) + \varepsilon$, $\mathrm{Var}(\varepsilon) = \sigma^2$ and expand both $\{h_m\}$ and $f^\star$ in orthonormal basis $\{\phi_k\}_{k \ge 1}$. Let $f^\star = \sum_{k \ge 1} \theta_k \phi_k$ and $h_m = \sum_{k \ge 1} a_{m,k} \phi_k$. The $k$-th mode under weighting $w$ is $b_k(w) = \sum_{m=1}^M w_m a_{m,k}$.

Theorem 3.1 yields:

$$\mathbb{E}\big\| \hat{f}_w - f^\star \big\|_{L^2}^2 = \underbrace{\sum_{k \ge 1} (b_k(w) - \theta_k)^2}_{\mathcal{A}(w)\;\text{(approximation geometry)}} + \underbrace{\sum_{k \ge 1} \mathrm{Var}(b_k(w))}_{\mathcal{V}(w)\;\text{(residual variance)}} + \sigma^2,$$

where for low-variance base learners, $\mathcal{V}(w)$ is dominated by $\sum w_m^2$. The primary terms become:

$$\mathcal{A}(w) = \sum_{k \ge 1} (b_k(w) - \theta_k)^2, \quad \mathcal{S}(w) = \sum_{k \ge 1} (\theta_k^2 - b_k(w)^2).$$

$\mathcal{A}(w)$ may be further decomposed to analyze underfitting and unrepresented modes, elucidating how $w$ reshapes hypothesis geometry.

3. Quadratic Programming for Optimal Structured Weights

The excess risk can be reformulated in matrix-vector notation:

$$\mathrm{Risk}(w) = \mathbb{E}\|\hat{f}_w - f^\star\|^2 = w^\top \Sigma w - 2c^\top w + \sigma^2,$$

where $\Sigma_{mm'} = \mathbb{E}[h_m(X) h_{m'}(X)]$ and $c_m = \mathbb{E}[h_m(X) f^\star(X)]$. Introducing an $\ell_2$ regularizer for strict convexity leads to the optimal weighting solution:

$$w^{\rm opt} = \arg\min_{w \in \mathcal{W}_M} \big\{ w^\top \Sigma w - 2c^\top w + \lambda\|w\|_2^2 \big\},$$

subject to front-greater and variance constraints, often tractable via convex optimization or reduced to parameter search in specific weighting families.

4. Dominance of Structured Over Uniform Weighting

Theorem 4.1 (Structured Weighting Dominance) stipulates: for a uniform weight $w^{\rm unif} = (1/M, \dots, 1/M)$, if there exists $w^\star \in \mathcal{W}_M$ such that:

• (C1) Strict approximation gain: $$\|f^\star - \Pi_{H_{w^\star}} f^\star \|_{L^2}^2 < \|f^\star - \Pi_{H_{w^{\rm unif}}} f^\star \|_{L^2}^2$$
• (C2) Controlled variance: $\|w^\star\|_2^2 \le \|w^{\rm unif}\|_2^2$

then ensemble risk is strictly improved:

$$\mathbb{E}\|\hat{f}_{w^\star} - f^\star\|^2 < \mathbb{E}\|\hat{f}_{w^{\rm unif}} - f^\star\|^2.$$

Theorem 4.3 demonstrates that under spectral decay $\lvert \theta_k \rvert \le C k^{-\alpha}$ ($\alpha > \frac{1}{2}$), there always exists a monotone, geometrically decaying $w^\star$ with strictly lower risk than uniform averaging.

5. Explicit Front-Greater Weighting Laws

Several parametric families enforce $w_1 \ge w_2 \ge \cdots \ge w_M$, each normalizing to $\sum w_m=1$:

Law	Parametric Form	Decay Behavior
Uniform	$w_m = 1/M$	No decay
Geometric	$w_m \propto \rho^{-m}$, $\rho > 1$	Exponential
Polynomial	$w_m \propto m^{-\alpha}$, $\alpha > 1$	Harmonic/Power Law
Sub-exponential	$w_m \propto \exp(-c m^\beta)$, $0<\beta<1$, $c>0$	Sub-exponential
Heavy-tailed	$w_m \propto m^{-(1+s)}$, $s > 0$	Pareto/Zipf/Heavy-tail
Fibonacci-based	$w_m^{\rm Fib} = F_{M+1-m}/\sum_{j=1}^M F_{M+1-j}$	Decay $\sim \varphi^{-m}$

Each pattern preserves monotonicity, with low-complexity learners prioritized.

6. Parameter Selection and Implementation in Practice

Parameterization should reflect the spectral decay of target coefficients: for $\theta_k = O(k^{-\alpha})$, set polynomial exponent $\alpha_w \approx \alpha$ or choose geometric rate $\rho$ so that the effective cutoff $K(\rho) \approx \frac{\log(M)}{\log(\rho)}$ balances between unrepresented tail and underfitting. Trade-off curves indicate that faster decay reduces residual variance but risks underfitting, while slower decay improves expressivity at variance cost.

Direct monitoring of $\mathcal{A}(w) + \mathcal{S}(w) + \mathcal{V}(w)$ or using cross-validation on held-out risk is recommended for operational tuning. Fibonacci weighting often approximates the Pareto-optimal intersection of expressivity and stability, serving as a robust default. Optimization over $\mathcal{W}_M$ is tractable via convex solvers or parameter search within geometric families.

---

Front-greater weighting provides a principled geometric and spectral framework for ensemble learning with ordered, regularized RKHS base learners, rigorously establishing conditions under which monotone-decay (front-greater) patterning outperforms uniform averaging through reshaped approximation geometry and spectral allocation (Fokoué, 25 Dec 2025).