Order Parameter Bootstrapping in Model Selection

• Order parameter bootstrapping is a statistical framework that combines Lepski's method, multiple testing, and wild bootstrap calibration to control type I error.
• It rigorously selects the optimal bias-variance trade-off in linear estimators while achieving oracle-like risk bounds in finite samples.
• The method leverages pairwise test statistics and tailored thresholds to adaptively manage heteroscedastic noise without relying on strict parametric assumptions.

The order parameter bootstrapping argument is a statistical framework for model selection within a family of linear estimators, with a focus on controlling type I error in a completely data-driven manner that is robust to unknown and potentially heteroscedastic noise. The method, developed primarily in the context of ordered model selection for smoothing estimators, advances the "smallest accepted" approach, integrating Lepski's method with contemporary multiple testing theory and leveraging wild (multiplier) bootstrap procedures for threshold calibration. The result is a rigorous, finite-sample-valid, heteroscedasticity-robust procedure for selecting the optimal point along a sequence of bias-variance trade-offs, equipped with provable oracle‐type risk bounds and explicit error control (Spokoiny et al., 2015).

1. Framework: Ordered Linear Estimators and Model Selection

The foundational setup involves observed data $Y = \theta + \epsilon$, where $\theta \in \mathbb{R}^n$ is an unknown signal and $\epsilon$ is a zero-mean noise vector with covariance $\Sigma$, which may be heteroscedastic. Candidate estimators are constructed as $\tilde{\theta}_m = S_m Y$ for $m = 1,2,\ldots,M$, with fixed $n \times n$ smoothing matrices $S_m$. These estimators are ordered in terms of variance: $$\operatorname{Var}(\tilde{\theta}_1) \ll \operatorname{Var}(\tilde{\theta}_2) \ll \cdots \ll \operatorname{Var}(\tilde{\theta}_M)$$. Each model $m$ in the sequence offers a specific bias-variance trade-off, with lower-$m$ models generally incurring higher bias and lower variance, and higher-$m$ models the opposite. An acceptance rule formalizes statistical indistinguishability: model $m$ is accepted if it cannot be reliably distinguished from all larger models $k>m$, based on the observed data.

2. Pairwise Test Statistics and Acceptance Thresholds

For each pair $(m, k)$ with $k > m$, the difference $\tilde{\theta}_k - \tilde{\theta}_m$ is considered, with covariance $$\Delta_{m,k} = \operatorname{Var}(\tilde{\theta}_k - \tilde{\theta}_m) = (S_k - S_m)\Sigma(S_k - S_m)^T$$. Under the null hypothesis that model $m$ is true ($\theta$ in the column space of $S_m$), this difference is zero-mean with covariance $\Delta_{m,k}$. The primary diagnostic is the quadratic form

$$T_{m,k} = (\tilde{\theta}_k - \tilde{\theta}_m)^T \Delta_{m,k}^{+} (\tilde{\theta}_k - \tilde{\theta}_m),$$

where $\Delta_{m,k}^{+}$ is a suitable generalized inverse, and under the null, $T_{m,k}$ is approximately $\chi^2$-distributed with $\operatorname{rank}(\Delta_{m,k})$ degrees of freedom. A model $m$ is accepted if $T_{m,k} \leq z_{m,k}$ for all $k > m$, where $z_{m,k}$ are critical values selected to control the family-wise error rate of false rejections via the propagation condition.

3. Wild Bootstrap Calibration for Threshold Selection

A central challenge is that $\Sigma$ is both unknown and potentially heteroscedastic. The method addresses this through a multiplier (wild) bootstrap to approximate the joint distribution of all test statistics under each model $m$:

• A pilot estimator $S_0$ (often $S_M$) defines initial residuals $R = Y - S_0 Y$.
• For $B$ bootstrap iterations, i.i.d. mean-zero multipliers $\{\xi_i^{(b)}\}$ with $\operatorname{Var}(\xi_i) = 1$ are generated (e.g., Rademacher or standard normal).
• Bootstrap pseudo-observations are constructed as $$Y^{(b)} = S_0 Y + \operatorname{Diag}(R)^{1/2} \xi^{(b)}$$.
• The test statistics $T_{m,k}^{(b)}$ are recomputed for all $m \leq k$ on the bootstrapped data.
• The empirical distribution $\hat{F}_{m,k}$ of $\{T_{m,k}^{(b)}\}$ is used to set $z_{m,k} = \hat{F}_{m,k}^{-1}(1 - \alpha_{m,k})$, where the levels $\alpha_{m,k}$ are chosen so that $\sum_{m<k} \alpha_{m,k} \leq \alpha$, with $\alpha$ the desired overall error level.

This procedure ensures that for any true model $m_0$, the probability of any false rejection among $T_{m,k}$ for $m < m_0 < k$ is bounded by $\alpha$.

4. Selection Rule and Type I Error Control

The final selection rule is

$$\hat{m} = \min\{m = 1, \ldots, M : T_{m,k} \leq z_{m,k} \text{ for all } k = m+1, \ldots, M\}.$$

Equivalently, the process begins at the lowest index and selects the smallest model $m$ not rejected by any larger model. By the construction of $\alpha_{m,k}$ and the bootstrap-based thresholds, it follows that, for any $\theta$ in the column space of some $S_{m_0}$,

$$P_\theta(\hat{m} < m_0) \leq \alpha + C B^{-1/2} + C' n^{-1/2},$$

where $C$ and $C'$ are constants dependent on the Monte-Carlo and non-Gaussian tail behavior of the test statistics, respectively. This upper bound demonstrates finite-sample family-wise error control for under-smoothing.

5. Finite-Sample Validity and Oracle Inequalities

Under suitable conditions (fixed design, sub-Gaussian noise, moderate dimensionality; see assumptions A1–A3 in the original work), the method guarantees not only type I error control but also risk performance nearly matching the oracle model. Specifically, with probability at least $1 - (\alpha+o(1))$,

$$\|\hat{\theta} - \theta\|^2 \leq C \min_{1 \leq m \leq M} \left\{\|(I - S_m)\theta\|^2 + \operatorname{Trace}(\operatorname{Var}(\tilde{\theta}_m))\right\} + R_n,$$

where $R_n = O(n^{-1/2})$ is a negligible remainder. The result holds equally for full parameter estimation or for linear functionals. This demonstrates that the method is data-adaptive and achieves a nearly optimal bias-variance balance, subject only to small finite-sample corrections (Spokoiny et al., 2015).

6. Methodological Context and Significance

The order parameter bootstrapping argument builds directly on Lepski's method and the theory of multiple testing, operationalized in a model selection context through the "smallest accepted" approach. Its primary contributions are the rigorous bootstrapped calibration of critical values in the presence of heteroscedastic noise, the explicit finite-sample family-wise error control, and the explicit characterization of the selected index set. The framework is fully data-driven, requiring neither prior variance structure nor parametric assumptions on the noise. These features position it as a robust alternative to classical penalized criteria, particularly in scenarios with non-constant variance and complex dependence structures in the residuals (Spokoiny et al., 2015).