Double Regression Analysis: Methods & Insights

• Double regression analysis is a statistical framework that integrates the study of double descent in overparameterized models with methods for handling doubly truncated and high-dimensional data.
• In the double descent context, risk peaks arise at the interpolation threshold and ridge regularization is shown to smooth variance explosions in estimators.
• Methodologies such as iterative L1 minimization and sequential screening deliver consistent, asymptotically normal estimators for truncated data and conditional independence testing.

Double regression analysis encompasses two distinct domains in the statistical literature: (i) the characterization and inference of the "double descent" phenomenon in regression and overparameterized models and (ii) methodologies for handling regression with doubly truncated data or via algorithmic "double regression" for high-dimensional conditional independence testing. Each usage reflects specific methodological and theoretical advances.

1. Double Descent in Regression

Double descent refers to the non-monotonic dependence of generalization error on model complexity, notably manifesting as a risk "peak" at the interpolation threshold, followed by a second decrease for highly overparameterized regimes. The prototypical context is ridgeless linear regression or minimum-norm least squares, with extensions to random features and kernel methods.

Formally, consider $n$ i.i.d. samples $(x_i,y_i)$ and a feature map $\phi:\mathbb{R}^d \to \mathbb{R}^p$. The double descent effect arises in the estimator

$\hat{f}(x) = \phi(x)^\top \Phi^+ y,$

where $\Phi$ is the $n \times p$ design matrix and $\Phi^+$ the Moore–Penrose pseudoinverse. Two principal regimes are distinguished:

• Underparameterized $(p<n)$: Risk lower bound $\sigma^2 \frac{p}{n+1-p}$, diverging as $p\uparrow n$.
• Overparameterized $(p>n)$: Risk lower bound $\sigma^2 \frac{n}{p+1-n}$, diverging as $p\downarrow n$.

At $p=n$, the lower bound becomes infinite in the presence of noise, resulting in an unavoidable generalization error peak—empirically and theoretically established as "double descent" (Holzmüller, 2020, McKelvey, 2023).

This phenomenon is universal under mild conditions:

• Full-rank feature matrices (almost surely)
• Invertible second-moment matrix $\Sigma$
• Non-zero noise variance

These results cover polynomial kernels, random Fourier features, and analytic random neural network feature maps. The Marchenko–Pastur law establishes asymptotic sharpness, matching empirical and closed-form finite-$n$ formulas.

2. Analysis of the Double Descent Peak

Bias–variance decomposition reveals that the double descent peak is rooted in the variance term's scaling as $1/\sigma_{\min}^2(X)$, where $\sigma_{\min}(X)$ is the smallest singular value of the design matrix. As model order grows to the sample size ($n=N$), the design matrix becomes nearly singular, inflating the estimator's variance and producing the classical risk peak. Once the model is overparameterized ($n>N$), the smallest singular value is bounded from below due to eigenvalue interlacing, and added redundancy dampens the explosion in variance, yielding a second test error descent (McKelvey, 2023, Holzmüller, 2020). Ridge regularization smooths or removes the peak: $$\hat{\beta}_{\lambda} = (X^T X + \lambda I)^{-1} X^T y.$$ Here, the variance term is bounded by $1/\lambda$, avoiding the double descent blowup.

3. Double Regression for Doubly Truncated Data

In a distinct context, double regression denotes regression methodologies developed for data subject to double truncation—where each observed response $Y_i$ is only retained if it lies within an observed, potentially individual-specific interval $(L_i, R_i)$. The statistical model is

$Y_i = \beta^\top X_i + \varepsilon_i$

with only those $i$ with $L_i < Y_i < R_i$ observed (Ying et al., 2017). The core estimation principle, generalizing the Mann–Whitney rank estimator, is: $$U_n(\beta) = \sum_{i<j} I\bigl\{L_j(\beta)<e_i(\beta)<R_j(\beta),\ L_i(\beta)<e_j(\beta)<R_i(\beta) \bigr\} (X_i - X_j) \operatorname{sgn}\{e_i(\beta) - e_j(\beta)\}.$$ This approach yields consistent and asymptotically normal estimators. A resampling scheme based on perturbation with random weights provides large-sample justified inference for the limiting distribution. The method is algorithmically implemented via iterative $L_1$ minimization on pairwise pseudo-observations, with software implementations in R (quantreg) and MATLAB.

Application to astronomical data (quasar luminosity evolution) demonstrates the practical utility, with model-based and nonparametric confidence intervals and $p$-values directly computed from the double regression estimator.

4. Double Regression Procedure for High-dimensional Conditional Independence

A third usage refers to algorithmic double regression for graphical model learning in high-dimensional nonlinear/non-Gaussian settings (Liang et al., 2022). The double regression method enables nonparametric conditional independence testing by dramatically reducing the conditioning set via two sequential regression/screening steps:

1. For node $i$, regress $X_i$ on $X_{V\backslash\{i\}}$ (screening or sparse deep neural network), producing candidate set $\widehat{S}_i$.
2. For each pair $(i,j)$, regress $X_j$ on $X_{V\backslash\{i,j\}}$, producing $\widehat{S}_{j\setminus i}$.
3. Test $X_i \perp X_j \mid X_{M_{ij}}$, where $$M_{ij} = (\widehat{S}_i \cup \widehat{S}_{j\setminus i}) \setminus \{j\}$$.

Theoretical guarantees are obtained under assumptions of Markov+faithfulness, polynomial dimensionality, uniform sure-screening, separation of null and alternative statistics, and tail-control. The methodology attains consistent variable selection, with computational feasibility superior to exhaustive conditioning, and outperforms a wide set of modern graph learning procedures in both simulated and real omics datasets.

Major Context	Main Result/Feature	Key Reference
Double Descent in Regression	Universal risk peak at interpolation under mild conditions	(Holzmüller, 2020, McKelvey, 2023)
Doubly Truncated Data	Iterative $L_1$ estimation with pairwise truncation indicators	(Ying et al., 2017)
High-Dim Double Regression	Sequential regression to shrink conditioning sets in independence tests	(Liang et al., 2022)

5. Methodological Considerations and Extensions

The double regression context requires careful distinction between the double descent regime of overparameterized models in classical regression and procedures explicitly termed "double regression" for addressing high-dimensionality or data truncation. Both fields involve compound or iterative regression steps to address nonstandard statistical challenges—overfitting in the presence of noise or inferential validity for incomplete or massive data.

The double descent analysis universally establishes that, in ridgeless estimators under full-rank/noise, no feature selection or kernel map can remove the central risk blow-up without regularization or alteration of the full-rank interpolation regime (Holzmüller, 2020). Conversely, in doubly truncated and high-dimensional independence models, double regression methodology enables unbiased and asymptotically consistent estimators for parameters or edge structure under complex sampling or dependence.

6. Practical Recommendations and Software

• Double regression estimators for doubly truncated data are implementable with standard quantile regression and $L_1$ minimization routines (e.g., R's quantreg::rq with paired differences). Resampling for inference is performed via repeated perturbation with random weights.
• For high-dimensional graphical models, the double regression approach leverages initial screening (Henze–Zirkler, sparse BNN) to restrict regression to feasible neighborhood sizes, facilitating powerful conditional independence testing compared to NOTEARS, DAG-GNN, and Gaussian methods.
• In regression with overparameterization, introducing ridge penalties smooths the double descent peak and restores classical variance control.

Pragmatically, double regression analysis fundamentally extends classical regression inference to settings of model complexity, limited or incomplete data, and massive variable interactions, with theoretical guarantees and computational viability across broad applied and theoretical contexts.