Variance Thresholding & Retained-Set PCA

• Variance thresholding and retained-set PCA are techniques that recover the support of sparse principal components in high dimensions by thresholding empirical variances and covariances.
• Covariance-thresholding refines the approach by incorporating off-diagonal information and leveraging probabilistic bounds to expand the feasible support regime for accurate recovery.
• These methods combine computational tractability with strong theoretical guarantees, using eigenanalysis on a retained coordinate set under the spiked covariance model.

Variance thresholding and retained-set principal component analysis (PCA) are algorithmic strategies for sparse PCA in high-dimensional settings, wherein the goal is to recover the support of sparse principal components (PCs) from noisy observations. These approaches combine thresholding of empirical variances or covariances with eigenanalysis restricted to a "retained" coordinate set, providing both computational tractability and theoretical guarantees for support recovery under the spiked covariance model. Covariance thresholding refines classical variance-thresholding by incorporating off-diagonal information and leveraging advanced probabilistic bounds, substantially expanding the feasible support size regime for accurate recovery (Deshpande et al., 2013).

1. Statistical Model: Sparse PCA and Spiked Covariance Framework

The core model for sparse PCA considered is the "spiked covariance" formulation. One observes $n$ independent samples $X_1,\ldots,X_n \in \mathbb{R}^p$:

$$X_i = \sum_{q=1}^r \sqrt{\beta_q} u_{q,i} v_q + \xi_i$$

where $v_1, \ldots, v_r \in \mathbb{R}^p$ are orthonormal population PCs, each with $\|v_q\|_0 \leq s_0$ ("$s_0$-sparse"), $u_{q,i} \sim \mathcal{N}(0,1)$, and $\xi_i \sim \mathcal{N}(0, I_p)$ are independent Gaussian noise vectors. The population covariance is thus $$\Sigma = \mathbb{E}[X_i X_i^\top] = I_p + \sum_{q=1}^r \beta_q v_q v_q^\top$$. Of particular interest is the high-dimensional regime, $p$ comparable to or much larger than $X_1,\ldots,X_n \in \mathbb{R}^p$0. The principal statistical objective is precise recovery of the supports $X_1,\ldots,X_n \in \mathbb{R}^p$1.

2. Methodologies: Classical Variance-Thresholding vs. Covariance-Thresholding

Two principal retained-set PCA algorithms address sparse PC recovery in this setting:

A. Variance-Thresholding (Johnstone–Lu Diagonal Method)

• Compute the empirical covariance $X_1,\ldots,X_n \in \mathbb{R}^p$2.
• Let $X_1,\ldots,X_n \in \mathbb{R}^p$3. For threshold $X_1,\ldots,X_n \in \mathbb{R}^p$4, define the retained index set:

$X_1,\ldots,X_n \in \mathbb{R}^p$5

(Adjustments may be made for non-unit noise variance via centering or alternative baselines.)

• Restrict $X_1,\ldots,X_n \in \mathbb{R}^p$6 to the principal submatrix on $X_1,\ldots,X_n \in \mathbb{R}^p$7, then compute its top-$X_1,\ldots,X_n \in \mathbb{R}^p$8 eigenvectors, padding with zeros off $X_1,\ldots,X_n \in \mathbb{R}^p$9 for estimates $$X_i = \sum_{q=1}^r \sqrt{\beta_q} u_{q,i} v_q + \xi_i$$0.
• Optional secondary thresholding on $$X_i = \sum_{q=1}^r \sqrt{\beta_q} u_{q,i} v_q + \xi_i$$1 can enforce exact sparsity.

B. Covariance-Thresholding (Deshpande–Montanari Method)

• Split data into two halves. From the first half, form $$X_i = \sum_{q=1}^r \sqrt{\beta_q} u_{q,i} v_q + \xi_i$$2, and subtract the identity: $$X_i = \sum_{q=1}^r \sqrt{\beta_q} u_{q,i} v_q + \xi_i$$3.
• Entrywise apply soft-thresholding at $$X_i = \sum_{q=1}^r \sqrt{\beta_q} u_{q,i} v_q + \xi_i$$4:

$$X_i = \sum_{q=1}^r \sqrt{\beta_q} u_{q,i} v_q + \xi_i$$5

• Extract the top $$X_i = \sum_{q=1}^r \sqrt{\beta_q} u_{q,i} v_q + \xi_i$$6 eigenvectors $$X_i = \sum_{q=1}^r \sqrt{\beta_q} u_{q,i} v_q + \xi_i$$7 of $$X_i = \sum_{q=1}^r \sqrt{\beta_q} u_{q,i} v_q + \xi_i$$8.
• On the second half, compute $$X_i = \sum_{q=1}^r \sqrt{\beta_q} u_{q,i} v_q + \xi_i$$9 and coordinate-wise scores $v_1, \ldots, v_r \in \mathbb{R}^p$0.
• For threshold $v_1, \ldots, v_r \in \mathbb{R}^p$1, define the retained set:

$v_1, \ldots, v_r \in \mathbb{R}^p$2

• Final support estimates are either principal eigenvectors of $v_1, \ldots, v_r \in \mathbb{R}^p$3 restricted to $v_1, \ldots, v_r \in \mathbb{R}^p$4 or thresholded $v_1, \ldots, v_r \in \mathbb{R}^p$5.

3. Support Recovery Theory

The statistical guarantees describe the support size regimes in which high-probability recovery is assured, and corresponding thresholds that yield optimal behavior:

Method	Support Size Regime	Threshold Choice	Guarantee
Variance-thresholding + retained PCA	$v_1, \ldots, v_r \in \mathbb{R}^p$6	$v_1, \ldots, v_r \in \mathbb{R}^p$7	$v_1, \ldots, v_r \in \mathbb{R}^p$8
Covariance-thresholding + retained PCA	$v_1, \ldots, v_r \in \mathbb{R}^p$9	$\|v_q\|_0 \leq s_0$0 (or $\|v_q\|_0 \leq s_0$1), $\|v_q\|_0 \leq s_0$2	$\|v_q\|_0 \leq s_0$3

Here, $\|v_q\|_0 \leq s_0$4 are universal positive constants, $\|v_q\|_0 \leq s_0$5 is the minimal magnitude parameter on the support of $\|v_q\|_0 \leq s_0$6, and $\|v_q\|_0 \leq s_0$7 indicates asymptotic proportionality.

A key finding is that covariance thresholding expands the feasible support regime for exact recovery from $\|v_q\|_0 \leq s_0$8 to $\|v_q\|_0 \leq s_0$9, under mild moment conditions and in certain $s_0$0 scenarios. The sample complexity bound in the rank-one case is $s_0$1, where $s_0$2 is universal. Lower bounds from computational complexity theory indicate that no polynomial-time algorithm can succeed for substantially larger $s_0$3 (Deshpande et al., 2013).

4. Analysis Techniques and Spectral Norm Bounds

The proof of tight support recovery relies on precise control of the spectral norm of the randomized matrices arising from the noise in $s_0$4 and the effect of entrywise thresholding. Given $s_0$5 cross-terms $s_0$6, soft-thresholding balances retention of the rank-one signal and substantial attenuation of the noise. The challenging task is bounding $s_0$7, where $s_0$8 is non-Lipschitz.

Earlier results addressed asymptotic spectral distributions or smooth functions; here, a non-asymptotic tail bound for the given kernel random matrix is required. The authors develop an $s_0$9-net argument combined with a new concentration lemma for non-Lipschitz Gaussian functionals, controlling $u_{q,i} \sim \mathcal{N}(0,1)$0 via discretization of the sphere and analyzing typical Gaussian behavior versus rare bad events (Deshpande et al., 2013).

5. Practical Considerations and Threshold Selection

In applied settings, the theoretical threshold prescriptions depend on unknown quantities such as $u_{q,i} \sim \mathcal{N}(0,1)$1 and $u_{q,i} \sim \mathcal{N}(0,1)$2. Suggested heuristics are:

• Threshold $u_{q,i} \sim \mathcal{N}(0,1)$3: Select $u_{q,i} \sim \mathcal{N}(0,1)$4; when $u_{q,i} \sim \mathcal{N}(0,1)$5 this is $u_{q,i} \sim \mathcal{N}(0,1)$6.
• Pragmatic Rule: Use an estimated noise-variance level: $u_{q,i} \sim \mathcal{N}(0,1)$7, where $u_{q,i} \sim \mathcal{N}(0,1)$8 is estimated from the median absolute deviation (MAD) of off-diagonal or diagonal elements, with $u_{q,i} \sim \mathcal{N}(0,1)$9.
• Spectral diagnostics via the Marchenko–Pastur edge location on the thresholded matrix can also guide $\xi_i \sim \mathcal{N}(0, I_p)$0 selection.

The table below summarizes the main computational steps and their leading costs for covariance-thresholding:

Step	Operation	Complexity
1	Split, form $\xi_i \sim \mathcal{N}(0, I_p)$1, $\xi_i \sim \mathcal{N}(0, I_p)$2	$\xi_i \sim \mathcal{N}(0, I_p)$3
2	Soft-threshold $\xi_i \sim \mathcal{N}(0, I_p)$4	$\xi_i \sim \mathcal{N}(0, I_p)$5
3	Top-$\xi_i \sim \mathcal{N}(0, I_p)$6 eigenvectors	$\xi_i \sim \mathcal{N}(0, I_p)$7 or $\xi_i \sim \mathcal{N}(0, I_p)$8
4	Form $\xi_i \sim \mathcal{N}(0, I_p)$9, actions on $$\Sigma = \mathbb{E}[X_i X_i^\top] = I_p + \sum_{q=1}^r \beta_q v_q v_q^\top$$0	$$\Sigma = \mathbb{E}[X_i X_i^\top] = I_p + \sum_{q=1}^r \beta_q v_q v_q^\top$$1
5	Submatrix eigendecomposition	$$\Sigma = \mathbb{E}[X_i X_i^\top] = I_p + \sum_{q=1}^r \beta_q v_q v_q^\top$$2

For $$\Sigma = \mathbb{E}[X_i X_i^\top] = I_p + \sum_{q=1}^r \beta_q v_q v_q^\top$$3 scenarios, randomized SVD or low-rank subspace iterations may provide efficient alternatives.

6. Connections to Literature and Limitations

Variance-thresholding and retained-set PCA, originally proposed by Johnstone and Lu, established foundational support-recovery thresholds in sparse PCA [Johnstone–Lu 2004]. Covariance-thresholding, as developed by Deshpande and Montanari, achieves strictly larger support recovery regimes without increased computational cost, and its guarantees match known computational lower bounds [berthet2013computational, ma2015sum]. These advances synthesize ideas from high-dimensional statistics, spectral random matrix theory, and computational complexity. No practical, polynomial-time algorithm currently exhibits better support recovery guarantees for the spiked covariance model under standard noise assumptions (Deshpande et al., 2013).

7. Summary and Contemporary Relevance

Variance-thresholding and retained-set PCA remain essential techniques for sparse principal component estimation in high dimensions. Covariance-thresholding generalizes diagonal thresholding, effectively leveraging both diagonal and off-diagonal information, and is distinguished by optimal support-recovery, robust performance in the high-dimensional regime, and refined non-asymptotic analysis. The introduction of sharp spectral norm bounds for thresholded random kernel matrices is a substantial technical contribution and informs the theoretical underpinnings of related algorithms in high-dimensional statistics and machine learning (Deshpande et al., 2013).