Truncated Power Method for Sparse Eigenvalue Problems

Abstract: This paper considers the sparse eigenvalue problem, which is to extract dominant (largest) sparse eigenvectors with at most $k$ non-zero components. We propose a simple yet effective solution called truncated power method that can approximately solve the underlying nonconvex optimization problem. A strong sparse recovery result is proved for the truncated power method, and this theory is our key motivation for developing the new algorithm. The proposed method is tested on applications such as sparse principal component analysis and the densest $k$-subgraph problem. Extensive experiments on several synthetic and real-world large scale datasets demonstrate the competitive empirical performance of our method.

• The paper introduces the truncated power method to compute sparse eigenvectors by iteratively truncating to maintain fixed sparsity.
• It establishes a theoretical framework showing that the method effectively recovers dominant eigenvectors under bounded perturbations.
• Empirical results on synthetic and real datasets highlight the method's efficiency and robustness in applications like sparse PCA.

Truncated Power Method for Sparse Eigenvalue Problems

The paper "Truncated Power Method for Sparse Eigenvalue Problems" by Xiao-Tong Yuan and Tong Zhang addresses the computational challenges associated with the sparse eigenvalue problem, a critical area in statistical machine learning with applications such as sparse PCA. This problem involves extracting the largest sparse eigenvectors, characterized by a fixed number $k$ of non-zero components, from a given symmetric positive semidefinite matrix $A$.

Problem Definition and Significance

The sparse eigenvalue problem is defined as maximizing $x^⊤Ax$ with the constraint that $x$ is a unit vector containing at most $k$ non-zero elements. The paper revisits this problem within the context of non-convex optimization, due to its NP-hard nature. Primarily motivated by the sparse PCA applications, the authors explore solutions that handle matrix perturbations and sparse recovery.

Proposed Method: Truncated Power Method

The authors propose a novel approach named the "Truncated Power Method" to approximate solutions to the non-convex optimization problem inherent in sparse eigenvalue extraction. This method extends the classical power iteration algorithm with a truncation operation ensuring the resulting vectors maintain desired sparsity. The Truncated Power Method simplifies computation by consistently truncating towards the $k$-largest entries, thereby iteratively refining $k$-sparse eigenvectors.

Theoretical Insights

A significant contribution of this research is the strong theoretical framework supporting the truncated power method's efficacy in sparse recovery. The authors establish that if the matrix $A$ has a sparse dominant eigenvector, under a suitable bounded spectral norm of submatrices of perturbations, the method can effectively recover approximate solutions. The paper shows that the quality of recovery is contingent upon the restricted matrix perturbation error, which grows as a function of sparsity $k$, unlike traditional eigenvalue problems dominated by matrix dimension $p$.

Empirical Evaluation

The paper provides empirical evidence through extensive experiments on both synthetic and real-world datasets, demonstrating that the truncated power method is competitive both in terms of performance and computational efficiency. In applications to sparse PCA and the densest $k$-subgraph problem, the proposed method showed robust recovery of sparse structures, underpinning theoretical claims with practical success. Furthermore, the algorithm's ability to handle various initializations adds versatility, illustrating the practical utility in different machine learning contexts.

Implications and Future Directions

The findings delineate new possibilities for addressing sparse eigenvalue issues in high-dimensional data contexts, potentially influencing methodologies in dimensionality reduction techniques like sparse PCA. Furthermore, the methodological framework offers a foundation for exploring other heuristic and approximate solutions to non-convex problems in machine learning. Future research might extend these theoretical insights to alternative sparse recovery frameworks or hybrid models combining convex relaxations with iterative algorithms, as well as investigate dynamic adaptations in multi-core or distributed computing environments to further enhance scalability.

In summary, the work constitutes an advancement in understanding and solving sparse eigenvalue problems, paving the way for more efficient algorithms in machine learning and statistical analysis. The truncated power method, with its theoretical grounding and empirical validation, stands out as a promising tool for researchers and practitioners dealing with high-dimensional data challenges.