Topological Exploration of High-Dimensional Empirical Risk Landscapes: general approach, and applications to phase retrieval

Abstract: We consider the landscape of empirical risk minimization for high-dimensional Gaussian single-index models (generalized linear models). The objective is to recover an unknown signal $\boldsymbolθ^\star \in \mathbb{R}^d$ (where $d \gg 1$) from a loss function $\hat{R}(\boldsymbolθ)$ that depends on pairs of labels $(\mathbf{x}i \cdot \boldsymbolθ, \mathbf{x}_i \cdot \boldsymbolθ^\star){i=1}^n$, with $\mathbf{x}_i \sim \mathcal{N}(0, I_d)$, in the proportional asymptotic regime $n \asymp d$. Using the Kac-Rice formula, we analyze different complexities of the landscape -- defined as the expected number of critical points -- corresponding to various types of critical points, including local minima. We first show that some variational formulas previously established in the literature for these complexities can be drastically simplified, reducing to explicit variational problems over a finite number of scalar parameters that we can efficiently solve numerically. Our framework also provides detailed predictions for properties of the critical points, including the spectral properties of the Hessian and the joint distribution of labels. We apply our analysis to the real phase retrieval problem for which we derive complete topological phase diagrams of the loss landscape, characterizing notably BBP-type transitions where the Hessian at local minima (as predicted by the Kac-Rice formula) becomes unstable in the direction of the signal. We test the predictive power of our analysis to characterize gradient flow dynamics, finding excellent agreement with finite-size simulations of local optimization algorithms, and capturing fine-grained details such as the empirical distribution of labels. Overall, our results open new avenues for the asymptotic study of loss landscapes and topological trivialization phenomena in high-dimensional statistical models.

• The paper introduces a scalar variational framework leveraging the Kac-Rice method to explicitly characterize critical points in high-dimensional phase retrieval.
• It demonstrates that BBP instability in the Hessian can signal optimization success prior to landscape trivialization, redefining recovery thresholds.
• Numerical gradient descent experiments validate predicted energy bands, Hessian spectra, and non-Gaussian label distributions, confirming the theoretical framework.

Topological Exploration of High-Dimensional Empirical Risk Landscapes: Landscape Geometry, BBP Instability, and Dynamics for Phase Retrieval

Introduction and Theoretical Foundations

The paper systematically investigates the geometry of empirical risk landscapes in high-dimensional Gaussian single-index models, focusing on the phase retrieval problem. The authors employ the Kac-Rice method to rigorously characterize the statistical and topological features of critical points—including local minima and sub-extensive-index saddles—of the empirical risk function $\hR(\btheta)$ defined as an average over random labels $(\bx_i \cdot \btheta, \bx_i \cdot \btheta^\star)$ with i.i.d. Gaussian $\bx_i$ in the asymptotic regime $n \asymp d$, $d \gg 1$. The approach generalizes beyond Gaussian random landscapes to realistic empirical losses in machine learning that are sums over data, such as generalized linear models.

A central contribution is the reduction of previously complex variational expressions for landscape complexities to explicit scalar variational problems with a finite set of parameters, enabling efficient numerical analysis and algorithmic exploration. The paper develops detailed asymptotic formulas for the annealed complexity (expected log-number) of all critical points, saddles of sub-extensive index, and a tighter upper bound for local minima. This refinement captures additional constraints—specifically, the requirement that the signal direction is not a descent direction at a local minimum—which is crucial for accurately identifying topological trivialization thresholds in phase retrieval.

The Kac-Rice-generated variational problems not only yield landscape complexity but also provide access to spectral properties of the loss Hessian at critical points and to the limiting empirical joint law of predicted and ground-truth labels. The scalar formulations are solved numerically, facilitating thorough exploration of loss landscape transitions and the nature of critical points in high-dimensions.

BBP Instability and the BBP-Kac-Rice Method

A key theoretical advance is the extension of Kac-Rice analysis via random matrix theory, specifically the Baik–Ben Arous–Péché (BBP) transition, to detect instabilities—negative outlier eigenvalues aligned with the signal direction—in the Hessian at typical critical points. The BBP-KR method enables computation of precise phase diagrams, identifying regions where critical points are stable or unstable, and exposes that instability can arise prior to vanishing complexity predicted by Kac-Rice alone.

Figure 1: In the phase retrieval problem, the Hessian density at typical minima predicts a negative "BBP" outlier, evidencing an instability aligned with the signal.

This analysis reveals that minima may acquire descents toward the signal before their complexity vanishes, implying that optimization success is more closely tied to BBP instability than the disappearance of minima itself. The method generalizes landscape analysis to arbitrary overlaps, quantitatively relates geometry to optimization dynamics, and provides thresholds for trivialization and algorithmic recovery.

Landscape Complexity and Phase Diagrams for Phase Retrieval

Application to real phase retrieval yields comprehensive phase diagrams characterizing thresholds for trivialization and BBP instability as functions of sample complexity $\alpha = n/d$ and overlap $q$.

Figure 2: For $a = 0.01$, $q = 0$, and increasing $\alpha$, the complexity $\tilde{\Sigma}_0$ as a function of loss, Hessian spectrum, and joint label distribution, revealing narrowing energy bands and trivialization.

The analysis demonstrates that for small $\alpha$, local minima exist in a distinct energy band, with predicted Hessians always marginally stable. As $\alpha$ increases, this band narrows and disappears at a finite $\alpha_{\text{triv}}$, signifying topological trivialization. Notably, the predicted joint law of labels at minima is sharply non-Gaussian, showing strong alignment with ground-truth even at zero overlap with the signal, contrasting with random configurations.

Figure 3: For $a = 0.01$, $q = 0.4$, complexities, Hessian spectra, and joint label distributions for critical points, illustrating trivialization transitions and alignment phenomena.

Figure 4: For $a = 0.01$, $q = 0.0$, the energy of typical, lowest, and highest minima versus $\alpha$, and corresponding Hessian densities; lowest-energy minima remain most stable under BBP instability.

The phase diagrams map the regions where local minima and finite-index saddles trivialize and delineate the progression of BBP-instability through the landscape.

Figure 5: For $q=0$, BBP thresholds for minima (high, typical, lowest-energy) and trivialization as $\alpha$ increases; BBP instability propagates through the energy landscape ahead of complexity vanishing.

Numerical Validation: Gradient Descent Dynamics

Theoretical predictions are extensively validated against numerical gradient descent in $d=512$ dimensions. Modified dynamics suppressing finite-size corrections are employed, and convergence properties are analyzed for varying $\alpha$ and $q$.

Figure 6: For $q \in \{0.0, 0.1, 0.2\}$, empirical energies of GD minima versus $\alpha$, aligning with theoretically predicted energy bands.

Empirically, energies of minima reached by GD closely track the typical energy predicted by Kac-Rice, and GD minima properties—Hessian spectra, second derivatives, and joint label distribution—are in remarkable agreement with theoretical predictions, including fine-grained non-Gaussian structural features.

Figure 7: For $\alpha = 3.5$, comparison of Hessian eigenvalue distributions, second derivative weights, and joint label distributions between GD minima and Kac-Rice predictions.

Figure 8: For $\alpha=4.5$, further comparison, confirming predictive accuracy for bulk and tails.

Algorithmic phase diagrams extracted from GD success rates map directly onto the BBP-KR predicted transitions, confirming that the onset of instability, rather than trivialization, controls practical recovery thresholds.

Figure 9: Complete $(\alpha, q)$ phase diagram for $a=0.1$, mapping trivialization and BBP-instability thresholds.

Contradictory and Bold Claims

The paper asserts that:

• The complexity of sub-extensive-index saddles never trivializes at any finite $\alpha$ for $q=0$.
• BBP instability of minima can occur in a regime where Kac-Rice predicts exponentially many minima, thus optimization success can be achieved well before landscape trivialization.
• Empirical label distributions at local minima exhibit highly non-Gaussian, aligned structure even at zero overlap, contradicting expectations from random high-dimensional geometry.

Practical and Theoretical Implications

Practically, the framework provides explicit formulas and robust numerical algorithms for landscape exploration in arbitrary single-index models, extending to logistic regression and other GLMs. Theoretical implications include a direct link between critical-point geometry and optimization dynamics, refined understanding of topological trivialization, and the identification of BBP instability as a controlling threshold for algorithmic recovery.

Future developments could entail rigorous mathematical proofs of the scalar variational formulations, extension to quenched complexities, improved characterization of basins of attraction, and systematic study of uniqueness and phase transitions in landscape solutions. The methodology is readily generalizable to multi-index models and other non-convex statistical estimation problems.

Conclusion

This work delivers a comprehensive framework for analyzing high-dimensional empirical risk landscapes, connects geometric thresholds to optimization performance, and quantitatively validates its predictions against numerical gradient descent. The scalar variational principle and BBP-KR method constitute powerful tools for phase diagram analysis in high-dimensional statistical models, with broad applicability and deep implications for loss landscape theory and optimization in AI.