Efficient Saddle Point Escape in High Dimensions via Adaptive Perturbation and Subspace Descent

Abstract: We investigate high-dimensional non-convex optimization, focusing on the algorithmic difficulties posed by saddle points and regions of flat curvature. We develop a unified framework that integrates stochastic perturbations, curvature-adaptive learning rates, and randomized subspace descent to improve escape efficiency and scalability. Our theoretical analysis shows that gradient flow almost surely avoids strict saddles, with escape likelihood increasing exponentially in the ambient dimension. For noise-perturbed gradient descent, we derive explicit escape-time bounds that depend on curvature and noise magnitude. Adaptive step sizes further reduce escape time by adjusting to local gradient variance. To improve scalability, we establish global convergence rates for randomized subspace descent using projections of logarithmic dimension that preserve descent direction with high probability. Numerical experiments on nonlinear and constrained objectives validate these results and demonstrate practical robustness in large-scale settings.