Superquantile-Gibbs Relaxation for Minima-selection in Bi-Level Optimization

Abstract: Minima selection is essential for defining Bi-Level Optimization (BLO) when the lower-level objective has multiple minimizers. While BLO is intractable in its general form, we show that restricting the lower-level objective to the recently proposed PL-circle functions (Gong et al., 2024) guarantees a continuous hyper-objective F_max. The PL-circle condition is strictly weaker than the global Polyak-Lojasiewicz condition used in prior BLO work and allows modeling of practical settings such as hyperparameter tuning in over-parameterized deep learning. However, even under this condition, F_max remains non-convex and non-smooth. To address this, we propose a relaxed solution concept: we approximate F_max with a continuously differentiable function F_max_tilde that is pointwise epsilon_v-close to F_max and seek an epsilon_g-stationary point of F_max_tilde. In this framework, we reduce the minima-selection subproblem to a sampling task using a novel Superquantile-Gibbs relaxation. By leveraging the manifold structure of the lower-level solution set under the PL-circle condition, our method finds a relaxed solution using poly(epsilon_v^{-k} epsilon_g^{-1}) queries to a Gibbs sampling oracle, which is efficiently implemented using Langevin dynamics. Here, k is the intrinsic dimension of the manifolds defining the lower-level solutions. This is the first work to characterize the complexity of BLO in terms of this intrinsic dimensionality.