Optimal Zeroth-Order Bilevel Optimization

Abstract: In this paper, we develop zeroth-order algorithms with provably (nearly) optimal sample complexity for stochastic bilevel optimization, where only noisy function evaluations are available. We propose two distinct algorithms: the first is inspired by Jacobian/Hessian-based approaches, and the second builds on using a penalty function reformulation. The Jacobian/Hessian-based method achieves a sample complexity of $\mathcal{O}(d^3/\epsilon^2)$, which is optimal in terms of accuracy $\epsilon$, albeit with polynomial dependence on the problem dimension $d$. In contrast, the penalty-based method sharpens this guarantee to $\mathcal{O}(d/\epsilon^2)$, optimally reducing the dimension dependence to linear while preserving optimal accuracy scaling. Our analysis is built upon Gaussian smoothing techniques, and we rigorously establish their validity under the stochastic bilevel settings considered in the existing literature. To the best of our knowledge, this is the first work to provide provably optimal sample complexity guarantees for a zeroth-order stochastic approximation method in bilevel optimization.