Universal Conditional Gradient Sliding for Convex Optimization

Abstract: In this paper, we present a first-order projection-free method, namely, the universal conditional gradient sliding (UCGS) method, for solving $\varepsilon$-approximate solutions to convex differentiable optimization problems. For objective functions with H\"older continuous gradients, we show that UCGS is able to terminate with $\varepsilon$-solutions with at most $O((M_\nu D_X^{1+\nu}/{\varepsilon})^{2/(1+3\nu)})$ gradient evaluations and $O((M_\nu D_X^{1+\nu}/{\varepsilon})^{4/(1+3\nu)})$ linear objective optimizations, where $\nu\in (0,1]$ and $M_\nu>0$ are the exponent and constant of the H\"older condition. Furthermore, UCGS is able to perform such computations without requiring any specific knowledge of the smoothness information $\nu$ and $M_\nu$. In the weakly smooth case when $\nu\in (0,1)$, both complexity results improve the current state-of-the-art $O((M_\nu D_X^{1+\nu}/{\varepsilon})^{1/\nu})$ results on first-order projection-free method achieved by the conditional gradient method. Within the class of sliding-type algorithms, to the best of our knowledge, this is the first time a sliding-type algorithm is able to improve not only the gradient complexity but also the overall complexity for computing an approximate solution. In the smooth case when $\nu=1$, UCGS matches the state-of-the-art complexity result but adds more features allowing for practical implementation.