History-Aware Adaptive High-Order Tensor Regularization

Abstract: In this paper, we develop a new adaptive regularization method for minimizing a composite function, which is the sum of a $p$th-order ($p \ge 1$) Lipschitz continuous function and a simple, convex, and possibly nonsmooth function. We use a history of local Lipschitz estimates to adaptively select the current regularization parameter, an approach we %have termed shall term the {\it history-aware adaptive regularization method}. Our method matches the complexity guarantees of the standard $p$th-order tensor method that require a known Lipschitz constant, for both convex and nonconvex objectives. In the nonconvex case, the number of iterations required to find an $(\epsilon_g,\epsilon_H)$-approximate second-order stationary point is bounded by $\mathcal{O}(\max{\epsilon_g^{-(p+1)/p}, \epsilon_H^{-(p+1)/(p-1)}})$. For convex functions, the iteration complexity improves to $\mathcal{O}(\epsilon^{-1/p})$ when finding an $\epsilon$-approximate optimal point. Furthermore, we propose several variants of this method. For practical consideration, we introduce cyclic and sliding-window strategies for choosing proper historical Lipschitz estimates, which mitigate the limitation of overly conservative updates. Theoretically, we introduce Nesterov's acceleration to develop an accelerated version for convex objectives, which attains an iteration complexity of $\mathcal{O}(\epsilon^{-1/(p+1)})$. Finally, extensive numerical experiments are conducted to demonstrate the effectiveness of our adaptive approach.