Learning Algorithm Hyperparameters for Fast Parametric Convex Optimization

Abstract: We introduce a machine-learning framework to learn the hyperparameter sequence of first-order methods (e.g., the step sizes in gradient descent) to quickly solve parametric convex optimization problems. Our computational architecture amounts to running fixed-point iterations where the hyperparameters are the same across all parametric instances and consists of two phases. In the first step-varying phase the hyperparameters vary across iterations, while in the second steady-state phase the hyperparameters are constant across iterations. Our learned optimizer is flexible in that it can be evaluated on any number of iterations and is guaranteed to converge to an optimal solution. To train, we minimize the mean square error to a ground truth solution. In the case of gradient descent, the one-step optimal step size is the solution to a least squares problem, and in the case of unconstrained quadratic minimization, we can compute the two and three-step optimal solutions in closed-form. In other cases, we backpropagate through the algorithm steps to minimize the training objective after a given number of steps. We show how to learn hyperparameters for several popular algorithms: gradient descent, proximal gradient descent, and two ADMM-based solvers: OSQP and SCS. We use a sample convergence bound to obtain generalization guarantees for the performance of our learned algorithm for unseen data, providing both lower and upper bounds. We showcase the effectiveness of our method with many examples, including ones from control, signal processing, and machine learning. Remarkably, our approach is highly data-efficient in that we only use $10$ problem instances to train the hyperparameters in all of our examples.

• The paper presents a novel two-phase framework that dynamically learns hyperparameters to accelerate convergence in first-order methods.
• It details a progressive training strategy that minimizes mean square error to optimize hyperparameters for algorithms like gradient descent and ADMM solvers.
• Empirical results show significant computational gains and data efficiency, achieving near-optimal solutions with as few as ten training instances.

Learning Algorithm Hyperparameters for Fast Parametric Convex Optimization

The paper "Learning Algorithm Hyperparameters for Fast Parametric Convex Optimization" by Rajiv Sambharya and Bartolomeo Stellato introduces a methodological framework to enhance the efficiency of solving parametric convex optimization problems by learning algorithm hyperparameters. This work particularly focuses on the application of machine learning techniques in optimizing the hyperparameter sequence used in first-order methods (FOMs), thereby facilitating faster convergence in fixed-point iterations.

Summary of Contributions

1. Algorithmic Framework: The authors propose a two-phase computational architecture for solving parametric convex optimization problems: a step-varying phase and a steady-state phase. Initially, in the step-varying phase, hyperparameters adjust across iterations. Subsequently, in the steady-state phase, these parameters stabilize. This structure guarantees convergence to an optimal solution when run for a sufficient number of iterations.
2. Training Strategy: The paper introduces a progressive training strategy aimed at optimizing hyperparameters for several popular algorithms such as gradient descent, proximal gradient descent, and ADMM-based solvers (OSQP and SCS). The training minimizes the mean square error relative to a ground truth solution, either by solving closed-form solutions for some specific cases or by backpropagation in other scenarios.
3. Sample Convergence Bound: Leveraging a sample convergence bound, the authors provide generalization guarantees for the learned algorithms. These guarantees translate into practical error and performance bounds for unseen data, furnishing both lower and upper bounds.
4. Data Efficiency: A notable highlight of this research is its data efficiency. The framework was trained using only ten problem instances across different tests without compromising the performance metrics. Such an advancement in sparsity of training data potentializes its utility in real-world applications where vast datasets might be impractical.

Numerical Results

The efficacy of this method was thoroughly explored through a series of applications, including those pertinent to control, signal processing, and machine learning. The results demonstrate significant improvements in optimization performance, indicating reduced computational cost and time relative to traditional methods. Importantly, the results show consistency in attaining near-optimal solutions rapidly even with minimal training data.

Theoretical and Practical Implications

Theoretically, this work advances the understanding and application of machine learning in the optimization domain. The combination of machine learning with optimization algorithms opens the door to exploring hyperparameter adaptation across broader classes of optimization problems. The method bridges a crucial gap in ensuring convergence while enhancing computational efficiencies, a challenge that traditional FOMs often face due to static hyperparameter choices.

Practically, these advancements imply a considerable improvement in real-world applications where optimization needs to be accomplished swiftly under dynamic conditions, such as in real-time control systems and adaptive signal processing frameworks. The requirement for fewer training instances also makes this approach particularly attractive for scenarios where data availability is limited.

Future Directions: The exploration of this framework in diverse optimization contexts, including non-convex domains, can further shed light on its applicability and robustness across more complex landscapes. Additionally, extending the framework to incorporate model-based and residual-based learning components may yield further empirical improvements and theoretical insights.

In conclusion, this paper provides a robust framework for enhancing the efficiency of parametric convex optimization through learned hyperparameters, together with substantial empirical validation and theoretical guarantees, thereby making a notable contribution to the ongoing research in optimization and machine learning integration.