Re-parameterizing Your Optimizers rather than Architectures

Abstract: The well-designed structures in neural networks reflect the prior knowledge incorporated into the models. However, though different models have various priors, we are used to training them with model-agnostic optimizers such as SGD. In this paper, we propose to incorporate model-specific prior knowledge into optimizers by modifying the gradients according to a set of model-specific hyper-parameters. Such a methodology is referred to as Gradient Re-parameterization, and the optimizers are named RepOptimizers. For the extreme simplicity of model structure, we focus on a VGG-style plain model and showcase that such a simple model trained with a RepOptimizer, which is referred to as RepOpt-VGG, performs on par with or better than the recent well-designed models. From a practical perspective, RepOpt-VGG is a favorable base model because of its simple structure, high inference speed and training efficiency. Compared to Structural Re-parameterization, which adds priors into models via constructing extra training-time structures, RepOptimizers require no extra forward/backward computations and solve the problem of quantization. We hope to spark further research beyond the realms of model structure design. Code and models \url{https://github.com/DingXiaoH/RepOptimizers}.

• The paper introduces Gradient Re-parameterization to reframe optimizers with model-specific hyper-parameters, effectively embedding structural priors.
• The paper demonstrates that using RepOptimizers with simple models like RepOpt-VGG can match or exceed the performance of complex architectures like EfficientNet.
• The paper incorporates a Hyper-Search method to efficiently determine hyper-parameters, thereby reducing training time and memory usage.

Overview of "Re-parameterizing Your Optimizers Rather than Architectures"

The paper presents a novel approach to improving neural network performance by re-parameterizing optimizers instead of architectures. The authors propose a methodology known as Gradient Re-parameterization (GR), leading to the development of RepOptimizers. This approach integrates model-specific prior knowledge into optimizers, which conventional model-agnostic optimizers like SGD do not leverage. The core idea is to modify gradients using a set of model-specific hyper-parameters, permitting even simple models, such as the VGG-style architecture used in this study, to achieve performance levels on par or better than more complex structures.

Key Contributions

1. Gradient Re-parameterization (GR):
   • The authors demonstrate a method for incorporating structural priors into model-specific optimizers by altering gradients according to pre-defined model-specific hyper-parameters. This innovative approach is termed Gradient Re-parameterization, and the resulting optimizers are referred to as RepOptimizers.
2. Empirical Performance:
   • The paper showcases the use of RepOptimizers with a simplistic architecture, namely RepOpt-VGG, which performs comparably or even superior to highly evolved models like EfficientNet. The VGG-style model, characterized by its basic structure of stacked 3$\times$3 convolution layers, benefits significantly from being paired with RepOptimizers, resulting in high inference speed and training efficiency without the computational overhead associated with structural changes.
3. Training Efficiency:
   • RepOptimizers offer significant practical benefits, such as reduced training time and memory usage. For instance, the RepOpt-VGG-B1 model achieves 1.8$\times$ the training speed of its baseline, RepVGG-B1, while maintaining comparable accuracy levels.
4. Quantization-Friendly Models:
   • The framework mitigates the challenges of quantization typically associated with structurally re-parameterized models, facilitating easier deployment across diverse hardware.
5. Hyper-Search Method for Hyper-Parameters:
   • The paper introduces a Hyper-Search (HS) process to obtain the hyper-parameters necessary for constructing RepOptimizers. These hyper-parameters affect training dynamics and are derived from an auxiliary model trained on a small dataset, reducing the computational demands of this step significantly.

Implications and Future Directions

This research pushes the boundaries of how we perceive model optimization, suggesting that altering optimizers can be as effective, if not more efficient, than redesigning architectures for specific tasks. The methodology extends beyond the VGG architecture, with the authors providing examples like RepGhostNet to highlight its flexibility and applicability to other models.

The paper invites exploration into additional models and optimization strategies, such as derivative-free methods, and further refinement of the HS process. Additionally, it speculates on the potential convergence of optimization methods with meta-learning to create more robust and efficient optimization systems. While RepOptimizers were demonstrated using first-order gradient-based optimizers, exploring their application and efficacy across high-order or combined optimization frameworks would be valuable.

Overall, this work proposes a paradigm shift in neural network optimization, emphasizing the transformation of optimizer dynamics via gradient re-parameterization as a pathway to achieving high model efficiency with simpler architectural designs. This insight opens new avenues for innovation in deploying neural networks across progressively resource-constrained environments.