Betty: An Automatic Differentiation Library for Multilevel Optimization

Abstract: Gradient-based multilevel optimization (MLO) has gained attention as a framework for studying numerous problems, ranging from hyperparameter optimization and meta-learning to neural architecture search and reinforcement learning. However, gradients in MLO, which are obtained by composing best-response Jacobians via the chain rule, are notoriously difficult to implement and memory/compute intensive. We take an initial step towards closing this gap by introducing Betty, a software library for large-scale MLO. At its core, we devise a novel dataflow graph for MLO, which allows us to (1) develop efficient automatic differentiation for MLO that reduces the computational complexity from O(d^3) to O(d^2), (2) incorporate systems support such as mixed-precision and data-parallel training for scalability, and (3) facilitate implementation of MLO programs of arbitrary complexity while allowing a modular interface for diverse algorithmic and systems design choices. We empirically demonstrate that Betty can be used to implement an array of MLO programs, while also observing up to 11% increase in test accuracy, 14% decrease in GPU memory usage, and 20% decrease in training wall time over existing implementations on multiple benchmarks. We also showcase that Betty enables scaling MLO to models with hundreds of millions of parameters. We open-source the code at https://github.com/leopard-ai/betty.

• The paper introduces Betty, a library that reduces gradient computation complexity from O(d^3) to O(d^2) for scalable multilevel optimization.
• It features a modular architecture with mixed-precision and data-parallel training to simplify implementation of complex MLO programs.
• Empirical results demonstrate up to 11% improvement in test accuracy, 14% reduced GPU memory usage, and 20% faster training times.

Essay: Betty: An Automatic Differentiation Library for Multilevel Optimization

The paper presents the development of "Betty," an automatic differentiation library designed specifically for multilevel optimization (MLO). This study is rooted in addressing the complexities involved in gradient-based MLO, an emerging framework tackling various optimization problems such as hyperparameter tuning, meta-learning, and neural architecture search. The primary challenges involve the computation and implementation intricacies of best-response Jacobians and substantial computational overhead.

Core Contributions

The authors introduce Betty to facilitate scalable MLO solutions with a focus on the following:

1. Efficient Automatic Differentiation: The paper details a novel dataflow graph for MLO that reduces computational complexity from $\mathcal{O}(d^3)$ to $\mathcal{O}(d^2)$, where $d$ represents dimensionality. This is achieved by interpreting MLO through specific graph paths, allowing for optimized differentiation.
2. Software Framework: Betty's design embodies a modular architecture supporting diverse algorithmic choices and system configurations. The modular design eases implementing MLO programs and incorporates efficiency-enhancing measures such as mixed-precision and data-parallel training.
3. Empirical Validation: The study demonstrates Betty's efficacy across various MLO programs, with significant improvements in test accuracy, GPU memory usage, and training wall time compared to existing solutions.

Numerical Results and Observations

The empirical findings underscore Betty's utility in improving performance metrics across different benchmarks:

• Test Accuracy: An increase of up to 11% was observed, affirming the computational and architectural enhancements provided by Betty.
• GPU Memory Efficiency: Achieved a reduction of 14% in memory usage through optimizations and system support.
• Training Wall Time: Demonstrated a 20% decrease, highlighting the enhanced computational efficiency.

These improvements are emphasized alongside Betty’s capability to handle models containing hundreds of millions of parameters, showcasing its scalability.

Theoretical and Practical Implications

Theoretically, Betty's dataflow graph interpretation advances the academic understanding of MLO by systematically addressing the bottlenecks in gradient calculation. Practically, it facilitates a streamlined integration of complex MLO solutions in machine learning pipelines, promising applications in domains like meta-learning and neural architecture design.

Speculations on Future Developments

Future work may likely explore expanding Betty's feature set to encompass model-parallel training and non-differentiable processes. Additionally, further exploration into memory optimization could enhance scalability further, addressing potential areas of bottleneck in increasingly complex MLO applications.

Conclusion

This paper significantly contributes to the field of multilevel optimization by developing a robust software framework, Betty, that integrates theoretical insights with practical efficiency. It bridges the gap in the current research landscape by providing a scalable, modular approach to tackling the inherent complexity of MLO problems. The study not only offers substantial computational improvements but also sets the stage for further exploration and advancement in automatic differentiation and optimization methodologies.