Per-module hyperparameter gains and transfer

Determine whether tuning hyperparameters on a per-module basis yields significant performance gains and whether those per-module hyperparameters empirically transfer across model scaling when using a parameterisation that enables hyperparameter transfer across width and depth (for example, μP, Depth-μP, or CompleteP) in transformer architectures.

Background

The paper discusses parameterisations such as μP, Depth-μP, and CompleteP that enable transferring optimal global hyperparameters across model sizes. Building on these, the authors pose whether more granular, per-module hyperparameter tuning can provide benefits and whether such hyperparameters also transfer under the right parameterisation when scaling model width and depth.

This question motivates their empirical investigation into per-module hyperparameter optimisation and transfer, aiming to handle the increased dimensionality and complexity of the hyperparameter landscape while leveraging principled scaling rules.