Minimal mathematical model capturing both noise-dominated and signal-dominated regimes

Construct a minimal mathematical model that simultaneously exhibits the noise-dominated regime for matrix parameters and the signal-dominated regime for scalar/vector parameters, to explain the observed differences in scale adaptation and equilibrium behavior during language-model training.

Background

Throughout the paper, matrix weights are shown to follow a noise–WD equilibrium scaling that constrains their norms, while learnable multipliers do not. A concise mathematical model capturing both behaviors would clarify underlying mechanisms and guide when and how to deploy multipliers.

References

Yet, many questions are left open. Hence, an interesting direction for future work is to mechanistically understand the difference between matrix and scalar/vector dynamics, find an empirically measurable indicator of the noise level, or build a minimal mathematical model exhibiting both training regimes.

— Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers  (2601.04890 - Velikanov et al., 8 Jan 2026) in Section 6: Conclusion and discussion