Principled proxy model sizing for modern transformer architectures

Establish principled guidelines for selecting the architecture and parameter scale of proxy models relative to large target transformer models in language model pretraining, and extend theoretical analysis of proxy-to-target transferability from random feature models to modern transformer architectures.

Background

The paper studies how small proxy models can reliably guide data curation decisions for large-scale LLM pretraining and provides theoretical support using random feature models for training with tiny learning rates. While these results offer initial insights into width requirements, they do not address how to choose proxy architectures or sizes in modern transformer-based settings.

The authors explicitly note that extending theoretical understanding from random feature models to transformers and deriving principled sizing rules for proxy models remains unresolved. This is crucial for practitioners who rely on small-scale experiments to select data recipes that will generalize to large, tuned models.