Scaling Performance of Large Language Model Pretraining

Abstract: LLMs show best-in-class performance across a wide range of natural language processing applications. Training these models is an extremely computationally expensive task; frontier AI research companies are investing billions of dollars into supercomputing infrastructure to train progressively larger models on increasingly massive datasets. Unfortunately, information about the scaling performance and training considerations of these large training pipelines is scarce in public literature. Working with large-scale datasets and models can be complex and practical recommendations are scarce in the public literature for tuning training performance when scaling up LLMs. In this paper, we aim to demystify the LLM pretraining pipeline somewhat - in particular with respect to distributed training, managing large datasets across hundreds of nodes, and scaling up data parallelism with an emphasis on fully leveraging available GPU compute capacity.