Scalable strong compression for large neural networks

Develop a scalable compression scheme for large neural networks that achieves sufficiently strong compression at scale to faithfully reflect the information learned by the models, overcoming the limitations of parameter-based compression and prequential coding that inflate code length irrespective of actual learned information.

Background

The paper argues that measuring compression is key to understanding generalization, but existing approaches fall short at scale. Parameter-based methods such as post-training quantization produce codes that scale with parameter count regardless of the information actually stored in the parameters. Prequential coding instead compresses the training trajectory but pays the full entropy of the data sequence, leading to codes that grow linearly with dataset size even after the model stops learning from unpredictable data.

Because modern models are trained far short of the information capacity of their parameters, and because prequential coding pays heavily for data entropy, the authors note that finding a truly strong compression method that works at modern scales remains unresolved. Their proposed requential coding is presented as a step toward addressing this challenge, but the general problem of discovering sufficiently good compression at scale is explicitly identified as open.