Leveraging forgotten information to reduce requential code length

Determine whether and how to exploit information that is learned and subsequently forgotten during training to reduce the requential code length, potentially enabling the code length to decrease during training rather than only grow monotonically with the number of training steps.

Background

Requential coding compresses a model by coding its training process via samples selected relative to a teacher model. As currently formulated, the code length only accumulates over training because each step adds additional bits corresponding to teacher–student divergence.

The authors point out that some information learned early may be overwritten or forgotten later in training and thus need not be retained in a final code for the resulting model. Understanding whether and how to remove such forgotten information from the code would push model compression further and clarify generalization behavior.

References

Moreover, requential coding compresses a model's training process, but does not account for the fact that some of the information is not retained over the course of training. It is a particularly exciting open question to understand whether one could leverage forgotten information to reduce the code length, as the code length presently only grows with training steps and never decreases.

— Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data  (2607.11883 - Qiu et al., 13 Jul 2026) in Discussion