Distinctiveness of NSP supervision beyond NTP

Establish that the next-sequence prediction objective provides sequence-level learning signals not captured by the next-token prediction objective when training fast weight language models, thereby explaining observed improvements in next-token prediction accuracy under mid-training.

Background

The paper compares supervised fine-tuning under next-token prediction with reinforcement learning under the proposed next-sequence prediction objective in fast weight models. Although next-token prediction directly optimizes token-level likelihood, the experiments report higher next-token prediction accuracy gains from next-sequence prediction training.

Motivated by these results, the authors explicitly conjecture that sequence-level supervision in next-sequence prediction provides learning signals beyond those available to next-token prediction, suggesting a need to formally validate this claim.