Finite-neuron analytical explanation of alignment and competition dynamics in modular addition networks

Establish a complete finite-neuron analytical characterization of alignment and competition dynamics—including layer-wise phase alignment and intra-neuron frequency competition—for two-layer fully connected neural networks trained on the modular addition task using standard gradient-based optimization with softmax cross-entropy loss, going beyond mean-field analyses and providing rigorous neuron-wise results.

Background

The paper surveys prior work on modular addition and grokking, noting that recent analyses often rely on mean-field approximations or modified training objectives. These approaches provide valuable insights but do not fully capture the finite, neuron-wise behavior under standard training conditions.

In particular, earlier studies characterize feature emergence and dynamics at a population level or under alternative loss formulations, leaving a gap in rigorous, neuron-wise analysis under commonly used settings such as gradient descent on cross-entropy loss with softmax. The authors explicitly identify the need for a full analytical result at the finite-neuron level that explains both the phase alignment between layers and the competitive dynamics among frequency components within individual neurons.