Spontaneous symmetry breaking and Goldstone modes for deep information propagation

Abstract: In physical systems, whenever a continuous symmetry is spontaneously broken, the system possesses excitations called Goldstone modes, which allow coherent information propagation over long distances and times. In this work, we study deep neural networks whose internal layers are equivariant under a continuous symmetry and may therefore support analogous Goldstone-like degrees of freedom. We demonstrate, both analytically and empirically, that these degrees of freedom enable coherent signal propagation across depth and recurrent iterations, providing a mechanism for stable information flow without relying on architectural stabilizers such as residual connections or normalization. In feedforward networks, this results in improved trainability and representational diversity across layers. In recurrent settings, we demonstrate the same mechanism is valuable for long-term memory by propagating information over recurrent iterations, thereby improving performance of RNNs and GRUs on long-sequence modeling tasks.

• The paper demonstrates that continuous symmetry breaking creates Goldstone modes, preserving robust information flow across deep layers.
• It employs analytical mean-field theory and stochastic path integrals to rigorously link symmetry breaking with improved trainability and gradient stability.
• Empirical results confirm that symmetry-protected channels enhance long-sequence memory and representation stability in various network architectures.

Spontaneous Symmetry Breaking and Goldstone Modes in Deep Information Propagation

Introduction

The paper "Spontaneous symmetry breaking and Goldstone modes for deep information propagation" (2605.14685) investigates how continuous symmetry breaking in neural network architectures induces Goldstone-like excitations, resulting in robust information transfer and improved trainability. Drawing analogies from equilibrium statistical mechanics, the authors demonstrate both analytically and empirically that internal layer equivariance can be leveraged independently of task symmetries, providing a mechanism for stable signal propagation across depth and time. The work explores feedforward, recurrent, and convolutional network settings, quantifies associated phase transitions, and establishes rigorous connections between symmetry, trainability, and information retention in very deep and long-sequence models.

Theoretical Framework: Symmetry, Equivariance, and Goldstone Modes

The study originates in the observation that the layer direction of a neural network can be regarded as analogous to "time" in physical systems. In physics, spontaneous breaking of a continuous symmetry results in Goldstone modes—long-lived excitations that propagate information coherently (see Goldstone's theorem in [Goldstone:1962es]). The authors import this principle into deep learning networks whose internal layers are equivariant under continuous symmetry groups such as $U(1)$ and $O(k)$.

The central construction involves equivariant internal transformations:

• For $U(1)$ symmetry, layers operate on $\mathbb{C}^N$ with complex matrices and nonlinearities that preserve phase, e.g., $\phi(z) = \frac{\tanh(|z|)}{|z|}z$.
• For $O(k)$ symmetry, blocks of $k$ coordinates per neuron are rotated, with block-diagonal weight matrices and nonlinearities acting on the norm.

Layerwise equivariance is formally encoded as $$\rho_g f^l(\mathbf{x}^l) = f^l(\rho_g \mathbf{x}^l)$$ for each $g \in G$ and internal layer $l$.

Phase Transitions and Information Propagation

The analytical results demonstrate that networks exhibit two distinct phases:

• Symmetry unbroken phase: All activations decay to zero, erasing input information.
• Symmetry spontaneously broken (SSB) phase: Activations retain nonzero magnitude, enabling symmetry operations to persist across layers.

A critical order parameter $O(k)$0 is defined as the average $O(k)$1-invariant activation norm. The depth transition, precisely at a critical weight variance $O(k)$2, is characterized through recursive mean-field equations:

Figure 1: Equilibrium configurations—minima of potential energy $O(k)$3—distinguish the symmetry-unbroken and spontaneously broken phases.

Empirically, when models operate in the SSB regime, the network retains a channel for symmetry direction information—the neural equivalent of a Goldstone mode—across arbitrarily deep architectures.

Large-$O(k)$4 Mean-Field Analysis

Mean-field theory and stochastic path integrals yield self-consistent equations for order parameters and input-output covariance evolution. Goldstone mode conservation appears in the invariant phase $O(k)$5 of multi-input covariance matrices: $O(k)$6 for all depths, establishing a protected, non-dissipative channel for mutual input correlations.

Additionally, a protected component of the input/output Jacobian is shown to remain $O(k)$7 in the SSB phase, substantiating the claim of robust gradient flow independent of depth:

Figure 2: Norm of the input/output Jacobian and its protected component, demonstrating stability in the SSB phase.

Empirical Results: Feedforward and Recurrent Models

Empirical studies reveal:

• Feedforward Networks: $O(k)$8 and $O(k)$9 equivariant models with 100 layers train successfully in the SSB phase with no normalization or skip connections, outperforming generic linear architectures.

Figure 3: Test accuracy after 5 epochs on Fashion-MNIST for $U(1)$0 (left) and $U(1)$1 (right) equivariant vs generic architectures, showing performance activation at symmetry breaking.

• RNNs and GRUs: $U(1)$2-equivariant recurrent architectures consistently outperform vanilla RNNs and GRUs on long-sequence copy tasks and permuted sequential MNIST, even with fewer real parameters.

  Figure 4: Comparison of RNNs and GRUs with $U(1)$3 equivariant counterparts on copy tasks of varying delay, revealing significant advantage for symmetry-protected models.

• Jacobian and Representation Rank: Equivariant architectures maintain effective representation rank and healthy gradients across depth, while generic models collapse.

  Figure 5: Protected component of Jacobian at initialization; stability in the symmetry broken phase versus exponential decay without symmetry.

Topological Defects and Convolutional RNNs

The study further explores the emergence of topological defects—e.g., vortices—in 2D convolutional $U(1)$4-equivariant RNNs trained on cognitive tasks. These defects persist over sequence iterations and suggest potential mechanisms for stable memory storage and spatial structuring of information in neural models.

Figure 6: Hidden state magnitude, phase, and vorticity for a single channel in a 2D convolutional $U(1)$5 equivariant RNN; long-lived vortices persist.

Figure 7: Visualization of another channel, with high persistent vorticity throughout the sequence.

Implications and Future Directions

The findings have practical and theoretical implications:

• Architecture Design: Equivariance can be used as an architectural tool independent of task symmetries, enabling efficient training in regimes where classical edge-of-chaos paradigms fail.
• Long-term Memory: Goldstone channels afford robust memory retention in RNNs and GRUs for sequence lengths exceeding 100-200 steps, suggesting compatible integration with state-of-the-art architectures.
• Representation Diversity: Equivariant models guarantee minimum effective rank across depth, preventing representation collapse.
• Physics-Inspired Modeling: The analogy to Goldstone modes and topological defects opens new perspectives on dynamical stability, information flow, and the origins of generalization in neural systems.

The study suggests further exploration of joint models combining symmetry-breaking mechanisms with advanced gating or normalization, analysis of higher-form symmetries, and investigation into the computational role of topological defects and vortices in both artificial and biological neural computation.

Conclusion

This paper introduces a formal, physics-motivated mechanism for deep information propagation by exploiting spontaneous symmetry breaking and Goldstone modes within neural network internal layers. Both theoretical derivations and rigorous empirical benchmarks confirm that these symmetry-protected channels yield substantial improvements in trainability, gradient stability, and long-sequence memory—without the necessity for architectural mitigations. The results substantiate the utility of continuous symmetry equivariance as an intrinsic tool for robust, scalable deep learning and chart new directions in the physics-inspired design of neural architectures.