Validity of the replica-symmetric assumption for tanh at low sample ratios

Ascertain whether the replica-symmetric assumption holds for the tanh-activation single-neuron autoencoder in the spiked cumulant model at low sample ratios (approximately α ≲ 2), and, if it does not hold, characterize the nature of the symmetry breaking responsible for the observed discrepancies between theory and gradient-descent solutions.

Background

The authors observe a mismatch between replica-symmetric ERM predictions and gradient descent for tanh at small α, while other activations agree well with theory. This suggests a possible failure of the RS assumption in that regime.

Determining whether RS breaks and analyzing the corresponding structure (e.g., replica-symmetry breaking) would clarify the limits of the current theoretical prediction for tanh and guide refinements.

References

This could signal that the replica symmetric assumption is not satisfied in this region. Further investigation of that possibility is left for future work (see also \cref{app:tanh_simulations}).

— A solvable high-dimensional model where nonlinear autoencoders learn structure invisible to PCA while test loss misaligns with generalization  (2602.10680 - Mendes et al., 11 Feb 2026) in Section 5 (Autoencoder: Empirical risk minimization), paragraph discussing tanh activation