Explosive neural networks via higher-order interactions in curved statistical manifolds

Abstract: Higher-order interactions underlie complex phenomena in systems such as biological and artificial neural networks, but their study is challenging due to the scarcity of tractable models. By leveraging a generalisation of the maximum entropy principle, here we introduce curved neural networks as a class of prototypical models with a limited number of parameters that are particularly well-suited for studying higher-order phenomena. Through exact mean-field descriptions, we show that these curved neural networks implement a self-regulating annealing process that can accelerate memory retrieval, leading to explosive order-disorder phase transitions with multi-stability and hysteresis effects. Moreover, by analytically exploring their memory-retrieval capacity using the replica trick near ferromagnetic and spin-glass phase boundaries, we demonstrate that these networks can enhance memory capacity and robustness of retrieval over classical associative-memory networks. Overall, the proposed framework provides parsimonious models amenable to analytical study, revealing novel higher-order phenomena in complex networks.

• The paper introduces a novel framework using maximum Rényi entropy to efficiently capture higher-order interactions in neural networks.
• The paper employs mean-field analysis to reveal explosive phase transitions, multi-stability, and hysteresis effects via self-regulated annealing.
• The paper demonstrates that curved networks achieve superior memory capacity by surpassing classical associative-memory limits near critical phase boundaries.

Explosive Neural Networks via Higher-Order Interactions in Curved Statistical Manifolds

The paper "Explosive neural networks via higher-order interactions in curved statistical manifolds" introduces a novel modeling framework for understanding higher-order interactions (HOIs) in neural networks. These models leverage the maximum entropy principle extended to curved statistical manifolds, effectively capturing the complexity of HOIs without succumbing to the combinatorial explosion typically associated with such detailed modeling.

Key Contributions

The study addresses the inherent difficulties in modeling systems with HOIs by employing a framework based on the maximum Rényi entropy. This approach curates a family of so-called curved neural networks, making them analytically tractable while still encapsulating higher-order phenomena. The primary contributions of the paper are:

1. Curved Neural Networks: Using the Rényi entropy, the authors introduce an additional parameter, $\gamma$, which allows these neural networks to account for higher-order dependencies efficiently.
2. Mean-Field Analysis: The paper provides mean-field descriptions of these curved neural networks, showing the emergence of explosive phase transitions, multi-stability, and hysteresis effects driven by a self-regulating annealing process.
3. Memory Capacity Analysis: Through analytical exploration near the ferromagnetic and spin-glass phase boundaries, the study demonstrates that these networks exceed the memory capacity of classical associative-memory networks.

Detailed Overview

Higher-Order Interactions in Curved Manifolds

The maximum entropy principle (MEP) has been extended to include not just low-order but high-order statistics:

• Maximum Entropy Principle (MEP): Traditionally, the MEP has been used to model systems by maximizing the Shannon entropy subject to constraints on observed statistics.
• Rényi Entropy: By expanding the MEP framework to include Rényi entropy, the authors capture higher-order interdependencies without an extensive number of parameters, avoiding the combinatorial explosion.
• Curved Statistical Manifolds: A deformation parameter $\gamma$ effectively curves the statistical manifold, inherently accounting for higher-order interactions even when the observable set is restricted to low-order statistics.

Curved Neural Networks

The generalization of neural network models includes an energy function: $p_{\gamma}(\bm{x}) = \exp(-\varphi_{\gamma}) \brpos{1 + \gamma \beta E(\bm{x})}^{1/\gamma},$ where $E(\bm{x})$ encapsulates the usual energy formulation, but the effective temperature $\beta'$ is state-dependent and dynamically regulated: $\beta' = \frac{\beta}{1 - \gamma \beta E(\bm{x})}.$

This adaptation leads to:

• Self-Regulated Annealing: Positive feedback between energy and temperature accelerates memory retrieval.
• Explosive Phase Transitions: Sudden, dramatic changes in activation rates, illustrated by extending classic mean-field solutions.

Memory Capacity Analysis and Explosive Phenomena

The inclusion of HOIs in these networks leads to significant findings:

• Enhanced Memory Capacity: By analyzing the system's behavior near saturation using the replica trick, it's demonstrated that curved neural networks hold more patterns (memories) compared to classical networks.
• Explosive Spin-Glass Transitions: The paper reveals that higher-order effects can lead to novel transitions within disordered systems, characterized by both abrupt and critical behaviors, separated by the extent of the deformation parameter $\gamma$.

Implications and Future Directions

The implications of this research are broad:

• Artificial Neural Networks: The insights offer pathways to improve current deep learning architectures by embedding higher-order statistics efficiently.
• Mathematical Framework: The study consolidates concepts from information geometry, entropy measures, and neural network dynamics, inspiring further theoretical work.
• Biological Neurons: Understanding sparsity and critical dynamics in biological neurons could benefit from applying such high-order statistical models, possibly explaining long periods of silence observed in neural activity.

Conclusion

In summary, this paper presents a rigorous generalization of neural networks that incorporates higher-order interactions through information geometric methods. The thorough analytical treatment and implications for both artificial and biological neural systems provide a robust platform for future exploration into complex network phenomena.

Published Researcher - [Repository / Journal]

This overview not only captures the breadth of the study but also underscores the vital numerical and theoretical contributions, laying a foundation for further research and practical application in AI and beyond.