Training Deep Normalization-Free Spiking Neural Networks with Lateral Inhibition

Abstract: Spiking neural networks (SNNs) have garnered significant attention as a central paradigm in neuromorphic computing, owing to their energy efficiency and biological plausibility. However, training deep SNNs has critically depended on explicit normalization schemes, such as batch normalization, leading to a trade-off between performance and biological realism. To resolve this conflict, we propose a normalization-free learning framework that incorporates lateral inhibition inspired by cortical circuits. Our framework replaces the traditional feedforward SNN layer with a circuit of distinct excitatory (E) and inhibitory (I) neurons that complies with Dale's law. The circuit dynamically regulates neuronal activity through subtractive and divisive inhibition, which respectively control the activity and the gain of excitatory neurons. To enable and stabilize end-to-end training of the biologically constrained SNN, we propose two key techniques: E-I Init and E-I Prop. E-I Init is a dynamic parameter initialization scheme that balances excitatory and inhibitory inputs while performing gain control. E-I Prop decouples the backpropagation of the E-I circuits from the forward propagation and regulates gradient flow. Experiments across several datasets and network architectures demonstrate that our framework enables stable training of deep SNNs with biological realism and achieves competitive performance without resorting to explicit normalizations. Therefore, our work not only provides a solution to training deep SNNs but also serves a computational platform for further exploring the functions of lateral inhibition in large-scale cortical computation.