A Theoretical Framework for Inference Learning

Abstract: Backpropagation (BP) is the most successful and widely used algorithm in deep learning. However, the computations required by BP are challenging to reconcile with known neurobiology. This difficulty has stimulated interest in more biologically plausible alternatives to BP. One such algorithm is the inference learning algorithm (IL). IL has close connections to neurobiological models of cortical function and has achieved equal performance to BP on supervised learning and auto-associative tasks. In contrast to BP, however, the mathematical foundations of IL are not well-understood. Here, we develop a novel theoretical framework for IL. Our main result is that IL closely approximates an optimization method known as implicit stochastic gradient descent (implicit SGD), which is distinct from the explicit SGD implemented by BP. Our results further show how the standard implementation of IL can be altered to better approximate implicit SGD. Our novel implementation considerably improves the stability of IL across learning rates, which is consistent with our theory, as a key property of implicit SGD is its stability. We provide extensive simulation results that further support our theoretical interpretations and also demonstrate IL achieves quicker convergence when trained with small mini-batches while matching the performance of BP for large mini-batches.

• The paper establishes a theoretical framework that links inference learning with implicit SGD, offering a biologically plausible and stable alternative to backpropagation.
• It employs local Hebbian-like rules and proximal updates, ensuring stability even at high learning rates through careful control of target output activity.
• Extensive simulations show that inference learning converges faster with small mini-batches while matching backpropagation performance on supervised and associative tasks.

Theoretical Examination of Inference Learning

Introduction and Context

The paper "A Theoretical Framework for Inference Learning" (2206.00164) investigates Inference Learning (IL) as an alternative to the Backpropagation (BP) algorithm commonly employed in deep learning. Although BP is highly effective, it is often critiqued for its lack of biological plausibility. This study seeks to bridge that gap by providing a theoretical framework for IL, demonstrating its close approximation to implicit stochastic gradient descent (implicit SGD), and identifying conditions under which IL can be more stable than BP.

Inference Learning and Implicit SGD

IL is proposed as a biologically consistent algorithm that minimizes an energy function termed free energy, which relies on local learning rules akin to Hebbian synaptic plasticity. A significant contribution of the paper is establishing the mathematical linkage between IL and implicit SGD, an optimization technique recognized for its stability. Unlike explicit SGD used by BP, implicit SGD ensures stable parameter updates by solving an optimization problem that implicitly restricts updates to remain proximal to current parameter values. The equivalence to implicit SGD provides IL with theoretical grounding, underpinning its potential stability advantages.

Theoretical Framework and Stability

Central to the paper is the development of Generalized Inference Learning (G-IL), which is underpinned by a detailed theoretical analysis. The authors derive that IL approximates implicit SGD, particularly when the network is trained with mini-batch size 1, a setting that is deemed biologically plausible.

For G-IL, the core finding is that it naturally resolves into proximal updates that maintain stability across learning rates, a property that is not inherently present in BP. This stability arises because the learning rate in IL is utilized to dictate the target output activity during the inference phase rather than directly scaling weight updates, thereby avoiding the instabilities associated with BP at higher learning rates.

Empirical Validation

The paper supports its theoretical claims with extensive simulations, comparing IL to BP across various data and tasks. Notably, IL matched BP's performance on supervised and associative tasks while showing faster convergence with smaller mini-batches. This efficiency is attributed not to larger update magnitudes but rather a more direct optimization path, aligning with the minimum norm trajectory to convergence.

Moreover, the analysis of weight updates provides empirical evidence that IL updates, unlike those of BP, do not interfere with the collective optimization path, enhancing its compatibility with pre-synaptic activities. This results in outputs that change in a manner more aligned with minimizing the global loss.

Implications and Future Directions

The theoretical grounding of IL as a practical alternative to BP could have far-reaching implications both in understanding synaptic plasticity within neural circuits and in developing AI systems grounded in biological principles. By formalizing the linkage between IL and implicit SGD, the research not only bolsters IL's biological plausibility but positions it as a stable, high-performing alternative learning algorithm.

The paper prompts new inquiries into the role of proximal updates in neural function and offers a promising avenue for more biologically realistic models of learning. Future research is invited to explore enhancements of IL's optimization properties, especially under conditions frequently occurring in machine learning practices, such as larger batch sizes.

Conclusion

The authors successfully establish a rigorous theoretical framework for IL, positing it as a biologically plausible, mathematically sound, and potentially more stable alternative to BP. The work paves the way for further exploration into biologically inspired learning algorithms that may enrich both neuroscience and machine learning applications.