Hierarchical Predictive Coding Models in a Deep-Learning Framework

Abstract: Bayesian predictive coding is a putative neuromorphic method for acquiring higher-level neural representations to account for sensory input. Although originating in the neuroscience community, there are also efforts in the machine learning community to study these models. This paper reviews some of the more well known models. Our review analyzes module connectivity and patterns of information transfer, seeking to find general principles used across the models. We also survey some recent attempts to cast these models within a deep learning framework. A defining feature of Bayesian predictive coding is that it uses top-down, reconstructive mechanisms to predict incoming sensory inputs or their lower-level representations. Discrepancies between the predicted and the actual inputs, known as prediction errors, then give rise to future learning that refines and improves the predictive accuracy of learned higher-level representations. Predictive coding models intended to describe computations in the neocortex emerged prior to the development of deep learning and used a communication structure between modules that we name the Rao-Ballard protocol. This protocol was derived from a Bayesian generative model with some rather strong statistical assumptions. The RB protocol provides a rubric to assess the fidelity of deep learning models that claim to implement predictive coding.

• The paper introduces the integration of Bayesian predictive coding with deep learning frameworks, demonstrating improved hierarchical sensory predictions.
• The paper details the implementation of models like PredNet, which utilize cLSTM networks to forecast video frames and reduce prediction errors.
• The paper discusses potential extensions to other sensory modalities and future research directions to align model behavior with neural computation.

Hierarchical Predictive Coding Models in a Deep-Learning Framework

Introduction

The paper "Hierarchical Predictive Coding Models in a Deep-Learning Framework" (2005.03230) presents an analysis of Bayesian predictive coding, proposing it as a neuromorphic method for acquiring neural representations that account for sensory input. The authors explore different models of predictive coding, their connectivity patterns, and attempt to align these models within deep learning frameworks. The core concept revolves around using top-down mechanisms to predict incoming sensory inputs, with prediction errors informing the learning process that aims to refine higher-level representations. The paper seeks to blend classical predictive coding approaches with modern deep learning techniques to enhance the study and application of these models.

Rao-Ballard Model

One of the seminal models discussed is the Rao-Ballard model, which implements predictive coding using a hierarchical structure of predictive elements (PEs). Each PE, consisting of internal representation and prediction error neurons, forms a layer in a predictive coding hierarchy.

Figure 1: Rao/Ballard diagram of a predictive element. A predictive element enclosed in the dashed box is a building block for a predictive coding hierarchy.

The model demonstrates how higher layers in the hierarchy make predictions about the activity of lower layers, which in turn generate prediction errors used to update higher layers via backpropagation. This cycle reduces prediction errors over time and is hypothesized to reflect computational processes in the neocortex, capturing context effects like end-stopping in neural receptive fields. Each layer adjusts based on the derived cost function, aiming to minimize predictive errors within its structure.

Bayesian Predictive Coding

The Bayesian predictive coding approach introduces a generative model where higher layers attempt to reconstruct lower layer outputs. The Bayesian framework integrates prior expectations with sensory inputs to form a hierarchical representation that captures causal factors responsible for input generation. While earlier models presented complex, computationally intensive Bayesian calculations, the advent of deep learning provides a flexible platform to scale these concepts, albeit at the cost of exact Bayesian operations.

PredNet Model and Deep Learning Integration

PredNet, an instantiation of predictive coding within a deep learning framework, leverages convolutional Long Short-Term Memory (cLSTM) networks to predict video frames. The architecture aligns with predictive coding principles, where each layer forecasts the next frame based on current and historical input, with prediction errors guiding layer updates.

Figure 2: Information flow in the lowest PredNet layer (training mode) where the input is a ground truth video frame. The representation, $\mathsf{R}$, and prediction error components, $\mathsf{E}$, are recurrently connected.

By implementing a predictive error hierarchy with deep learning tools, PredNet models demonstrate potential scalability and versatility across complex tasks, deviating from the traditional Bayesian models. PredNet's distinct connectivity design—a departure from the Rao-Ballard protocol—suggests a novel way to compute video predictions, showing significant performance improvements in scenarios like next-frame video prediction.

Implications and Future Directions

The integration of predictive coding within deep learning frameworks offers a promising direction for building biologically plausible models capable of complex sensory predictions. Despite the departure from exact Bayesian principles, these models can scale and adapt, offering competitive performance across various tasks. Future work could explore refining these deep learning implementations to enhance their explanatory power and compatibility with established neural theories.

There's scope in extending such models to other sensory domains and leveraging their hierarchical nature for tasks beyond video prediction. The exploration of information transfer mechanisms, alongside biological realism, remains a critical area of research. Future studies should aim to quantify and operationalize the predictive power of these models in aligning more closely with human and animal neural computation principles.

Conclusion

The paper presents a detailed exploration of hierarchical predictive coding models within contemporary deep learning contexts, merging classical theoretical insights with modern computational capabilities. The analysis reveals potential pathways for both improving model performance and advancing our understanding of predictive processes in the brain, with implications for artificial intelligence and computational neuroscience.