Pedestrian Trajectory Prediction with Structured Memory Hierarchies

Abstract: This paper presents a novel framework for human trajectory prediction based on multimodal data (video and radar). Motivated by recent neuroscience discoveries, we propose incorporating a structured memory component in the human trajectory prediction pipeline to capture historical information to improve performance. We introduce structured LSTM cells for modelling the memory content hierarchically, preserving the spatiotemporal structure of the information and enabling us to capture both short-term and long-term context. We demonstrate how this architecture can be extended to integrate salient information from multiple modalities to automatically store and retrieve important information for decision making without any supervision. We evaluate the effectiveness of the proposed models on a novel multimodal dataset that we introduce, consisting of 40,000 pedestrian trajectories, acquired jointly from a radar system and a CCTV camera system installed in a public place. The performance is also evaluated on the publicly available New York Grand Central pedestrian database. In both settings, the proposed models demonstrate their capability to better anticipate future pedestrian motion compared to existing state of the art.

• The paper introduces structured LSTM cells that hierarchically model spatial and temporal dependencies for improved trajectory prediction.
• It integrates multimodal data from radar and video using gated fusion mechanisms to enhance prediction accuracy.
• Experimental results on challenging datasets show significant improvement in ADE, FDE, and non-linear ADE compared to baselines.

Pedestrian Trajectory Prediction with Structured Memory Hierarchies

Introduction and Motivation

The paper "Pedestrian Trajectory Prediction with Structured Memory Hierarchies" (1807.08381) addresses the complex task of predicting human trajectories using a novel framework that integrates structured memory components. Inspired by recent advances in neuroscience, this work leverages neural memory networks to enhance the predictive performance of human motion models, particularly in scenarios involving multimodal data from video and radar sources. By incorporating structured long short-term memory (LSTM) cells, the authors aim to capture both short-term and long-term spatiotemporal contexts to improve the accuracy of pedestrian trajectory forecasts.

Structured LSTM Cells

Central to the proposed framework is the development of structured LSTM (St-LSTM) cells. These cells are designed to preserve the hierarchical and structured nature of spatial memory, allowing the network to model complex spatial dependencies over time.

Figure 1: The operations of the proposed St-LSTM cell. It considers the current representation of the respective memory cell and the 3 adjacent neighbours as well as the previous time step outputs and utilises gated operations to render the output in the present time step.

The St-LSTM cells operate by hierarchically summarizing the spatial and temporal memory content. This hierarchical processing ensures that the network effectively captures salient information from past sequences, enabling it to anticipate future movements with greater fidelity.

Multimodal Information Fusion

Another critical aspect of the paper is the integration of multimodal data through separate memory modules for each modality. The authors propose a method for coupling data from radar and video streams, allowing for complementary information from each to enhance prediction accuracy.

Figure 2: Coupling multimodal information through multiple memory modules. The information from each modality is stored separately. Note that the figure shows only the top most layer in each memory.

This multimodal approach, which involves gated fusion mechanisms, allows the system to prioritize and integrate salient features from both input streams, thus aiding in more robust trajectory predictions in various environmental conditions.

Evaluation and Results

The proposed Structured Memory Network (SMN) is evaluated on a new multimodal dataset comprising radar and video data, as well as the New York Grand Central pedestrian database. The experimental results indicate that the SMN outperforms several state-of-the-art models across different metrics, demonstrating its capability to model human navigational behavior with improved accuracy.

Quantitatively, the SMN shows substantial improvements in average displacement error (ADE), final displacement error (FDE), and non-linear average displacement error (n-ADE) compared to baseline models. The hierarchical structuring of memory and multimodal integration in the SMN architecture significantly contribute to these performance gains.

Future Directions and Implications

The research presents significant implications for the development of intelligent systems capable of human behavior prediction in dynamic environments. By effectively capturing structured memory and leveraging multimodal data, this trajectory prediction framework can be extended to various applications in surveillance, robotics, and autonomous navigation.

Future work could explore the scalability of the structured memory framework to even more complex multimodal environments, as well as the adaptation of such models for real-time predictions in highly dynamic scenarios. Furthermore, the approach may be enhanced by investigating additional modalities and advanced memory architectures.

Conclusion

The paper introduces a cutting-edge approach to pedestrian trajectory prediction by structuring memory hierarchies and integrating multimodal inputs. The results demonstrate that this model effectively captures the nuanced, hierarchical nature of human navigation, pointing towards promising advancements in trajectory prediction methodologies. This work sets the stage for future exploration into more comprehensive and context-aware predictive models in the domain of human behavior analysis.