Learning Human-Object Interactions by Graph Parsing Neural Networks

Abstract: This paper addresses the task of detecting and recognizing human-object interactions (HOI) in images and videos. We introduce the Graph Parsing Neural Network (GPNN), a framework that incorporates structural knowledge while being differentiable end-to-end. For a given scene, GPNN infers a parse graph that includes i) the HOI graph structure represented by an adjacency matrix, and ii) the node labels. Within a message passing inference framework, GPNN iteratively computes the adjacency matrices and node labels. We extensively evaluate our model on three HOI detection benchmarks on images and videos: HICO-DET, V-COCO, and CAD-120 datasets. Our approach significantly outperforms state-of-art methods, verifying that GPNN is scalable to large datasets and applies to spatial-temporal settings. The code is available at https://github.com/SiyuanQi/gpnn.

• The paper introduces a novel Graph Parsing Neural Network that effectively models spatial and temporal dependencies for HOI detection.
• It utilizes an iterative message passing mechanism to refine graph structures, demonstrating a 13.11% mAP increase on the HICO-DET benchmark.
• The approach offers a scalable framework for advanced visual recognition tasks in robotics and automated surveillance.

Learning Human-Object Interactions by Graph Parsing Neural Networks

This paper introduces a novel approach to detect and recognize human-object interactions (HOI) in images and videos through the utilization of Graph Parsing Neural Networks (GPNN). The authors present a framework designed to parse graph structures integrating both spatial and temporal dependencies into a single, coherent model capable of being trained end-to-end.

Methodology

GPNNs extend the principles of Message Passing Neural Networks (MPNN) by incorporating graph structures focusing on HOI detection and recognition tasks. The framework's central component is the ability to infer a parse graph from a given scene, specifically addressing:

• Graph Structure Representation: This is achieved through an adjacency matrix inferred using link functions that evaluate node connectivities from edge features.
• Node Label Inference: The node labels describe the specific interactions, facilitated through iterative message passing, update functions, and readout functions.

The innovative aspect of GPNN is its iterative parsing mechanism, which refines graph structures and message propagation in tandem, allowing for dynamic adaptation to various relationships present across different scenes.

Performance Evaluation

The paper demonstrates the efficacy of GPNN through comprehensive testing on three HOI benchmarks: HICO-DET, V-COCO, and CAD-120 datasets. Experimental results highlight that GPNN significantly surpasses state-of-the-art methods in both image and video contexts. Specifically:

• HICO-DET Dataset: Achieved an increase in mean average precision (mAP) of 13.11% for full classes, with particularly strong performance in rare HOI categories.
• V-COCO Dataset: Yielded improvements reflected in mAP scores across single and dual-object action settings, showcasing the model’s robustness in complex interaction scenarios.
• CAD-120 Dataset: Noteworthy enhancements were seen in the F1-score for both detection and anticipation tasks, indicating the model’s capability to handle temporal data.

Implications and Future Directions

The results suggest that GPNN’s integration of graphical models within neural network architectures facilitates a deeper semantic understanding of HOIs. The model's ability to parse and represent interactions in both static and dynamic domains underscores its potential applicability across varied visual recognition tasks.

Theoretically, this approach paves the way for further exploration into graph-based neural architectures, while practically, it offers a foundational tool for advancements in cognitive robotics and automated video surveillance.

Future developments could focus on extending the GPNN framework to support more complex graphs and accommodate larger-scale datasets, further pushing the boundaries of adaptability and recognition accuracy in real-world applications. Moreover, exploring the introduction of hierarchical or multi-scale graph parsing could enhance the representation of multi-object interactions, supporting richer scene interpretation capabilities.

In conclusion, the GPNN presents a scalable, flexible, and efficient approach for advancing the understanding of human-object interactions, providing a valuable contribution to the field of computer vision.