Zero-Shot Human-Object Interaction Recognition via Affordance Graphs

Abstract: We propose a new approach for Zero-Shot Human-Object Interaction Recognition in the challenging setting that involves interactions with unseen actions (as opposed to just unseen combinations of seen actions and objects). Our approach makes use of knowledge external to the image content in the form of a graph that models affordance relations between actions and objects, i.e., whether an action can be performed on the given object or not. We propose a loss function with the aim of distilling the knowledge contained in the graph into the model, while also using the graph to regularise learnt representations by imposing a local structure on the latent space. We evaluate our approach on several datasets (including the popular HICO and HICO-DET) and show that it outperforms the current state of the art.

• The paper introduces an affordance graph integrated with GCNs to address zero-shot HOI learning using a dual-representation strategy.
• The paper demonstrates superior performance with a mAP of 16.02 on the HICO dataset, outperforming previous state-of-the-art methods.
• The paper leverages a unique loss function and weakly-supervised label estimation to effectively structure semantic relationships in unseen interactions.

Zero-Shot Human-Object Interaction Recognition via Affordance Graphs

Introduction

The paper addresses the challenge of Zero-Shot Learning (ZSL) for Human-Object Interaction (HOI) recognition, focusing on interactions involving unseen actions. This is achieved by leveraging an affordance graph that models the relationships between actions and objects. The paper proposes a novel loss function to distill knowledge from this graph into the model, facilitating structured learning even in the absence of explicit interaction data. This approach is demonstrated to outperform existing state-of-the-art methods on popular datasets, including HICO and HICO-DET.

Problem Statement

Traditional object-action interaction datasets are heavily annotated with specific $\langle human, action, object \rangle$ triplets, which presents scalability issues due to the combinatorial increase in possible interactions. The problem is exacerbated in scenarios where datasets are non-exhaustive, creating a need for models capable of inference beyond seen data, i.e., zero-shot scenarios. This research positions actions as pivotal in defining interactions, supported by psychological and neurobiological evidence that objects’ semantics are affordance-driven.

Methodology

The core component of the proposed method is the affordance graph, constructed from available datasets and external sources, which maps feasible action-object pairs. The model employs a Graph Convolutional Network (GCN) to contextualize and propagate semantic relationships. It is trained using a dual-representation approach: internal representations for seen interactions leveraging complete supervision, and external representations for unseen interactions facilitated by semi-supervised learning over the affordance graph. The proposed training strategy includes a unique loss function that reinforces graph-induced affordances and local semantic structuring in the latent space.

Figure 1: Overview of the proposed model for HOI Recognition during training.

Results and Evaluation

The method is benchmarked against contemporaneous approaches on multiple datasets, showing significant improvements in mean Average Precision (mAP) for both seen and unseen classes. For instance, on the HICO dataset, the proposed approach achieves a mAP of 16.02, substantially outperforming previous best results. A noteworthy component contributing to these results is the novel weakly-supervised label estimation approach and affordance regularizer.

Figure 2: Visualisation of action class representations highlighting the effectiveness of the proposed approach in clustering functionally similar actions.

Related Work and Theoretical Implications

HOI detection benefits from external knowledge utilization, akin to LLMs that augment data with functionally similar interactions. Zero-shot recognition is an evolving field tackling the lack of exhaustive labels in datasets by transferring learned representations to novel classes. The research provides an insightful mapping of existing Zero-Shot methodologies and highlights the unique affordance-driven graph utilization, differentiating it from earlier word embedding-based approaches.

Conclusion and Future Directions

The research effectively integrates affordance knowledge into deep learning frameworks using GCNs, offering robust ZSL capabilities for HOI recognition tasks. By transcending limitations of dataset completeness and scaling challenges, it sets the stage for future explorations into composite HOI understanding and extending affordance graphs with richer semantics. Further advancements could explore integrating dynamic spatiotemporal affordances and applying this framework to real-world robotics for enhanced interaction comprehension.

Overall, this paper represents a critical step towards enhancing zero-shot learning in human-object interactions by marrying deep learning architectures with structured knowledge representation — a promising frontier in cognitive modeling and artificial intelligence.