Scene Graph Reasoning with Prior Visual Relationship for Visual Question Answering

Abstract: One of the key issues of Visual Question Answering (VQA) is to reason with semantic clues in the visual content under the guidance of the question, how to model relational semantics still remains as a great challenge. To fully capture visual semantics, we propose to reason over a structured visual representation - scene graph, with embedded objects and inter-object relationships. This shows great benefit over vanilla vector representations and implicit visual relationship learning. Based on existing visual relationship models, we propose a visual relationship encoder that projects visual relationships into a learned deep semantic space constrained by visual context and language priors. Upon the constructed graph, we propose a Scene Graph Convolutional Network (SceneGCN) to jointly reason the object properties and relational semantics for the correct answer. We demonstrate the model's effectiveness and interpretability on the challenging GQA dataset and the classical VQA 2.0 dataset, remarkably achieving state-of-the-art 54.56% accuracy on GQA compared to the existing best model.