Visual Parser: Representing Part-whole Hierarchies with Transformers

Abstract: Human vision is able to capture the part-whole hierarchical information from the entire scene. This paper presents the Visual Parser (ViP) that explicitly constructs such a hierarchy with transformers. ViP divides visual representations into two levels, the part level and the whole level. Information of each part represents a combination of several independent vectors within the whole. To model the representations of the two levels, we first encode the information from the whole into part vectors through an attention mechanism, then decode the global information within the part vectors back into the whole representation. By iteratively parsing the two levels with the proposed encoder-decoder interaction, the model can gradually refine the features on both levels. Experimental results demonstrate that ViP can achieve very competitive performance on three major tasks e.g. classification, detection and instance segmentation. In particular, it can surpass the previous state-of-the-art CNN backbones by a large margin on object detection. The tiny model of the ViP family with $7.2\times$ fewer parameters and $10.9\times$ fewer FLOPS can perform comparably with the largest model ResNeXt-101-64$\times$4d of ResNe(X)t family. Visualization results also demonstrate that the learnt parts are highly informative of the predicting class, making ViP more explainable than previous fundamental architectures. Code is available at https://github.com/kevin-ssy/ViP.

• The paper presents the Visual Parser (ViP) model that leverages transformers to explicitly represent part-whole hierarchies in visual data.
• The methodology employs an iterative encoder-decoder mechanism with attention to refine both part-level and whole-level image features.
• Experimental results on ImageNet, MS COCO, and segmentation tasks demonstrate ViP’s competitive accuracy and efficiency compared to CNNs and standard transformers.

Visual Parser: Representing Part-whole Hierarchies with Transformers

The paper presents the Visual Parser (ViP), which leverages transformer architectures to explicitly model part-whole hierarchies in visual representations. The research builds on psychological evidence indicating that human vision naturally parses scenes into hierarchical structures. Importantly, the proposed ViP aims to replicate this ability by dividing visual representations into two distinct levels: the part level and the whole level, enhancing interpretability and performance in visual tasks.

Methodology and Framework

ViP utilizes an encoder-decoder mechanism to bridge the part-whole representation gap. At the core of ViP is a novel iterative process:

• Encoding: Visual information from the whole image is encoded into part vectors using attention mechanisms, capturing essential features.
• Decoding: The encoded part vectors subsequently inform a whole-level representation by redistribution through transformers, facilitating bi-directional feature refinement.

The above operations form the basic building block of the ViP which is iteratively applied to enhance feature representation across multiple levels. Unlike conventional CNN counterparts that often struggle with hierarchical representation, transformers provide the dynamic capabilities necessary for flexible neuron allocation and part activation.

Experimental Validation and Results

The ViP demonstrates robust performance across several important computer vision tasks, including classification, object detection, and instance segmentation. Its architectural flexibility enables it to perform competitively on a wide scale of model sizes:

1. On ImageNet-1K classification, ViP outperforms both CNN and transformer-based architectures with comparable or fewer parameters and operations. For instance, ViP-M achieves a top-1 accuracy of 83.3% with 49.6M parameters, highlighting its efficient use of computational resources.
2. In object detection with MS COCO, ViP is integrated with RetinaNet and Cascade Mask R-CNN. When compared with state-of-the-art models, ViP variants demonstrate superior performance, notably surpassing larger models in efficiency and precision. For instance, ViP-Ti matches the performance of significantly larger ResNeXt configurations.
3. ViP's instance segmentation capabilities also show marked improvements, attributed to the hierarchical amalgamation of part and whole-level features, translating into better precision metrics across various scales.

Implications and Future Prospects

The introduction of ViP marks a significant step towards replicating hierarchical image parsing within AI systems. Practically, ViP's architecture can be directly adopted as a superior alternative to traditional visual backbones in varied applications, from real-time detection systems to complex scene understanding. Theoretically, ViP opens avenues for further research into transformer-based dynamic attention mechanisms that better mimic human visual perception.

Future developments could explore extending the hierarchical levels or generalizing the part-whole concept to encapsulate more complex relationships within multi-modal datasets. Applying ViP to other domains, such as natural language processing or audio signal processing, remains a promising exploration to determine the generalizability of hierarchical attention models.

In conclusion, ViP represents a noteworthy advancement in the efficient and effective modeling of part-whole hierarchies, setting the stage for further innovations in transformer-based architectures within computer vision and beyond.