Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification from the Bottom Up

Abstract: Given a training dataset composed of images and corresponding category labels, deep convolutional neural networks show a strong ability in mining discriminative parts for image classification. However, deep convolutional neural networks trained with image level labels only tend to focus on the most discriminative parts while missing other object parts, which could provide complementary information. In this paper, we approach this problem from a different perspective. We build complementary parts models in a weakly supervised manner to retrieve information suppressed by dominant object parts detected by convolutional neural networks. Given image level labels only, we first extract rough object instances by performing weakly supervised object detection and instance segmentation using Mask R-CNN and CRF-based segmentation. Then we estimate and search for the best parts model for each object instance under the principle of preserving as much diversity as possible. In the last stage, we build a bi-directional long short-term memory (LSTM) network to fuze and encode the partial information of these complementary parts into a comprehensive feature for image classification. Experimental results indicate that the proposed method not only achieves significant improvement over our baseline models, but also outperforms state-of-the-art algorithms by a large margin (6.7%, 2.8%, 5.2% respectively) on Stanford Dogs 120, Caltech-UCSD Birds 2011-200 and Caltech 256.

• The paper proposes a weakly supervised multi-stage pipeline using Mask R-CNN and LSTMs to detect, model, and encode complementary object parts for enhanced fine-grained image classification.
• The model achieves state-of-the-art results, improving accuracy by up to 6.7% on datasets like Stanford Dogs 120, CUB-200, and Caltech 256 by effectively leveraging complementary part information.
• This work demonstrates the efficacy of weakly supervised learning for enhancing parts-based models and suggests future research directions in refining part modeling and applying the approach to other fine-grained discrimination tasks.

Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification

The paper under review presents a novel approach to enhancing fine-grained image classification by leveraging weakly supervised learning to identify and integrate complementary object parts. Traditional deep convolutional neural networks (CNNs) often excel at locating the most discriminative parts of an image but tend to neglect other potentially informative components. This work proposes a complementary parts model to counteract this limitation, allowing the capture of additional part information from the image that supports more robust classification outcomes.

Overview

The proposed methodology involves a multi-stage pipeline executed in a weakly supervised manner to enrich image classification models with complementary object parts. The process begins with weakly supervised object detection and instance segmentation using a combination of Mask R-CNN and CRF-based segmentation techniques. Subsequently, the authors model the detected object parts to maximize diversity and complementarity, which are then encoded through a bi-directional long short-term memory (LSTM) network for enhanced image classification. This approach is shown to achieve significant improvements over baseline models, surpassing state-of-the-art algorithms on benchmark datasets such as Stanford Dogs 120, Caltech-UCSD Birds 2011-200, and Caltech 256 by substantial margins.

Methodology

The paper introduces three primary stages in its framework:

1. Weakly Supervised Object Detection and Segmentation: Utilizing image-level labels, the framework performs object detection and segmentations iteratively, leveraging a deep classification network for initial CAM generation and refining instance masks through CRFs. This phase refines the localization and segmentation of object parts using a Mask R-CNN network iteratively trained on the pseudo labels derived from CRFs.
2. Complementary Parts Model: The model aims to exploit the rich information contained within object proposals post-detection by identifying informative parts that complement each other. The criterion for these parts involves minimizing informational redundancy while maximizing the coverage of object information. The search for optimal parts is performed using a heuristic that balances object information with part diversity.
3. Image Classification with Context Encoding: The final stage introduces a sophisticated use of bi-directional stacked LSTMs to encode the multifaceted information from the identified parts into a comprehensive classification feature. This approach explores both content and context by linearly combining part features and overall image features to inform the classification decision.

Results and Implications

The experimental results on high-profile datasets demonstrate that the proposed model consistently outperforms state-of-the-art fine-grained classifiers. Notable improvements include a 6.7% accuracy boost on Stanford Dogs 120, a 2.8% increase on Caltech-UCSD Birds 2011-200, and a 5.2% gain on Caltech 256, highlighting the efficacy of leveraging complementary information. The proposed method provides further insights into how weakly supervised learning can be effectively harnessed to address limitations in parts-based models and image feature extraction.

Future Directions

This work establishes groundwork for future exploration in weakly supervised learning for fine-grained image classification. Potential avenues for development include the refinement of the parts model generation and exploration of different network architectures to enhance the fusion of complementary part information. Moreover, the approach could be adapted to other domains requiring fine-grained discrimination, such as medical imaging and document analysis, providing a broad spectrum of applications. The pursuit of a more efficient and accurate complementary parts model formulation represents an intriguing and impactful research direction.

By addressing the fundamental challenge of enhancing model comprehension of objects in their entirety, this paper contributes to the discourse on leveraging weakly supervised object detection to achieve superior performance in image classification tasks.