Fine-Grained Image Analysis with Deep Learning: A Survey

Abstract: Fine-grained image analysis (FGIA) is a longstanding and fundamental problem in computer vision and pattern recognition, and underpins a diverse set of real-world applications. The task of FGIA targets analyzing visual objects from subordinate categories, e.g., species of birds or models of cars. The small inter-class and large intra-class variation inherent to fine-grained image analysis makes it a challenging problem. Capitalizing on advances in deep learning, in recent years we have witnessed remarkable progress in deep learning powered FGIA. In this paper we present a systematic survey of these advances, where we attempt to re-define and broaden the field of FGIA by consolidating two fundamental fine-grained research areas -- fine-grained image recognition and fine-grained image retrieval. In addition, we also review other key issues of FGIA, such as publicly available benchmark datasets and related domain-specific applications. We conclude by highlighting several research directions and open problems which need further exploration from the community.

• The paper provides a comprehensive survey of deep learning-based fine-grained image analysis, integrating both recognition and retrieval approaches.
• The paper details advanced methodologies such as localization-classification subnetworks, end-to-end feature encoding, deep metric learning, and multi-modal matching to overcome subtle inter-class variations.
• The paper outlines future directions including enhanced benchmarking, model interpretability, few-shot learning, and applications to 3D tasks, addressing current open challenges.

Deep Learning for Fine-Grained Image Analysis: A Comprehensive Survey

This essay provides an in-depth survey of recent advancements in fine-grained image analysis (FGIA) driven by deep learning techniques. The survey broadens the scope of FGIA by integrating fine-grained image recognition and retrieval, offering a consolidated view of the field's landscape, challenges, and future directions.

FGIA: Recognition vs. Retrieval

Traditional surveys in FGIA have primarily focused on fine-grained recognition, neglecting the crucial role of fine-grained retrieval. This survey addresses this gap by providing a comprehensive overview of both areas, highlighting their synergies and differences. Fine-grained recognition is framed as a closed-world task with a fixed number of subordinate categories, while fine-grained retrieval is characterized as an open-world problem with unlimited sub-categories. Despite these differences, both tasks share common techniques, such as deep metric learning and multi-modal matching, and can benefit from each other's advancements.

(Figure 1)

Figure 1: Fine-grained image analysis vs. generic image analysis (using visual classification as an example).

Problem Formulation and Challenges

FGIA focuses on analyzing images belonging to multiple subordinate categories within the same meta-category. The core challenge lies in discerning subtle visual differences between objects that are highly similar in overall appearance but differ in fine-grained features. The survey formally defines FGIA, contrasting it with generic image analysis and instance-level analysis. It highlights the key challenges of FGIA, including small inter-class variations and large intra-class variations in pose, scale, and rotation.

(Figure 2)

Figure 2: Overview of the landscape of deep learning based fine-grained image analysis (FGIA), as well as future directions.

Benchmark Datasets for FGIA

The availability of benchmark datasets has been crucial for the progress of FGIA. The survey provides an extensive overview of publicly available datasets, covering diverse domains such as birds, dogs, cars, airplanes, flowers, food, fashion, and retail products. Each dataset is characterized by its meta-category, number of images, number of categories, and available supervision, including bounding boxes, part annotations, hierarchical labels, attributes, and text descriptions. Noteworthy datasets include CUB200-2011, iNat2017, and RPC, which pose unique challenges such as large-scale data, hierarchical structure, domain gaps, and long-tailed distributions.

(Figure 3)

Figure 3: An illustration of fine-grained image analysis which lies in the continuum between the basic-level category analysis (i.e., generic image analysis) and the instance-level analysis (e.g., car identification).

Fine-Grained Image Recognition Techniques

The survey categorizes fine-grained recognition approaches into three main paradigms: recognition by localization-classification subnetworks, recognition by end-to-end feature encoding, and recognition with external information. Recognition by localization-classification subnetworks involves capturing discriminative semantic parts of fine-grained objects and constructing mid-level representations for classification. Techniques in this paradigm include employing detection or segmentation, utilizing deep filters, and leveraging attention mechanisms. Recognition by end-to-end feature encoding aims to learn a unified, discriminative image representation by performing high-order feature interactions and designing novel loss functions. The third paradigm, recognition with external information, leverages web data, multi-modal data, or human-computer interactions to enhance recognition performance.

(Figure 4)

Figure 4: Key challenges of fine-grained image analysis, i.e., small inter-class variations and large intra-class variations. Here we present four different Tern species from \cite{WahCUB200_2011}, one species per row, with different instances in the columns.

Fine-Grained Image Retrieval Methods

Fine-grained retrieval aims to rank images based on their relevance to a query, focusing on subtle differences between fine-grained categories. The survey distinguishes between content-based fine-grained image retrieval (FG-CBIR) and sketch-based fine-grained image retrieval (FG-SBIR). FG-CBIR methods utilize deep learning to select meaningful deep descriptors and employ supervised metric learning to improve retrieval accuracy. FG-SBIR addresses the sketch-photo domain gap by training a joint embedding space where sketches and photos can be compared.

(Figure 5)

Figure 5: An illustration of fine-grained content-based image retrieval (FG-CBIR). Given a query image (aka probe) depicting a Dodge Charger Sedan 2012'', fine-grained retrieval is required to return images of the same car model from a car database (aka galaxy). In this figure, the fourth returned image, marked with a red outline, is incorrect as it is a different car model, it is aDodge Caliber Wagon 2012''.

Shared Techniques in Recognition and Retrieval

Fine-grained recognition and retrieval share common techniques, including deep metric learning and multi-modal matching. Deep metric learning maps image data to an embedding space where similar images are close together and dissimilar images are far apart. Multi-modal matching methods leverage textual information and visual cues to boost performance in both tasks.

(Figure 6)

Figure 6: An illustration of fine-grained sketch-based image retrieval (FG-SBIR), where a free-hand human sketch serves as the query for instance-level retrieval of images. FG-SBIR is challenging due to 1) the fine-grained and cross-domain nature of the task and 2) free-hand sketches are highly abstract, making fine-grained matching even more difficult.

Future Directions and Open Problems

The survey identifies several future directions and open problems in FGIA. These include developing a precise definition of "fine-grained," creating next-generation fine-grained datasets, applying FGIA to 3D tasks, obtaining robust and interpretable fine-grained representations, exploring fine-grained few-shot learning, developing fine-grained hashing techniques, automating fine-grained models, and analyzing FGIA in more realistic settings.

Figure 7: Chronological overview of representative deep learning based fine-grained recognition methods which are categorized by different learning approaches. (Best viewed in color.)

Conclusion

This survey provides a comprehensive overview of recent advances in deep learning-based FGIA, advocating for a broadened definition that integrates fine-grained recognition and retrieval. It highlights gaps in existing research, points out emerging topics, and underscores important future research directions, emphasizing that the problem of FGIA is still far from solved.