Towards Better Exploiting Convolutional Neural Networks for Remote Sensing Scene Classification

Abstract: We present an analysis of three possible strategies for exploiting the power of existing convolutional neural networks (ConvNets) in different scenarios from the ones they were trained: full training, fine tuning, and using ConvNets as feature extractors. In many applications, especially including remote sensing, it is not feasible to fully design and train a new ConvNet, as this usually requires a considerable amount of labeled data and demands high computational costs. Therefore, it is important to understand how to obtain the best profit from existing ConvNets. We perform experiments with six popular ConvNets using three remote sensing datasets. We also compare ConvNets in each strategy with existing descriptors and with state-of-the-art baselines. Results point that fine tuning tends to be the best performing strategy. In fact, using the features from the fine-tuned ConvNet with linear SVM obtains the best results. We also achieved state-of-the-art results for the three datasets used.

• The paper demonstrates that fine tuning pre-trained ConvNets outperforms full training and feature extraction, achieving up to 99.47% accuracy on the UCMerced dataset.
• The paper compares three strategies using six popular ConvNet architectures across three diverse datasets to assess performance under varying spectral conditions.
• The paper highlights the practical benefits of transfer learning in remote sensing, reducing computational demands and the need for extensive labeled data.

Towards Better Exploiting Convolutional Neural Networks for Remote Sensing Scene Classification

Overview

The paper "Towards Better Exploiting Convolutional Neural Networks for Remote Sensing Scene Classification" presents a thorough analysis of three distinct strategies for leveraging the capabilities of Convolutional Neural Networks (ConvNets) in the field of remote sensing. Specifically, the strategies examined are full training, fine tuning, and using ConvNets as feature extractors. The paper systematically evaluates these strategies using six popular ConvNet architectures across three different remote sensing datasets, ultimately concluding that fine tuning tends to offer the best performance.

Experimental Setup and Methodology

The authors conducted experiments using three datasets: UCMerced Land-use, RS19, and Brazilian Coffee Scenes. Each dataset presents unique challenges such as variations in color, texture, and spectral information. This diversity is crucial for a comprehensive assessment of the ConvNet strategies.

Strategies Assessed

1. Full Training: This approach involves training a ConvNet from scratch on the target dataset. While this offers the potential for the network to learn dataset-specific features, it requires significant computational resources and large volumes of labeled data—a common limitation in remote sensing applications.
2. Fine Tuning: This is a transfer learning approach where a pre-trained ConvNet is further trained on the new dataset. Fine tuning can involve updating all layers or just the later layers of the network. This strategy leverages pre-trained weights, thus reducing the data and computational demands compared to full training.
3. Feature Extraction: Here, a pre-trained ConvNet is used as a fixed feature extractor. The features are then classified using an external classifier, such as a linear SVM. This method significantly decreases the computational overhead and is particularly useful when data is scarce.

Results

Numerical Findings

• The study indicates that fine tuning pre-trained ConvNets achieves the highest classification accuracy across different datasets. For instance, using fine-tuned GoogLeNet and linear SVM classifier achieved an impressive accuracy of 99.47% for the UCMerced dataset.
• Full training was found less effective for the aerial datasets (UCMerced and RS19) but showed competitive results for the Brazilian Coffee Scenes dataset. The latter dataset benefited from full training due to its significantly different spectral properties.
• Using ConvNets as feature extractors also produced robust results, especially in well-established aerial datasets, but lagged behind fine-tuned networks.

Comparative Analysis with Classical Methods

When compared to traditional methods such as low-level descriptors (e.g., SASI, BIC) and mid-level Bag of Visual Words (BoVW) representations, ConvNets, particularly fine-tuned ones, demonstrated superior performance. This comparative analysis establishes the strong generalization capabilities of deep learning models even when applied to domains for which they were not explicitly trained.

Implications

The findings have significant practical and theoretical implications. Fine tuning emerges as a versatile and effective strategy for adapting ConvNets to new datasets, especially where labeled data is limited. This emphasizes the potential of transfer learning in remote sensing applications.

Moreover, using ConvNets as feature extractors offers a computationally efficient alternative, though it may not always match the performance of fine-tuning. Full training, while resource-intensive, still holds value for datasets markedly different from the domains typical ConvNets are pre-trained on.

Future Directions

Further research could explore the relationship between dataset size, the number of ConvNet parameters, and the effectiveness of these strategies. Additionally, the methodologies presented could be extended to other domains beyond remote sensing to validate their general applicability. An interesting avenue would be to investigate hybrid approaches that combine elements of these strategies or to employ advanced techniques like domain adaptation.

Conclusion

This paper comprehensively evaluates three strategies for leveraging ConvNets in remote sensing scene classification, establishing fine tuning as the most effective approach. By achieving state-of-the-art results across multiple datasets, this work underscores the adaptability and robustness of ConvNets, paving the way for their broader application in remote sensing and potentially other fields with limited data availability. The provided analyses offer valuable insights into optimizing ConvNet performance, facilitating more informed decision-making in the deployment of deep learning solutions.