FGP: Feature-Gradient-Prune for Efficient Convolutional Layer Pruning

Abstract: To reduce computational overhead while maintaining model performance, model pruning techniques have been proposed. Among these, structured pruning, which removes entire convolutional channels or layers, significantly enhances computational efficiency and is compatible with hardware acceleration. However, existing pruning methods that rely solely on image features or gradients often result in the retention of redundant channels, negatively impacting inference efficiency. To address this issue, this paper introduces a novel pruning method called Feature-Gradient Pruning (FGP). This approach integrates both feature-based and gradient-based information to more effectively evaluate the importance of channels across various target classes, enabling a more accurate identification of channels that are critical to model performance. Experimental results demonstrate that the proposed method improves both model compactness and practicality while maintaining stable performance. Experiments conducted across multiple tasks and datasets show that FGP significantly reduces computational costs and minimizes accuracy loss compared to existing methods, highlighting its effectiveness in optimizing pruning outcomes. The source code is available at: https://github.com/FGP-code/FGP.

• The paper proposes a novel pruning method that integrates feature and gradient information to assess channel importance in CNNs.
• The method dynamically ranks and retains channels based on integrated support values, achieving significant FLOPs reduction with minimal accuracy loss.
• FGP demonstrates robust performance across diverse tasks and CNN architectures, making it ideal for deployment on resource-constrained devices.

Feature-Gradient-Prune: Advancements in Convolutional Layer Pruning

Introduction

The paper "FGP: Feature-Gradient-Prune for Efficient Convolutional Layer Pruning" (2411.12781) proposes a novel pruning methodology to enhance the efficiency of Convolutional Neural Networks (CNNs). The need for reducing computational overhead and memory usage in CNNs has driven research in model pruning techniques. This study introduces the Feature-Gradient Pruning (FGP) approach, which leverages both feature and gradient information to assess the importance of convolutional channels. Unlike traditional methods that often fail to capture inter-channel relationships or contribution to task optimization, FGP aims to refine the channel evaluation mechanism leading to more accurate pruning while preserving model performance.

Methodology

Pruning Framework

FGP integrates both feature-based and gradient-based evaluations in identifying critical channels within CNN layers. It uses the strengths of feature information for a global understanding and gradient information for task-specific channel contributions. The core steps in FGP involve the calculation of channel importance using integrated information, ranking of channels based on their overall contributions across all classes, and dynamically retaining top-performing channels for pruning.

Figure 1: The visualization results show one channel from each of the four convolutional layers, along with heatmaps for three classes. FGP retains Channel-2 and 4, and pruned model keeps only those channels with strong support values across all classes.

Heatmap Utilization

The methodology employs heatmaps generated from feature and gradient data to gauge channel significance. Experiments highlighted the correlation between heatmap values and task accuracy, underscoring the heatmap's efficacy as a pruning criterion. By consolidating feature gradient activations, FGP effectively determines the channels that maintain optimal performance across diverse target classes.

Figure 2: FGP pruning framework, where $\text{Conv}_j$ is used as an example. The process calculates each channel’s support value across all classes in the dataset.

Experimental Results

The experimental evaluations span multiple datasets and model architectures, demonstrating FGP's robustness across both image classification and segmentation tasks. By applying FGP to CNNs such as VGG-16 and ResNet-50 on datasets like CIFAR-10, CIFAR-100, CamVid, and Cityscapes, the paper illustrates significant FLOPs reduction and parameter efficiency. Notably, FGP manages to maintain accuracy close to that of the unpruned models, highlighting its effectiveness.

Performance Metrics

FGP's performance was measured through standard metrics such as parameter count, FLOPs, and task-specific accuracy (e.g., mIoU for segmentation tasks). Compared to existing pruning strategies, FGP consistently showed better compactness and a minimal drop in performance, reinforcing the superiority of its dual-criteria pruning mechanism.

Figure 3: Parametric experiments with Top $k$ and classes demonstrate the pruning impact on model accuracy and efficiency.

Figure 4: Accuracy for different channel retention configurations showcases the effectiveness of FGP’s dynamic channel selection.

Implications and Future Work

The FGP method posits notable implications for deploying CNNs on resource-constrained devices. The ability to dynamically prune channels based on integrated feature and gradient information ensures that the pruned models retain high performance, making them suitable for real-time applications where computational resources and latency are critical factors.

Future research directions may include extending FGP to non-convolutional architectures and exploring its adaptability within emerging fields such as federated learning, where model efficiency is paramount. Additionally, investigating the integration of FGP with other model optimization techniques could further enhance its efficacy.

Conclusion

FGP introduces a significant advancement in convolutional layer pruning, effectively combining feature and gradient information for comprehensive channel evaluation. Its applicability across various tasks and architectures underscores its potential as a standard for efficient CNN deployment. This pioneering approach not only optimizes computational resources but also paves the way for further innovations in model acceleration strategies.