Papers
Topics
Authors
Recent
Search
2000 character limit reached

A Differentiable Framework for End-to-End Learning of Hybrid Structured Compression

Published 21 Sep 2023 in cs.LG, cs.AI, and eess.IV | (2309.13077v1)

Abstract: Filter pruning and low-rank decomposition are two of the foundational techniques for structured compression. Although recent efforts have explored hybrid approaches aiming to integrate the advantages of both techniques, their performance gains have been modest at best. In this study, we develop a \textit{Differentiable Framework~(DF)} that can express filter selection, rank selection, and budget constraint into a single analytical formulation. Within the framework, we introduce DML-S for filter selection, integrating scheduling into existing mask learning techniques. Additionally, we present DTL-S for rank selection, utilizing a singular value thresholding operator. The framework with DML-S and DTL-S offers a hybrid structured compression methodology that facilitates end-to-end learning through gradient-base optimization. Experimental results demonstrate the efficacy of DF, surpassing state-of-the-art structured compression methods. Our work establishes a robust and versatile avenue for advancing structured compression techniques.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (72)
  1. Low-rank compression of neural nets: Learning the rank of each layer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8049–8059, 2020.
  2. Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8018–8027, 2020.
  3. Edp: An efficient decomposition and pruning scheme for convolutional neural network compression. IEEE Transactions on Neural Networks and Learning Systems, 32(10):4499–4513, 2020.
  4. Neural network pruning by cooperative coevolution. arXiv preprint arXiv:2204.05639, 2022.
  5. Chip: Channel independence-based pruning for compact neural networks. Advances in Neural Information Processing Systems, 34:24604–24616, 2021.
  6. Discrimination-aware channel pruning for deep neural networks. arXiv preprint arXiv:1810.11809, 2018.
  7. Eie: Efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3):243–254, 2016.
  8. Faster cnns with direct sparse convolutions and guided pruning. arXiv preprint arXiv:1608.01409, 2016.
  9. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv:1808.06866, 2018a.
  10. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pages 304–320, 2018.
  11. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3296–3305, 2019.
  12. Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. Pattern Recognition, 107:107461, 2020a.
  13. Efficient neural network compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12569–12577, 2019a.
  14. Compressing neural networks: Towards determining the optimal layer-wise decomposition. Advances in Neural Information Processing Systems, 34:5328–5344, 2021.
  15. Decomposable-net: Scalable low-rank compression for neural networks. arXiv preprint arXiv:1910.13141, 2019.
  16. Combinatorial optimization: papers from the DIMACS Special Year, volume 20. American Mathematical Soc., 1995.
  17. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7370–7379, 2017.
  18. Towards compact cnns via collaborative compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6438–6447, 2021.
  19. Coreset-based neural network compression. In Proceedings of the European Conference on Computer Vision (ECCV), pages 454–470, 2018.
  20. Compressing by learning in a low-rank and sparse decomposition form. IEEE Access, 7:150823–150832, 2019.
  21. Deep neural network acceleration based on low-rank approximated channel pruning. IEEE Transactions on Circuits and Systems I: Regular Papers, 67(4):1232–1244, 2020.
  22. Exploiting weight-level sparsity in channel pruning with low-rank approximation. In 2019 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1–5. IEEE, 2019.
  23. Heuristic rank selection with progressively searching tensor ring network. Complex & Intelligent Systems, 8(2):771–785, 2022.
  24. Compression of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.06530, 2015.
  25. Coordinating filters for faster deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 658–666, 2017.
  26. Trp: Trained rank pruning for efficient deep neural networks. arXiv preprint arXiv:2004.14566, 2020.
  27. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence, 38(10):1943–1955, 2015.
  28. Compression-aware training of deep networks. Advances in neural information processing systems, 30:856–867, 2017.
  29. Chong Li and CJ Shi. Constrained optimization based low-rank approximation of deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 732–747, 2018.
  30. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016.
  31. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4340–4349, 2019a.
  32. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pages 5058–5066, 2017.
  33. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389–1397, 2017.
  34. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pages 2736–2744, 2017.
  35. Operation-aware soft channel pruning using differentiable masks. In International Conference on Machine Learning, pages 5122–5131. PMLR, 2020.
  36. Pruning from scratch. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12273–12280, 2020.
  37. Tamara Gibson Kolda. Multilinear operators for higher-order decompositions. Technical report, Sandia National Laboratories (SNL), Albuquerque, NM, and Livermore, CA …, 2006.
  38. Tensor decompositions and applications. SIAM review, 51(3):455–500, 2009.
  39. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866, 2014.
  40. Convolutional neural networks with low-rank regularization. arXiv preprint arXiv:1511.06067, 2015.
  41. A singular value thresholding algorithm for matrix completion. SIAM Journal on optimization, 20(4):1956–1982, 2010.
  42. Nimble: Lightweight and parallel gpu task scheduling for deep learning. Advances in Neural Information Processing Systems, 33:8343–8354, 2020.
  43. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.
  44. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104–12113, 2022.
  45. Trar: Routing the attention spans in transformer for visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2074–2084, 2021.
  46. Stable low-rank tensor decomposition for compression of convolutional neural network. In European Conference on Computer Vision, pages 522–539. Springer, 2020.
  47. Chex: Channel exploration for cnn model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12287–12298, 2022.
  48. Channel gating neural networks. Advances in Neural Information Processing Systems, 32, 2019.
  49. Scop: Scientific control for reliable neural network pruning. Advances in Neural Information Processing Systems, 33:10936–10947, 2020.
  50. Decore: Deep compression with reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12349–12359, 2022.
  51. Soft and hard filter pruning via dimension reduction. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2021a.
  52. Towards efficient model compression via learned global ranking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1518–1528, 2020.
  53. An effective low-rank compression with a joint rank selection followed by a compression-friendly training. Neural Networks, 2023.
  54. Differentiable pruning method for neural networks. CoRR, 2019b.
  55. Amc: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pages 784–800, 2018b.
  56. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270, 2018.
  57. Network pruning via performance maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9270–9280, 2021.
  58. Convolutional neural network pruning with structural redundancy reduction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14913–14922, 2021.
  59. Topology-aware network pruning using multi-stage graph embedding and reinforcement learning. In International Conference on Machine Learning, pages 25656–25667. PMLR, 2022.
  60. Fire together wire together: A dynamic pruning approach with self-supervised mask prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12454–12463, 2022.
  61. Asymptotic soft filter pruning for deep convolutional neural networks. IEEE transactions on cybernetics, 50(8):3594–3604, 2019b.
  62. Softer pruning, incremental regularization. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 224–230. IEEE, 2021b.
  63. Toward compact convnets via structure-sparsity regularized filter pruning. IEEE transactions on neural networks and learning systems, 31(2):574–588, 2019a.
  64. Towards optimal structured cnn pruning via generative adversarial learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2790–2799, 2019b.
  65. Weak sub-network pruning for strong and efficient neural networks. Neural Networks, 144:614–626, 2021.
  66. Neural network pruning with residual-connections and limited-data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1458–1467, 2020b.
  67. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1529–1538, 2020.
  68. Prior gradient mask guided pruning-aware fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 1, 2022.
  69. Provable filter pruning for efficient neural networks. arXiv preprint arXiv:1911.07412, 2019.
  70. Dmcp: Differentiable markov channel pruning for neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1539–1547, 2020.
  71. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv:1810.05331, 2018.
  72. Manifold regularized dynamic network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5018–5028, 2021.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up