2000 character limit reached
ARQ: A Mixed-Precision Quantization Framework for Accurate and Certifiably Robust DNNs
Published 31 Oct 2024 in cs.LG, cs.CR, and cs.CV | (2410.24214v2)
Abstract: Mixed precision quantization has become an important technique for optimizing the execution of deep neural networks (DNNs). Certified robustness, which provides provable guarantees about a model's ability to withstand different adversarial perturbations, has rarely been addressed in quantization due to unacceptably high cost of certifying robustness. This paper introduces ARQ, an innovative mixed-precision quantization method that not only preserves the clean accuracy of the smoothed classifiers but also maintains their certified robustness. ARQ uses reinforcement learning to find accurate and robust DNN quantization, while efficiently leveraging randomized smoothing, a popular class of statistical DNN verification algorithms. ARQ consistently performs better than multiple state-of-the-art quantization techniques across all the benchmarks and the input perturbation levels. The performance of ARQ quantized networks reaches that of the original DNN with floating-point weights, but with only 1.5% instructions and the highest certified radius. ARQ code is available at https://anonymous.4open.science/r/ARQ-FE4B.
- Josh Achiam. Spinning up in deep rl. https://spinningup.openai.com/en/latest/algorithms/ddpg.html#the-policy-learning-side-of-ddpg, 2018.
- Gradient ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT regularization for quantization robustness, 2020. URL https://arxiv.org/abs/2002.07520.
- Towards mixed-precision quantization of neural networks via constrained optimization, 2021.
- Pact: Parameterized clipping activation for quantized neural networks, 2018.
- The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404–413, 1934. ISSN 00063444. URL http://www.jstor.org/stable/2331986.
- Certified adversarial robustness via randomized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 1310–1320. PMLR, 2019. URL http://proceedings.mlr.press/v97/cohen19c.html.
- Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009.
- Hawq: Hessian aware quantization of neural networks with mixed-precision, 2019.
- Releq: A reinforcement learning approach for deep quantization of neural networks, 2020.
- How many bits does it take to quantize your neural network? In Armin Biere and David Parker (eds.), Tools and Algorithms for the Construction and Analysis of Systems, pp. 79–97, Cham, 2020. Springer International Publishing. ISBN 978-3-030-45237-7.
- Model compression with adversarial robustness: A unified optimization framework, 2019a.
- Model compression with adversarial robustness: A unified optimization framework, 2019b. URL https://arxiv.org/abs/1902.03538.
- Single path one-shot neural architecture search with uniform sampling, 2020.
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices, pp. 815–832. Springer International Publishing, 2018. ISBN 9783030012342. doi: 10.1007/978-3-030-01234-2_48. URL http://dx.doi.org/10.1007/978-3-030-01234-2_48.
- Adam: A method for stochastic optimization, 2017.
- Cifar-10 (canadian institute for advanced research). 2009. URL http://www.cs.toronto.edu/~kriz/cifar.html.
- A tunable robust pruning framework through dynamic network rewiring of dnns, 2020. URL https://arxiv.org/abs/2011.03083.
- Quantization-aware interval bound propagation for training certifiably robust quantized neural networks, 2022. URL https://arxiv.org/abs/2211.16187.
- Sok: Certified robustness for deep neural networks. In 2023 IEEE symposium on security and privacy (SP), pp. 1289–1310. IEEE, 2023.
- Continuous control with deep reinforcement learning, 2019. URL https://arxiv.org/abs/1509.02971.
- Integer-arithmetic-only certified robustness for quantized neural networks, 2021. URL https://arxiv.org/abs/2108.09413.
- Defensive quantization: When efficiency meets robustness, 2019a.
- Defensive quantization: When efficiency meets robustness, 2019b. URL https://arxiv.org/abs/1904.08444.
- Autoq: Automated kernel-wise neural network quantization, 2020.
- Bayesian compression for deep learning, 2017.
- Priyadarshini Panda. Quanos- adversarial noise sensitivity driven hybrid quantization of neural networks, 2020. URL https://arxiv.org/abs/2004.11233.
- Stochastic-shield: A probabilistic approach towards training-free adversarial defense in quantized cnns, 2021. URL https://arxiv.org/abs/2105.06512.
- Certified defenses against adversarial examples. arXiv preprint arXiv:1801.09344, 2018.
- Towards compact and robust deep neural networks, 2019. URL https://arxiv.org/abs/1906.06110.
- Hydra: Pruning adversarially robust neural networks, 2020. URL https://arxiv.org/abs/2002.10509.
- Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural network. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pp. 764–775, 2018. doi: 10.1109/ISCA.2018.00069.
- Boosting robustness certification of neural networks. In International Conference on Learning Representations, 2019.
- Mixed-precision neural network quantization via learned layer-wise importance, 2023.
- Proof transfer for fast certification of multiple approximate neural networks. Proc. ACM Program. Lang., 6(OOPSLA1), April 2022. doi: 10.1145/3527319. URL https://doi.org/10.1145/3527319.
- Incremental verification of neural networks. Proceedings of the ACM on Programming Languages, 7(PLDI):1920–1945, June 2023. ISSN 2475-1421. doi: 10.1145/3591299. URL http://dx.doi.org/10.1145/3591299.
- Incremental randomized smoothing certification. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=SdeAPV1irk.
- Haq: Hardware-aware automated quantization with mixed precision, 2019.
- Towards certificated model robustness against weight perturbations. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):6356–6363, 2020. doi: 10.1609/aaai.v34i04.6105. URL https://doi.org/10.1609/aaai.v34i04.6105.
- Mixed precision quantization of convnets via differentiable neural architecture search, 2018.
- Randomized smoothing of all shapes and sizes, 2020.
- Hawqv3: Dyadic neural network quantization, 2021. URL https://arxiv.org/abs/2011.10680.
- Adversarial robustness vs model compression, or both?, 2021. URL https://arxiv.org/abs/1903.12561.
- Black-box certification with randomized smoothing: A functional optimization based framework. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2316–2326. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1896a3bf730516dd643ba67b4c447d36-Paper.pdf.
- A branch and bound framework for stronger adversarial attacks of ReLU networks. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 26591–26604, 2022.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.