Papers
Topics
Authors
Recent
Search
2000 character limit reached

Learning A Spiking Neural Network for Efficient Image Deraining

Published 10 May 2024 in cs.CV | (2405.06277v1)

Abstract: Recently, spiking neural networks (SNNs) have demonstrated substantial potential in computer vision tasks. In this paper, we present an Efficient Spiking Deraining Network, called ESDNet. Our work is motivated by the observation that rain pixel values will lead to a more pronounced intensity of spike signals in SNNs. However, directly applying deep SNNs to image deraining task still remains a significant challenge. This is attributed to the information loss and training difficulties that arise from discrete binary activation and complex spatio-temporal dynamics. To this end, we develop a spiking residual block to convert the input into spike signals, then adaptively optimize the membrane potential by introducing attention weights to adjust spike responses in a data-driven manner, alleviating information loss caused by discrete binary activation. By this way, our ESDNet can effectively detect and analyze the characteristics of rain streaks by learning their fluctuations. This also enables better guidance for the deraining process and facilitates high-quality image reconstruction. Instead of relying on the ANN-SNN conversion strategy, we introduce a gradient proxy strategy to directly train the model for overcoming the challenge of training. Experimental results show that our approach gains comparable performance against ANN-based methods while reducing energy consumption by 54%. The code source is available at https://github.com/MingTian99/ESDNet.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. Unpaired deep image deraining using dual contrastive learning. In CVPR, pages 2017–2026, 2021.
  2. Simple baselines for image restoration. In ECCV, pages 17–33, 2022.
  3. Learning a sparse transformer network for effective image deraining. In CVPR, pages 5896–5905, 2023.
  4. Towards unified deep image deraining: A survey and a new benchmark. arXiv preprint arXiv:2310.03535, 2023.
  5. Rethinking multi-scale representations in deep deraining transformer. In AAAI, volume 38, pages 1046–1053, 2024.
  6. Bidirectional multi-scale implicit neural representations for image deraining. arXiv preprint arXiv:2404.01547, 2024.
  7. Selective frequency network for image restoration. In ICLR, 2022.
  8. Temporal efficient training of spiking neural network via gradient re-weighting. arXiv preprint arXiv:2202.11946, 2022.
  9. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In IJCNN, pages 1–8, 2015.
  10. Single image rain and snow removal via guided l0 smoothing filter. MTA, 75(5):2697–2712, 2016.
  11. Deep residual learning in spiking neural networks. NeurIPS, 34:21056–21069, 2021.
  12. Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances, 9(40):eadi1480, 2023.
  13. Lightweight pyramid networks for image deraining. IEEE TNNLS, 31(6):1794–1807, 2019.
  14. Efficientderain: Learning pixel-wise dilation filtering for high-efficiency single-image deraining. In AAAI, volume 35, pages 1487–1495, 2021.
  15. Ternary spike: Learning ternary spikes for spiking neural networks. arXiv preprint arXiv:2312.06372, 2023.
  16. Rmp-loss: Regularizing membrane potential distribution for spiking neural networks. In ICCV, pages 17391–17401, 2023.
  17. Image quality metrics: Psnr vs. ssim. In IEEE ICIP, pages 2366–2369. IEEE, 2010.
  18. Spiking deep residual networks. IEEE TNNLS, 2021.
  19. Advancing residual learning towards powerful deep spiking neural networks. arXiv e-prints, pages arXiv–2112, 2021.
  20. Multi-scale progressive fusion network for single image deraining. In CVPR, pages 8346–8355, 2020.
  21. Magic elf: Image deraining meets association learning and transformer. In ACM MM, pages 827–836, 2022.
  22. Unifying activation-and timing-based learning rules for spiking neural networks. NeurIPS, 33:19534–19544, 2020.
  23. Rain streak removal using layer priors. In ICCV, pages 2736–2744, 2016.
  24. Recurrent squeeze-and-excitation context aggregation net for single image deraining. In ECCV, pages 254–269, 2018.
  25. Drt: A lightweight single image deraining recursive transformer. In CVPRw, pages 589–598, 2022.
  26. Removing rain from a single image via discriminative sparse coding. In ICCV, pages 3397–3405, 2015.
  27. Gated attention coding for training high-performance and efficient spiking neural networks. In AAAI, 2023.
  28. Progressive image deraining networks: A better and simpler baseline. In CVPR, pages 3937–3946, 2019.
  29. Learning an effective transformer for remote sensing satellite image dehazing. IEEE GRSL, 2023.
  30. Image deraining transformer with sparsity and frequency guidance. In ICME, pages 1889–1894, 2023.
  31. Deep directly-trained spiking neural networks for object detection. In ICCV, pages 6555–6565, 2023.
  32. Blind image quality evaluation using perception based features. In NCC, pages 1–6, 2015.
  33. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4):600–612, 2004.
  34. A model-driven deep neural network for single image rain removal. In CVPR, pages 3103–3112, 2020.
  35. Hpcnet: A hybrid progressive coupled network for image deraining. In ICME, pages 2747–2752, 2023.
  36. Smartassign: Learning a smart knowledge assignment strategy for deraining and desnowing. In CVPR, pages 3677–3686, 2023.
  37. Masked spiking transformer. In ICCV, pages 1761–1771, 2023.
  38. Image de-raining transformer. IEEE TPAMI, pages 1–18, 2022.
  39. Nyströmformer: A nyström-based algorithm for approximating self-attention. In AAAI, volume 35, pages 14138–14148, 2021.
  40. Deep joint rain detection and removal from a single image. In CVPR, pages 1357–1366, 2017.
  41. Spike-driven transformer. arXiv preprint arXiv:2307.01694, 2023.
  42. Attention spiking neural networks. IEEE TPAMI, 2023.
  43. Multi-stage progressive image restoration. In CVPR, pages 14821–14831, 2021.
  44. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, pages 5728–5739, 2022.
  45. He Zhang and Vishal M Patel. Density-aware single image de-raining using a multi-stream dense network. In CVPR, pages 695–704, 2018.
  46. Image de-raining using a conditional generative adversarial network. IEEE TCSVT, 30(11):3943–3956, 2019.
  47. Going deeper with directly-trained larger spiking neural networks. In AAAI, volume 35, pages 11062–11070, 2021.
  48. Curricular contrastive regularization for physics-aware single image dehazing. In CVPR, pages 5785–5794, 2023.
  49. Spikformer: When spiking neural network meets transformer. In ICLR, 2022.
  50. Metaiqa: Deep meta-learning for no-reference image quality assessment. In CVPR, pages 14143–14152, 2020.
  51. Dreaming to prune image deraining networks. In CVPR, pages 6023–6032, 2022.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up