Papers
Topics
Authors
Recent
Search
2000 character limit reached

MS-Glance: Bio-Insipred Non-semantic Context Vectors and their Applications in Supervising Image Reconstruction

Published 31 Oct 2024 in eess.IV, cs.AI, and cs.CV | (2410.23577v2)

Abstract: Non-semantic context information is crucial for visual recognition, as the human visual perception system first uses global statistics to process scenes rapidly before identifying specific objects. However, while semantic information is increasingly incorporated into computer vision tasks such as image reconstruction, non-semantic information, such as global spatial structures, is often overlooked. To bridge the gap, we propose a biologically informed non-semantic context descriptor, \textbf{MS-Glance}, along with the Glance Index Measure for comparing two images. A Global Glance vector is formulated by randomly retrieving pixels based on a perception-driven rule from an image to form a vector representing non-semantic global context, while a local Glance vector is a flattened local image window, mimicking a zoom-in observation. The Glance Index is defined as the inner product of two standardized sets of Glance vectors. We evaluate the effectiveness of incorporating Glance supervision in two reconstruction tasks: image fitting with implicit neural representation (INR) and undersampled MRI reconstruction. Extensive experimental results show that MS-Glance outperforms existing image restoration losses across both natural and medical images. The code is available at \url{https://github.com/Z7Gao/MSGlance}.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (85)
  1. Non-semantic facial parts for face verification. In 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), pages 725–729. IEEE, 2015.
  2. Chentao Cao et al. High-frequency space diffusion model for accelerated mri. IEEE Transactions on Medical Imaging, 43(5):1853–1865, 2024.
  3. Localisation and mapping of self-driving vehicles based on fuzzy k-means clustering: A non-semantic approach. In 2022 International Conference on Intelligent Systems and Computer Vision (ISCV), pages 1–8. IEEE, 2022.
  4. Videoinr: Learning video implicit neural representation for continuous space-time super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2047–2057, 2022.
  5. Model learning: Primal dual networks for fast mr imaging. In MICCAI, 2019.
  6. Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction. In CVPR, 2022.
  7. Score-based diffusion models for accelerated mri. Medical Image Analysis, page 102479, 2022.
  8. Fast unsupervised mri reconstruction without fully-sampled ground truth data using generative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3988–3997, 2021.
  9. Deep residual dense u-net for resolution enhancement in accelerated mri acquisition. In Medical Imaging 2019: Image Processing, volume 10949, pages 110–117. SPIE, 2019.
  10. Mri reconstruction by completing under-sampled k-space data with learnable fourier interpolation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 676–685. Springer, 2022.
  11. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
  12. Coin: Compression with implicit neural representations. arXiv preprint arXiv:2103.03123, 2021.
  13. Taejoon Eo et al. Kiki-net: cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images. Magnetic resonance in medicine, 80(5):2188–2201, 2018.
  14. A segmentation-aware deep fusion network for compressed sensing mri. In Proceedings of the European Conference on Computer Vision (ECCV), pages 55–70, 2018.
  15. Mrpd: Undersampled mri reconstruction by prompting a large latent diffusion model, 2024.
  16. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016.
  17. Mr image reconstruction using deep learning: Evaluation of network structure and loss functions. Quantitative Imaging in Medicine and Surgery, 9:1516–1527, 09 2019.
  18. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  19. The briefest of glances the time course of natural scene understanding. Psychological science, 20:464–72, 05 2009.
  20. Generalized autocalibrating partially parallel acquisitions (grappa). Magnetic Resonance in Medicine, 47(6):1202–1210, 2002.
  21. Correlated and multi-frequency diffusion modeling for highly under-sampled mri reconstruction. IEEE Transactions on Medical Imaging, 2024.
  22. Alper Güngör et al. Adaptive diffusion priors for accelerated mri reconstruction. Medical Image Analysis, 88:102872, 2023.
  23. Ralph N Haber. Visual perception. Annual review of psychology, 1978.
  24. Compressed-sensing mri with random encoding. IEEE Transactions on Medical Imaging, 30(4):893–903, 2011.
  25. Runet: A robust unet architecture for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  26. Jiahao Huang et al. Swin transformer for fast mri. Neurocomputing, 493:281–304, 2022.
  27. Jiahao Huang et al. Cdiffmr: Can we replace the gaussian noise with k-space undersampling for fast mri? In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 3–12. Springer, 2023.
  28. Fr-net: Joint reconstruction and segmentation in compressed sensing cardiac mri. In Functional Imaging and Modeling of the Heart: 10th International Conference, FIMH 2019, Bordeaux, France, June 6–8, 2019, Proceedings 10, pages 352–360. Springer, 2019.
  29. Deep learning for undersampled mri reconstruction. Physics in Medicine & Biology, 63(13):135007, 2018.
  30. Deciphering the feature representation of deep neural networks for high-performance ai. IEEE transactions on pattern analysis and machine intelligence, PP, 02 2024.
  31. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  32. Ajil Jalal et al. Robust compressed sensing mri with deep generative priors. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 14938–14954. Curran Associates, Inc., 2021.
  33. Focal frequency loss for image reconstruction and synthesis. In Proceedings of the IEEE/CVF international conference on computer vision, pages 13919–13929, 2021.
  34. Kyong Hwan Jin et al. Deep convolutional neural network for inverse problems in imaging. IEEE Transactions on Image Processing, 26(9):4509–4522, 2017.
  35. Implicit neural representations for medical imaging segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 433–443. Springer, 2022.
  36. Self-supervised mri reconstruction with unrolled diffusion models. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 491–501. Springer, 2023.
  37. Accelerated motion correction for mri using score-based generative models. In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pages 1–5, 2023.
  38. Accelerating sense using compressed sensing. Magnetic Resonance in Medicine, 62(6):1574–1584, 2009.
  39. Microsoft coco: Common objects in context, 2015.
  40. Brain-id: Learning contrast-agnostic anatomical representations for brain imaging. In European Conference on Computer Vision (ECCV), 2024.
  41. PEPSI: Pathology-enhanced pulse-sequence-invariant representations for brain MRI. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2024.
  42. Dbgan: A dual-branch generative adversarial network for undersampled mri reconstruction. Magnetic Resonance Imaging, 89:77–91, 2022.
  43. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  44. Compressed sensing mri. IEEE signal processing magazine, 25(2):72–82, 2008.
  45. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
  46. Asking friendly strangers: non-semantic attribute transfer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  47. Single image super-resolution via a dual interactive implicit neural network. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4936–4945, 2023.
  48. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision, 42:145–175, 2001.
  49. The role of context in object recognition. Trends in cognitive sciences, 11(12):520–527, 2007.
  50. Batu Ozturkler et al. Regularization by denoising diffusion process for MRI reconstruction. In NeurIPS 2023 Workshop on Deep Learning and Inverse Problems, 2023.
  51. Batu Ozturkler et al. Smrd: Sure-based robust mri reconstruction with diffusion models. In MICCAI. Springer, 2023.
  52. Gradient-based image recovery methods from incomplete fourier measurements. IEEE Transactions on Image Processing, 21(1):94–105, 2011.
  53. Cheng Peng et al. Towards performant and reliable undersampled mr reconstruction via diffusion model sampling. MICCAI, 2022.
  54. Mary C Potter. Meaning in visual search. Science, 187(4180):965–966, 1975.
  55. Joint calibrationless reconstruction and segmentation of parallel mri. In European Conference on Computer Vision, pages 437–453. Springer, 2022.
  56. Klaas P Pruessmann et al. Sense: sensitivity encoding for fast mri. Magnetic Resonance in Medicine, 42(5):952–962, 1999.
  57. Patrick Putzky et al. i-rim applied to the fastmri challenge. arXiv preprint arXiv:1910.08952, 2019.
  58. Chen Qin et al. Convolutional recurrent neural networks for dynamic mr image reconstruction. IEEE transactions on medical imaging, 38(1):280–290, 2018.
  59. Compressed sensing mri reconstruction using a generative adversarial network with a cyclic loss. IEEE transactions on medical imaging, 37(6):1488–1497, 2018.
  60. Learning transferable visual models from natural language supervision, 2021.
  61. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234–241. Springer, 2015.
  62. Jo Schlemper et al. A deep cascade of convolutional neural networks for mr image reconstruction. In IPMI, 2017.
  63. From blobs to boundary edges: Evidence for time- and spatial-scale-dependent scene recognition. Psychological Science, 5(4):195–200, 1994.
  64. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020.
  65. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  66. Anuroop Sriram et al. End-to-end variational networks for accelerated mri reconstruction. In Medical Image Computing and Computer Assisted Intervention 2020, pages 64–73. Springer, 2020.
  67. Kent Stevens. The vision of david marr. Perception, 41:1061–72, 09 2012.
  68. Implicit neural representations for image compression. In European Conference on Computer Vision, pages 74–91. Springer, 2022.
  69. A survey of deep learning approaches to image restoration. Neurocomputing, 487:46–65, 2022.
  70. ⟂perpendicular-to\perp⟂-loss: A symmetric loss function for magnetic resonance imaging reconstruction and image registration with deep learning. Medical Image Analysis, 80:102509, 2022.
  71. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689, 2021.
  72. Shanshan Wang et al. Accelerating magnetic resonance imaging via deep learning. In 2016 IEEE 13th international symposium on biomedical imaging (ISBI), pages 514–517. IEEE, 2016.
  73. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  74. A standard model for foveal detection of spatial contrast. Journal of vision, 5:717–40, 02 2005.
  75. S3im: Stochastic structural similarity and its unreasonable effectiveness for neural fields. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 17978–17988, 2023.
  76. Deep convolutional neural network for image deconvolution. Advances in neural information processing systems, 27, 2014.
  77. Guang Yang et al. Dagan: deep de-aliasing generative adversarial networks for fast compressed sensing mri reconstruction. IEEE transactions on medical imaging, 37(6):1310–1321, 2017.
  78. Deep learning for single image super-resolution: A brief review. IEEE Transactions on Multimedia, 21(12):3106–3121, 2019.
  79. fastMRI: An open dataset and benchmarks for accelerated MRI, 2018.
  80. Deep unfolding network for image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3217–3226, 2020.
  81. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
  82. The unreasonable effectiveness of deep features as a perceptual metric, 2018.
  83. Reducing uncertainty in undersampled mri reconstruction with active acquisition. In CVPR, 2019.
  84. Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging, 3(1):47–57, 2017.
  85. Dudornet: learning a dual-domain recurrent network for fast mri reconstruction with deep t1 prior. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4273–4282, 2020.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 5 likes about this paper.

Sign Up for Free