Papers
Topics
Authors
Recent
Search
2000 character limit reached

3D Brain and Heart Volume Generative Models: A Survey

Published 12 Oct 2022 in eess.IV and cs.CV | (2210.05952v2)

Abstract: Generative models such as generative adversarial networks and autoencoders have gained a great deal of attention in the medical field due to their excellent data generation capability. This paper provides a comprehensive survey of generative models for three-dimensional (3D) volumes, focusing on the brain and heart. A new and elaborate taxonomy of unconditional and conditional generative models is proposed to cover diverse medical tasks for the brain and heart: unconditional synthesis, classification, conditional synthesis, segmentation, denoising, detection, and registration. We provide relevant background, examine each task and also suggest potential future directions. A list of the latest publications will be updated on Github to keep up with the rapid influx of papers at https://github.com/csyanbin/3D-Medical-Generative-Survey.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (145)
  1. Medical image denoising system based on stacked convolutional autoencoder for enhancing 2-dimensional gel electrophoresis noise reduction. Biomedical Signal Processing and Control 69 (2021), 102842.
  2. Stanford AIMI. 2022. COCA - Coronary Calcium and chest CT’s dataset. https://stanfordaimi.azurewebsites.net/datasets/e8ca74dc-8dd4-4340-815a-60b41f6cb2aa
  3. Deep learning for brain MRI segmentation: state of the art and future directions. Journal of digital imaging 30, 4 (2017), 449–459.
  4. Manal AlAmir and Manal AlGhamdi. 2022. The Role of Generative Adversarial Network in Medical Image Analysis: An in-depth survey. ACM Computing Surveys (CSUR) (2022).
  5. The role of generative adversarial networks in brain MRI: a scoping review. Insights into Imaging 13, 1 (2022), 1–15.
  6. Xcat-gan for synthesizing 3d consistent labeled cardiac mr images on anatomically variable xcat phantoms. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 128–137.
  7. Alexander Andreopoulos and John K Tsotsos. 2008. Efficient and generalizable statistical models of shape and appearance for analysis of cardiac MRI. Medical image analysis 12, 3 (2008), 335–357.
  8. Wasserstein generative adversarial networks. In International conference on machine learning. PMLR, 214–223.
  9. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientific data 4, 1 (2017), 1–13.
  10. An unsupervised learning model for deformable medical image registration. In Proceedings of the IEEE conference on computer vision and pattern recognition. 9252–9260.
  11. VoxelMorph: a learning framework for deformable medical image registration. IEEE transactions on medical imaging 38, 8 (2019), 1788–1800.
  12. Subject-specific lesion generation and pseudo-healthy synthesis for multiple sclerosis brain images. In International Workshop on Simulation and Synthesis in Medical Imaging. Springer, 1–11.
  13. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE transactions on medical imaging 37, 11 (2018), 2514–2525.
  14. Learning interpretable anatomical features through deep generative models: Application to cardiac remodeling. In International conference on medical image computing and computer-assisted intervention. Springer, 464–471.
  15. Ali Borji. 2019. Pros and cons of gan evaluation measures. Computer Vision and Image Understanding 179 (2019), 41–65.
  16. Hervé Bourlard and Yves Kamp. 1988. Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics 59, 4 (1988), 291–294.
  17. Automatic Time-Resolved Cardiovascular Segmentation of 4D Flow MRI Using Deep Learning. Journal of Magnetic Resonance Imaging (2022).
  18. Med3d: Transfer learning for 3d medical image analysis. arXiv preprint arXiv:1904.00625 (2019).
  19. S3D-UNet: separable 3D U-Net for brain tumor segmentation. In International MICCAI Brainlesion Workshop. Springer, 358–368.
  20. QSMGAN: improved quantitative susceptibility mapping using 3D generative adversarial networks with increased receptive field. NeuroImage 207 (2020), 116389.
  21. Generative adversarial networks in medical image augmentation: a review. Computers in Biology and Medicine (2022), 105382.
  22. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2014).
  23. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 8789–8797.
  24. Chee Keong Chong and Eric Tatt Wei Ho. 2021. Synthesis of 3D MRI brain images with shape and texture generative adversarial deep neural networks. IEEE Access 9 (2021), 64747–64760.
  25. Introduction to the non-rigid image registration evaluation project (NIREP). In International workshop on biomedical image registration. Springer, 128–135.
  26. Two-stage deep learning for accelerated 3D time-of-flight MRA without matched training data. Medical Image Analysis 71 (2021), 102047.
  27. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention. Springer, 424–432.
  28. Vox2Vox: 3D-GAN for brain tumour segmentation. In International MICCAI Brainlesion Workshop. Springer, 274–284.
  29. Brainweb: Online interface to a 3D MRI simulated brain database. In NeuroImage. Citeseer.
  30. Natural language processing (almost) from scratch. Journal of machine learning research 12, ARTICLE (2011), 2493–2537.
  31. MSSEG challenge proceedings: multiple sclerosis lesions segmentation challenge using a data management and processing infrastructure. In Miccai.
  32. ADHD-200 consortium. 2012. The ADHD-200 consortium: a model to advance the translational potential of neuroimaging in clinical neuroscience. Frontiers in systems neuroscience 6 (2012), 62.
  33. Harvard aging brain study: dataset and accessibility. Neuroimage 144 (2017), 255–258.
  34. ResViT: Residual vision transformers for multi-modal medical image synthesis. arXiv preprint arXiv:2106.16031 (2021).
  35. Image synthesis in multi-contrast MRI with conditional generative adversarial networks. IEEE transactions on medical imaging 38, 10 (2019), 2375–2388.
  36. IXI Dataset. 2010. IXI brain development homepage. https://brain-development.org/ixi-dataset/
  37. Submillisievert coronary calcium quantification using model-based iterative reconstruction: a within-patient analysis. European journal of radiology 85, 11 (2016), 2152–2159.
  38. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.
  39. Challenges of Deep Learning in Medical Image Analysis—Improving Explainability and Trust. IEEE Transactions on Technology and Society 4, 1 (2023), 68–75.
  40. Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems 34 (2021), 8780–8794.
  41. The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism. Molecular psychiatry 19, 6 (2014), 659–667.
  42. VoxelAtlasGAN: 3D left ventricle segmentation on echocardiography with atlas guided generation and voxel-to-voxel discrimination. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 622–629.
  43. Deep complex convolutional network for fast reconstruction of 3D late gadolinium enhancement cardiac MRI. NMR in Biomedicine 33, 7 (2020), e4312.
  44. GP-GAN: Brain tumor growth prediction using stacked 3D generative adversarial networks from longitudinal MR Images. Neural Networks 132 (2020), 321–332.
  45. Neural architecture search: A survey. The Journal of Machine Learning Research 20, 1 (2019), 1997–2017.
  46. Automatic liver segmentation using U-Net with Wasserstein GANs. Journal of Image and Graphics 6, 2 (2018), 152–159.
  47. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12873–12883.
  48. Farzan Farnia and Asuman Ozdaglar. 2020. Do GANs always have Nash equilibria?. In International Conference on Machine Learning. PMLR, 3029–3039.
  49. Bruce Fischl. 2012. FreeSurfer. Neuroimage 62, 2 (2012), 774–781.
  50. Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning. PMLR, 1050–1059.
  51. An improved Sobel edge detection. In 2010 3rd International conference on computer science and information technology, Vol. 5. IEEE, 67–71.
  52. Ross Girshick. 2015. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision. 1440–1448.
  53. The MCIC collection: a shared repository of multi-modal, multi-site brain image data from a clinical investigation of schizophrenia. Neuroinformatics 11, 3 (2013), 367–388.
  54. Lovedeep Gondara. 2016. Medical image denoising using convolutional denoising autoencoders. In 2016 IEEE 16th international conference on data mining workshops (ICDMW). IEEE, 241–246.
  55. Generative adversarial nets. Advances in neural information processing systems 27 (2014).
  56. SlabGAN: a method for generating efficient 3D anisotropic medical volumes using generative adversarial networks. In Medical Imaging 2021: Image Processing, Vol. 11596. SPIE, 329–335.
  57. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing. Ieee, 6645–6649.
  58. Deep autoregressive networks. In International Conference on Machine Learning. PMLR, 1242–1250.
  59. Improved training of wasserstein gans. Advances in neural information processing systems 30 (2017).
  60. Whole Heart Segmentation Using 3D FM-Pre-ResNet Encoder–Decoder Based Architecture with Variational Autoencoder Regularization. Applied Sciences 11, 9 (2021), 3912.
  61. MADGAN: Unsupervised medical anomaly detection GAN using multiple adjacent brain MRI slice reconstruction. BMC bioinformatics 22, 2 (2021), 1–20.
  62. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16000–16009.
  63. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision. 2961–2969.
  64. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  65. Monica Hernandez. 2014. Gauss–Newton inspired preconditioned optimization in large deformation diffeomorphic metric mapping. Physics in Medicine & Biology 59, 20 (2014), 6085.
  66. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017).
  67. GE Hinton. 1994. RS n emel. Autoencoders, minimum description length, and Helmholtz free energy. J. D. C owan, G. Tesauro, and J. Alspector, editors, dvances in£ e u ra l I n f ormation rocessin g ystems q. Morgan K aufmann P ublishers, San Francisco, C A (1994).
  68. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine 29, 6 (2012), 82–97.
  69. Improved techniques for training single-image gans. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1300–1309.
  70. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33 (2020), 6840–6851.
  71. Cascaded Diffusion Models for High Fidelity Image Generation. J. Mach. Learn. Res. 23 (2022), 47–1.
  72. CANDIShare: A Resource for Pediatric Neuroimaging Data. In Front. Neuroinform. Conference Abstract: Neuroinformatics.
  73. Brain Genomics Superstruct Project initial data release with structural, functional, and behavioral measures. Scientific data 2, 1 (2015), 1–16.
  74. 3d-stylegan: A style-based generative adversarial network for generative modeling of three-dimensional medical images. In Deep Generative Models, and Data Augmentation, Labelling, and Imperfections. Springer, 24–34.
  75. 3D brain MRI reconstruction based on 2D super-resolution technology. In 2020 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, 18–23.
  76. Prospective study on the mismatch concept in acute stroke patients within the first 24 h after symptom onset-1000Plus study. BMC neurology 9, 1 (2009), 1–8.
  77. Few-shot learning for multi-label intent detection. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 13036–13044.
  78. Mcmt-gan: multi-task coherent modality transferable gan for 3d brain image synthesis. IEEE Transactions on Image Processing 29 (2020), 8187–8198.
  79. Generative adversarial networks and its applications in the biomedical image segmentation: a comprehensive survey. International Journal of Multimedia Information Retrieval (2022), 1–36.
  80. iSeg 2017. 2022. 6-month infant MRI brain segmentation. https://iseg2017.web.unc.edu/reference/
  81. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1125–1134.
  82. Spatial transformer networks. Advances in neural information processing systems 28 (2015).
  83. Systematic review of generative adversarial networks (gans) for medical image classification and segmentation. Journal of Digital Imaging, 1–16.
  84. 3D convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence 35, 1 (2012), 221–231.
  85. Thomas Joyce and Sebastian Kozerke. 2019. 3D medical image synthesis by factorised representation and deformable model learning. In International Workshop on Simulation and Synthesis in Medical Imaging. Springer, 110–119.
  86. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 (2017).
  87. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4401–4410.
  88. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8110–8119.
  89. GANs for medical image analysis. Artificial Intelligence in Medicine 109 (2020), 101938.
  90. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT. 4171–4186.
  91. Boah Kim and Jong Chul Ye. 2022. Diffusion Deformable Model for 4D Temporal Medical Image Generation. arXiv preprint arXiv:2206.13295 (2022).
  92. Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
  93. Mindboggling morphometry of human brains. PLoS computational biology 13, 2 (2017), e1005350.
  94. Planar 3D Transfer Learning for End to End Unimodal MRI Unbalanced Data Segmentation. In 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 6051–6058.
  95. Resource efficient 3d convolutional neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 0–0.
  96. Learning a generative motion model from image sequences based on a latent motion matrix. IEEE Transactions on Medical Imaging 40, 5 (2021), 1405–1416.
  97. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012).
  98. CBIR system using Capsule Networks and 3D CNN for Alzheimer’s disease diagnosis. Informatics in Medicine Unlocked 14 (2019), 59–68.
  99. Standardized assessment of automatic segmentation of white matter hyperintensities and results of the WMH segmentation challenge. IEEE transactions on medical imaging 38, 11 (2019), 2556–2568.
  100. Generation of 3D brain MRI using auto-encoding generative adversarial networks. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 118–126.
  101. OASIS-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and Alzheimer disease. MedRxiv (2019).
  102. SC-GAN: 3D self-attention conditional GAN with spectral normalization for multi-modal neuroimaging synthesis. bioRxiv (2020).
  103. Hugo Larochelle and Iain Murray. 2011. The neural autoregressive distribution estimator. In Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 29–37.
  104. Yann LeCun. 1987. PhD thesis: Modeles connexionnistes de l’apprentissage (connectionist learning models). (1987).
  105. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324.
  106. Pop-Out Motion: 3D-Aware Image Deformation via Learning the Shape Laplacian. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18532–18541.
  107. Michael M Lell and Marc Kachelrieß. 2020. Recent and upcoming technological developments in computed tomography: high speed, low dose, deep learning, multienergy. Investigative radiology 55, 1 (2020), 8–19.
  108. A novel public MR image dataset of multiple sclerosis patients with lesion segmentations based on multi-rater consensus. Neuroinformatics 16, 1 (2018), 51–63.
  109. Chuan Li and Michael Wand. 2016. Precomputed real-time texture synthesis with markovian generative adversarial networks. In European conference on computer vision. Springer, 702–716.
  110. Shape-aware semi-supervised 3D semantic segmentation for medical images. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 552–561.
  111. More knowledge is better: Cross-modality volume completion and 3D+ 2D segmentation for intracardiac echocardiography contouring. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 535–543.
  112. A large, open source dataset of stroke anatomical brain images and manual lesion segmentations. Scientific data 5, 1 (2018), 1–11.
  113. Wanyun Lin. 2020. Synthesizing missing data using 3D reversible GAN for alzheimer’s disease. In Proceedings of the 2020 international symposium on artificial intelligence in medical sciences. 208–213.
  114. Bidirectional mapping of brain MRI and PET with 3D reversible GAN for the diagnosis of Alzheimer’s disease. Frontiers in Neuroscience 15 (2021), 646013.
  115. Glioma subregions segmentation with a discriminative adversarial regularized 3D Unet. In Proceedings of the Third International Symposium on Image Computing and Digital Medicine. 269–273.
  116. Few-shot unsupervised image-to-image translation. In Proceedings of the IEEE/CVF international conference on computer vision. 10551–10560.
  117. Deep learning in medical ultrasound analysis: a review. Engineering 5, 2 (2019), 261–275.
  118. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering (2021).
  119. Inflating 2D Convolution Weights for Efficient Generation of 3D Medical Images. Computer Methods and Programs in Biomedicine (2023), 107685.
  120. Learning to propagate labels: Transductive propagation network for few-shot learning. In 7th International Conference on Learning Representations, ICLR 2019.
  121. A 3D fully convolutional neural network with top-down attention-guided refinement for accurate and robust automatic segmentation of amygdala and its subnuclei. Frontiers in Neuroscience 14 (2020), 260.
  122. Deep learning based brain tumor segmentation: a survey. Complex & Intelligent Systems (2022), 1–26.
  123. Alexander Selvikvåg Lundervold and Arvid Lundervold. 2019. An overview of deep learning in medical imaging focusing on MRI. Zeitschrift für Medizinische Physik 29, 2 (2019), 102–127.
  124. ISLES 2015-A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI. Medical image analysis 35 (2017), 250–269.
  125. Open Access Series of Imaging Studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults. Journal of cognitive neuroscience 19, 9 (2007), 1498–1507.
  126. The Parkinson progression marker initiative (PPMI). Progress in neurobiology 95, 4 (2011), 629–635.
  127. MRBrainS challenge: online evaluation framework for brain image segmentation in 3T MRI scans. Computational intelligence and neuroscience 2015 (2015).
  128. Strategies for training large scale neural network language models. In 2011 IEEE Workshop on Automatic Speech Recognition & Understanding. IEEE, 196–201.
  129. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV). IEEE, 565–571.
  130. Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014).
  131. Making a “completely blind” image quality analyzer. IEEE Signal processing letters 20, 3 (2012), 209–212.
  132. Few-shot 3d multi-modal medical image segmentation using generative adversarial learning. arXiv preprint arXiv:1810.12241 (2018).
  133. NAMIC Multimodality. 2022. NAMIC Multimodality dataset download link. https://insight-journal.org/midas/collection/view/190
  134. Clinical evaluation of an arterial-spin-labeling product sequence in steno-occlusive disease of the brain. PLoS One 9, 2 (2014), e87143.
  135. Andriy Myronenko. 2018. 3D MRI brain tumor segmentation using autoencoder regularization. In International MICCAI Brainlesion Workshop. Springer, 311–320.
  136. Data harmonisation for information fusion in digital healthcare: A state-of-the-art systematic review, meta-analysis and future research directions. Information Fusion 82 (2022), 99–122.
  137. Medical image segmentation with 3D convolutional neural networks: A survey. Neurocomputing 493 (2022), 397–413.
  138. Detection and quantification of left atrial structural remodeling with delayed-enhancement magnetic resonance imaging in patients with atrial fibrillation. Circulation 119, 13 (2009), 1758–1767.
  139. Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning. PMLR, 2642–2651.
  140. Few-shot image generation via cross-domain correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10743–10752.
  141. Self-supervised Learning for Few-shot Medical Image Segmentation. IEEE Transactions on Medical Imaging (2022).
  142. Three dimensional mr image synthesis with progressive generative adversarial networks. arXiv preprint arXiv:2101.05218 (2020).
  143. Synthesizing missing PET from MRI with cycle-consistent generative adversarial networks for Alzheimer’s disease diagnosis. In International conference on medical image computing and computer-assisted intervention. Springer, 455–463.
  144. Running experiments on amazon mechanical turk. Judgment and Decision making 5, 5 (2010), 411–419.
  145. Deep structural causal models for tractable counterfactual inference. Advances in Neural Information Processing Systems 33 (2020), 857–869.
Citations (5)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up