Papers
Topics
Authors
Recent
Search
2000 character limit reached

DiaMond: Dementia Diagnosis with Multi-Modal Vision Transformers Using MRI and PET

Published 30 Oct 2024 in cs.CV and cs.AI | (2410.23219v1)

Abstract: Diagnosing dementia, particularly for Alzheimer's Disease (AD) and frontotemporal dementia (FTD), is complex due to overlapping symptoms. While magnetic resonance imaging (MRI) and positron emission tomography (PET) data are critical for the diagnosis, integrating these modalities in deep learning faces challenges, often resulting in suboptimal performance compared to using single modalities. Moreover, the potential of multi-modal approaches in differential diagnosis, which holds significant clinical importance, remains largely unexplored. We propose a novel framework, DiaMond, to address these issues with vision Transformers to effectively integrate MRI and PET. DiaMond is equipped with self-attention and a novel bi-attention mechanism that synergistically combine MRI and PET, alongside a multi-modal normalization to reduce redundant dependency, thereby boosting the performance. DiaMond significantly outperforms existing multi-modal methods across various datasets, achieving a balanced accuracy of 92.4% in AD diagnosis, 65.2% for AD-MCI-CN classification, and 76.5% in differential diagnosis of AD and FTD. We also validated the robustness of DiaMond in a comprehensive ablation study. The code is available at https://github.com/ai-med/DiaMond.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. On the path to 2025: understanding the alzheimer’s disease continuum. Alzheimer’s research & therapy, 9:1–10, 2017.
  2. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  3. Data-driven analysis of regional brain metabolism in behavioral frontotemporal dementia and late-onset primary psychiatric diseases with frontal lobe syndrome: A pet/mri study. Neurobiology of Aging, 137:47–54, 2024.
  4. 3d transunet: Advancing medical image segmentation through vision transformers. arXiv preprint arXiv:2310.07781, 2023.
  5. The use of neuroimaging techniques in the early and differential diagnosis of dementia. Molecular Psychiatry, 28(10):4084–4097, 2023.
  6. Combined evaluation of fdg-pet and mri improves detection and differentiation of dementia. PloS one, 6(3):e18111, 2011.
  7. Deep learning framework for alzheimer’s disease diagnosis via 3d-cnn and fsbi-lstm. IEEE Access, 7:63605–63618, 2019.
  8. Multimodal transformer network for incomplete image generation and diagnosis of alzheimer’s disease. Computerized Medical Imaging and Graphics, 110:102303, 2023.
  9. H-vit: A hierarchical vision transformer for deformable image registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11513–11523, 2024.
  10. Regbn: Batch normalization of multimodal data with regularization. Advances in Neural Information Processing Systems, 36, 2024.
  11. Evaluating 2-[18f] fdg-pet in differential diagnosis of dementia using a data-driven decision model. NeuroImage: Clinical, 27:102267, 2020.
  12. H2former: An efficient hierarchical hybrid transformer for medical image segmentation. IEEE Transactions on Medical Imaging, 42(9):2763–2775, 2023.
  13. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
  14. Diagnosis of alzheimer’s disease via multi-modality 3d convolutional neural network. Frontiers in neuroscience, 13:509, 2019.
  15. Japanese and north american alzheimer’s disease neuroimaging initiative studies: Harmonization for international trials. Alzheimer’s & Dementia, 14(8):1077–1087, 2018.
  16. The alzheimer’s disease neuroimaging initiative (adni): Mri methods. Journal of magnetic resonance imaging: JMRI, 27:685–91, 05 2008.
  17. Efficient multimodel method based on transformers and coatnet for alzheimer’s diagnosis. Digital Signal Processing, 143:104229, 2023.
  18. Dfenet: A dual-branch feature enhanced network integrating transformers and convolutional feature learning for multimodal medical image fusion. Biomedical Signal Processing and Control, 80:104402, 2023.
  19. From barlow twins to triplet training: Differentiating dementia with limited data. In Medical Imaging with Deep Learning, 2024.
  20. Bidirectional mapping of brain mri and pet with 3d reversible gan for the diagnosis of alzheimer’s disease. Frontiers in Neuroscience, 15:646013, 2021.
  21. Multi-modality cascaded convolutional neural networks for alzheimer’s disease diagnosis. Neuroinformatics, 16:295–308, 2018.
  22. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
  23. Multimodal and multiscale deep neural networks for the early diagnosis of alzheimer’s disease using structural mr and fdg-pet images. Scientific reports, 8(1):5697, 2018.
  24. Metadata normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10917–10927, 2021.
  25. Mmtfn: Multi-modal multi-scale transformer fusion network for alzheimer’s disease diagnosis. International Journal of Imaging Systems and Technology, 34(1):e22970, 2024.
  26. Is a pet all you need? a multi-modal study for alzheimer’s disease using 3d cnns. In MICCAI, 2022.
  27. Interpretable differential diagnosis for alzheimer’s disease and frontotemporal dementia. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 55–65. Springer, 2022.
  28. Multiple instance neuroimage transformer. In International Workshop on PRedictive Intelligence In MEdicine, 2022.
  29. A survey of multimodal sentiment analysis. Image and Vision Computing, 65:3–14, 2017.
  30. An effective multimodal image fusion method using mri and pet for alzheimer’s disease diagnosis. Frontiers in digital health, 3:637386, 2021.
  31. Multimodal diagnosis model of alzheimer’s disease based on improved transformer. BioMedical Engineering OnLine, 23(1):8, 2024.
  32. High-quality fusion and visualization for mr-pet brain tumor images via multi-dimensional features. IEEE Transactions on Image Processing, 2024.
  33. Frontotemporal dementia: latest evidence and clinical implications. Therapeutic Advances in Psychopharmacology, 8:33 – 48, 2018.
  34. An end-to-end multimodal 3d cnn framework with multi-level features for the prediction of mild cognitive impairment. Knowledge-Based Systems, 281:111064, 2023.
  35. Transformer-based multimodal fusion for early diagnosis of alzheimer’s disease using structural mri and pet. In IEEE 20th International Symposium on Biomedical Imaging (ISBI), 2023.
  36. Effective feature learning and fusion of multimodality data using stage-wise deep neural network for dementia diagnosis. Human brain mapping, 40(3):1001–1016, 2019.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 1 like about this paper.

Sign Up for Free