Papers
Topics
Authors
Recent
Search
2000 character limit reached

Double Banking on Knowledge: Customized Modulation and Prototypes for Multi-Modality Semi-supervised Medical Image Segmentation

Published 23 Oct 2024 in cs.CV | (2410.17565v1)

Abstract: Multi-modality (MM) semi-supervised learning (SSL) based medical image segmentation has recently gained increasing attention for its ability to utilize MM data and reduce reliance on labeled images. However, current methods face several challenges: (1) Complex network designs hinder scalability to scenarios with more than two modalities. (2) Focusing solely on modality-invariant representation while neglecting modality-specific features, leads to incomplete MM learning. (3) Leveraging unlabeled data with generative methods can be unreliable for SSL. To address these problems, we propose Double Bank Dual Consistency (DBDC), a novel MM-SSL approach for medical image segmentation. To address challenge (1), we propose a modality all-in-one segmentation network that accommodates data from any number of modalities, removing the limitation on modality count. To address challenge (2), we design two learnable plug-in banks, Modality-Level Modulation bank (MLMB) and Modality-Level Prototype (MLPB) bank, to capture both modality-invariant and modality-specific knowledge. These banks are updated using our proposed Modality Prototype Contrastive Learning (MPCL). Additionally, we design Modality Adaptive Weighting (MAW) to dynamically adjust learning weights for each modality, ensuring balanced MM learning as different modalities learn at different rates. Finally, to address challenge (3), we introduce a Dual Consistency (DC) strategy that enforces consistency at both the image and feature levels without relying on generative methods. We evaluate our method on a 2-to-4 modality segmentation task using three open-source datasets, and extensive experiments show that our method outperforms state-of-the-art approaches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. Q. Dou et al., “Unpaired multi-modal segmentation via knowledge distillation,” IEEE Transactions on Medical Imaging, vol. 39, no. 7, pp. 2415–2425, 2020.
  2. X. Chen et al., “Mass: Modality-collaborative semi-supervised segmentation by exploiting cross-modal consistency from unpaired ct and mri images,” Medical Image Analysis, vol. 80, p. 102506, 2022.
  3. S. Zhang et al., “Multi-modal contrastive mutual learning and pseudo-label re-learning for semi-supervised medical image segmentation,” Medical Image Analysis, vol. 83, p. 102656, 2023.
  4. L. Zhu et al., “Semi-supervised unpaired multi-modal learning for label-efficient medical image segmentation,” in Proceedings of the Medical Image Computing and Computer Assisted Intervention (MICCAI), pp. 394–404, 2021.
  5. A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” in Proceedings of the Advances in Neural Information Processing Systems (NIPS), vol. 30, 2017.
  6. Z. Zhou et al., “Generalizable medical image segmentation via random amplitude mixup and domain-specific image restoration,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 420–436, 2022.
  7. F. Wu and X. Zhuang, “Unsupervised domain adaptation with variational approximation for cardiac segmentation,” IEEE Transactions on Medical Imaging, vol. 40, no. 12, pp. 3555–3567, 2021.
  8. S. Guo et al., “Causal knowledge fusion for 3d cross-modality cardiac image segmentation,” Information Fusion, vol. 99, p. 101864, 2023.
  9. J.-Y. Zhu et al., “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2223–2232, 2017.
  10. M. Jaderberg et al., “Spatial transformer networks,” Proceedings of the Advances in Neural Information Processing Systems (NIPS), vol. 28, 2015.
  11. O. Ronneberger et al., “U-net: Convolutional networks for biomedical image segmentation,” in Proceedings of the Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 234–241, 2015.
  12. Z. Yang et al., “Hypernetwork-based physics-driven personalized federated learning for ct imaging,” IEEE Transactions on Neural Networks and Learning Systems, 2023.
  13. X. Li et al., “Fedbn: Federated learning on non-iid features via local batch normalization,” arXiv preprint arXiv:2102.07623, 2021.
  14. W. Xia et al., “Ct reconstruction with pdf: Parameter-dependent framework for data from multiple geometries and dose levels,” IEEE Transactions on Medical Imaging, vol. 40, no. 11, pp. 3065–3076, 2021.
  15. S. Laine and T. Aila, “Temporal ensembling for semi-supervised learning,” arXiv preprint arXiv:1610.02242, 2016.
  16. X. Chen et al., “Semi-supervised semantic segmentation with cross pseudo supervision,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2613–2622, 2021.
  17. H. Peiris et al., “Uncertainty-guided dual-views for semi-supervised volumetric medical image segmentation,” Nature Machine Intelligence, 2023.
  18. V. V. Valindria et al., “Multi-modal learning from unpaired images: Application to multi-organ segmentation in ct and mri,” in Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 547–556, 2018.
  19. D. Nie et al., “Fully convolutional networks for multi-modality isointense infant brain image segmentation,” in Proceedings of the IEEE International Symposium on Biomedical Imaging (ISBI), pp. 1342–1345, 2016.
  20. G. van Tulder and M. de Bruijne, “Learning cross-modality representations from multi-modal images,” IEEE Transactions on Medical Imaging, vol. 38, no. 2, pp. 638–648, 2018.
  21. C. Pei et al., “Disentangle domain features for cross-modality cardiac image segmentation,” Medical Image Analysis, vol. 71, p. 102078, 2021.
  22. T. Cover and P. Hart, “Nearest neighbor pattern classification,” IEEE Transactions on Information Theory, vol. 13, no. 1, pp. 21–27, 1967.
  23. B. J. Knowlton and L. R. Squire, “The learning of categories: Parallel brain systems for item memory and category knowledge,” Science, vol. 262, no. 5140, pp. 1747–1749, 1993.
  24. E. H. Rosch, “Natural categories,” Cognitive Psychology, vol. 4, no. 3, pp. 328–350, 1973.
  25. G. Li et al., “Adaptive prototype learning and allocation for few-shot segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8334–8343, 2021.
  26. T. Zhou et al., “Rethinking semantic segmentation: A prototype view,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2582–2593, 2022.
  27. Z. Zhou et al., “Generalizable cross-modality medical image segmentation via style augmentation and dual normalization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20856–20865, 2022.
  28. Y. M. Asano et al., “Self-labelling via simultaneous clustering and representation learning,” arXiv preprint arXiv:1911.05371, 2019.
  29. M. Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transport,” Proceedings of the Advances in Neural Information Processing Systems (NIPS), vol. 26, 2013.
  30. H. Jeffreys, “An invariant form for the prior probability in estimation problems,” Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, vol. 186, pp. 453–461, 1946.
  31. “Mm-whs: Multi-modality whole heart segmentation.” https://zmiclab.github.io/zxh/0/mmwhs/, 2023.
  32. “Multi-atlas labeling beyond the cranial vault - workshop and challenge.” https://www.synapse.org/\#!Synapse:syn3193805/wiki/89480, 2015.
  33. A. E. Kavur et al., “Chaos challenge - combined (ct-mr) healthy abdominal organ segmentation,” Medical Image Analysis, vol. 69, p. 101950, 2021.
  34. L. Yu et al., “Uncertainty-aware self-ensembling model for semi-supervised 3d left atrium segmentation,” in Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 605–613, 2019.
  35. V. Verma et al., “Interpolation consistency training for semi-supervised learning,” Neural Networks, vol. 145, pp. 90–106, 2022.
  36. Y. Chen et al., “Evidence-based uncertainty-aware semi-supervised medical image segmentation,” Computers in Biology and Medicine, vol. 170, p. 108004, 2024.
  37. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  38. Y. Shi et al., “Variational mixture-of-experts autoencoders for multi-modal deep generative models,” Proceedings of the Advances in Neural Information Processing Systems (NIPS), vol. 32, 2019.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up