Papers
Topics
Authors
Recent
Search
2000 character limit reached

Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction

Published 3 Jan 2024 in cs.CV | (2401.01646v2)

Abstract: Multimodal learning significantly benefits cancer survival prediction, especially the integration of pathological images and genomic data. Despite advantages of multimodal learning for cancer survival prediction, massive redundancy in multimodal data prevents it from extracting discriminative and compact information: (1) An extensive amount of intra-modal task-unrelated information blurs discriminability, especially for gigapixel whole slide images (WSIs) with many patches in pathology and thousands of pathways in genomic data, leading to an intra-modal redundancy" issue. (2) Duplicated information among modalities dominates the representation of multimodal data, which makes modality-specific information prone to being ignored, resulting in aninter-modal redundancy" issue. To address these, we propose a new framework, Prototypical Information Bottlenecking and Disentangling (PIBD), consisting of Prototypical Information Bottleneck (PIB) module for intra-modal redundancy and Prototypical Information Disentanglement (PID) module for inter-modal redundancy. Specifically, a variant of information bottleneck, PIB, is proposed to model prototypes approximating a bunch of instances for different risk levels, which can be used for selection of discriminative instances within modality. PID module decouples entangled multimodal data into compact distinct components: modality-common and modality-specific knowledge, under the guidance of the joint prototypical distribution. Extensive experiments on five cancer benchmark datasets demonstrated our superiority over other methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. Deep variational information bottleneck. In International Conference on Learning Representations, 2016.
  2. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine, 25(8):1301–1309, 2019.
  3. Generalized product of experts for automatic and principled fusion of gaussian process predictions. arXiv preprint arXiv:1410.7827, 2014.
  4. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Transactions on Medical Imaging, 41(4):757–770, 2020.
  5. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  4015–4025, 2021.
  6. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  16144–16155, 2022a.
  7. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell, 40(8):865–878, 2022b.
  8. Disentangle first, then distill: A unified framework for missing modality imputation and alzheimer’s disease diagnosis. IEEE Transactions on Medical Imaging, 2023.
  9. Learning disentangled representations for counterfactual regression via mutual information minimization. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.  1802–1806, 2022.
  10. Club: A contrastive log-ratio upper bound of mutual information. In International conference on machine learning, pp.  1779–1788. PMLR, 2020.
  11. Integrated genomic analysis identifies subclasses and prognosis signatures of kidney cancer. Oncotarget, 6, 03 2015. doi: 10.18632/oncotarget.3294.
  12. David R Cox. Regression models and life-tables. Journal of the Royal Statistical Society: Series B (Methodological), 34(2):187–202, 1972.
  13. David R Cox. Partial likelihood. Biometrika, 62(2):269–276, 1975.
  14. Analysis of multimodal data fusion from an information theory perspective. Information Sciences, 623:164–183, 2023.
  15. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
  16. Us food and drug administration approval of whole slide imaging for primary diagnosis: A key milestone is reached and new questions are raised. Archives of pathology and laboratory medicine, 142, 04 2018. doi: 10.5858/arpa.2017-0496-CP.
  17. Learning robust representations via multi-view information bottleneck. In 8th International Conference on Learning Representations. OpenReview. net, 2020.
  18. Ji Feng and Zhi Hua Zhou. Deep miml network. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
  19. The reactome pathway knowledgebase 2022. Nucleic acids research, 50(D1):D687–D692, 2022.
  20. Node-aligned graph convolutional network for whole-slide image representation and classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  18813–18823, 2022.
  21. Do ssl models have d\\\backslash\’ej\\\backslash\a vu? a case of unintended memorization in self-supervised learning. arXiv preprint arXiv:2304.13850, 2023.
  22. Balázs Győrffy. Survival analysis across the entire transcriptome identifies biomarkers with the highest prognostic power in breast cancer. Computational and structural biotechnology journal, 19:4101–4109, 2021.
  23. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. arXiv preprint arXiv:2109.00412, 2021.
  24. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in medicine, 15(4):361–387, 1996.
  25. Simon Haykin. Neural networks: a comprehensive foundation. Prentice Hall PTR, 1998.
  26. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  27. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670, 2018.
  28. Computational pathology: A survey review and the way forward. arXiv preprint arXiv:2304.05482, 2023.
  29. Patch-based convolutional neural network for whole slide tissue image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  2424–2433, 2016.
  30. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ digital medicine, 3(1):136, 2020.
  31. Attention-based deep multiple instance learning. In International conference on machine learning, pp.  2127–2136. PMLR, 2018.
  32. The single-cell pathology landscape of breast cancer. Nature, 578(7796):615–620, 2020.
  33. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. arXiv preprint arXiv:2305.10160, 2023.
  34. Modeling dense multimodal interactions between biological pathways and histology for survival prediction. arXiv preprint arXiv:2304.06819, 2023.
  35. Stephen P Jenkins. Survival analysis. Unpublished manuscript, Institute for Social and Economic Research, University of Essex, Colchester, UK, 42:54–56, 2005.
  36. Identification of a nuclear mitochondrial-related multi-genes signature to predict the prognosis of bladder cancer. Frontiers in Oncology, 11:746029, 2021a.
  37. Development and validation of a deep learning ct signature to predict survival and chemotherapy benefit in gastric cancer: a multicenter, retrospective study. Annals of surgery, 274(6):e1153–e1161, 2021b.
  38. Nonparametric estimation from incomplete observations. Journal of the American statistical association, 53(282):457–481, 1958.
  39. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  40. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
  41. Changhee Lee and Mihaela Van der Schaar. A variational information bottleneck approach to multi-omics data integration. In International Conference on Artificial Intelligence and Statistics, pp.  1513–1521. PMLR, 2021.
  42. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  14318–14328, 2021.
  43. Task-specific fine-tuning via variational information bottleneck for weakly-supervised pathology whole slide image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  7454–7463, 2023a.
  44. Hfbsurv: hierarchical multimodal fusion with factorized bilinear models for cancer survival prediction. Bioinformatics, 38(9):2587–2594, 2022.
  45. Survival prediction via hierarchical multimodal co-attention transformer: A computational histology-radiology solution. IEEE Transactions on Medical Imaging, 2023b.
  46. Quantifying & modeling feature interactions: An information decomposition framework. arXiv preprint arXiv:2302.12247, 2023.
  47. The molecular signatures database hallmark gene set collection. Cell systems, 1(6):417–425, 2015.
  48. Advmil: Adversarial multiple instance learning for the survival analysis on whole-slide images. arXiv preprint arXiv:2212.06515, 2022.
  49. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555–570, 2021.
  50. Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations. IEEE Transactions on Multimedia, 2022.
  51. Nathan Mantel et al. Evaluation of survival data and two new rank order statistics arising in its consideration. Cancer Chemother Rep, 50(3):163–170, 1966.
  52. Predicting cancer outcomes from histology and genomics using convolutional networks. Proceedings of the National Academy of Sciences, 115(13):E2970–E2979, 2018.
  53. Genetic mutation and biological pathway prediction based on whole slide images in breast carcinoma using deep learning. NPJ precision oncology, 5(1):87, 2021.
  54. Stephen Salerno and Yi Li. High-dimensional survival analysis: Methods and applications. Annual review of statistics and its application, 10:25–49, 2023.
  55. Learning disentangled representations via mutual information estimation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16, pp.  205–221. Springer, 2020.
  56. A framework for learning ante-hoc explainable models via concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10286–10295, 2022.
  57. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136–2147, 2021.
  58. Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. Proceedings of the National Academy of Sciences, 102(43):15545–15550, 2005.
  59. The information bottleneck method. arXiv preprint physics/0004057, 2000.
  60. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
  61. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical Image Analysis, 81:102559, 2022. ISSN 1361-8415. doi: https://doi.org/10.1016/j.media.2022.102559. URL https://www.sciencedirect.com/science/article/pii/S1361841522002043.
  62. Gpdbn: deep bilinear network integrating both genomic data and pathological images for breast cancer prognosis prediction. Bioinformatics, 37(18):2963–2970, 2021.
  63. Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
  64. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65:101789, 2020.
  65. Deepprognosis: Preoperative prediction of pancreatic cancer survival and surgical margin via comprehensive understanding of dynamic contrast-enhanced ct imaging and tumor-vascular contact parsing. Medical image analysis, 73:102150, 2021.
  66. Bias in cross-entropy-based training of deep survival networks. IEEE transactions on pattern analysis and machine intelligence, 43(9):3126–3137, 2020.
  67. Multimodal intelligence: Representation learning, information fusion, and applications. IEEE Journal of Selected Topics in Signal Processing, 14(3):478–493, 2020.
  68. Tformer: A throughout fusion transformer for multi-modal skin lesion diagnosis. Computers in Biology and Medicine, 157:106712, 2023.
  69. Cross-modal translation and alignment for survival analysis. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
  70. Wsisa: Making survival prediction from whole slide histopathological images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  7234–7242, 2017.
  71. Identify consistent imaging genomic biomarkers for characterizing the survival-associated interactions between tumor-infiltrating lymphocytes and tumors. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp.  222–231. Springer, 2022.
Citations (16)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free