A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction

Abstract: Spatial transcriptomics (ST) enables gene expression mapping within anatomical context but remains costly and low-throughput. Hematoxylin and eosin (H&E) staining offers rich morphology yet lacks molecular resolution. We present \textbf{\ours} (\textbf{S}patial \textbf{T}ranscriptomics and hist\textbf{O}logy \textbf{R}epresentation \textbf{M}odel), a foundation model trained on 1.2 million spatially resolved transcriptomic profiles with matched histology across 18 organs. Using a hierarchical architecture integrating morphological features, gene expression, and spatial context, STORM bridges imaging and omics through robust molecular--morphological representations. STORM enhances spatial domain discovery, producing biologically coherent tissue maps, and outperforms existing methods in predicting spatial gene expression from H&E images across 11 tumor types. The model is platform-agnostic, performing consistently across Visium, Xenium, Visium HD, and CosMx. Applied to 23 independent cohorts comprising 7,245 patients, STORM significantly improves immunotherapy response prediction and prognostication over established biomarkers, providing a scalable framework for spatially informed discovery and clinical precision medicine.

• The paper introduces STORM, a hierarchical multimodal model that integrates spatial transcriptomics and histology for precise biomarker imputation and risk stratification.
• The paper details a two-level encoder architecture with masked modeling pretraining, achieving significant improvements in gene expression prediction across various cancers.
• The paper demonstrates enhanced spatial domain discovery and clinical prognosis prediction, outperforming previous models on diverse tissues and technical platforms.

A Foundation Model for Multimodal Spatial Transcriptomics and Histology in Biological Discovery and Clinical Prediction

Introduction

The integration of spatial transcriptomics (ST) with histological imaging is critical for resolving the interplay between tissue morphology and molecular state. The manuscript "A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction" (2604.03630) introduces STORM—a hierarchical, spatially aware multimodal foundation model trained on an unprecedented scale: 1.2 million matched spatial transcriptomic profiles and high-resolution hematoxylin and eosin (H&E) stained histological images from 18 organs. The model is designed for unified representation learning, cross-platform generalizability, and strong performance in molecular discovery, clinical biomarker imputation, and outcome prognostication.

Figure 1: Overview of the STORM architecture, data diversity (18 organs, four platforms), and applications spanning discovery and translational prediction.

STORM Architecture and Pretraining Strategy

STORM leverages a two-level hierarchical encoder. At the spot level, H&E image and ST feature encoders (pretrained on pathology and transcriptomics foundation models, respectively) produce dense embeddings. At the spatial level, a transformer-based spatial encoder jointly processes features from neighborhoods of tissue spots, capturing both local and extended tissue organization. The multimodal integration is realized by a masked modeling pretraining objective: neighborhoods of tissue spots are randomly masked in either modality and reconstructed, requiring the model to develop spatially holistic, context-aware representations. The architecture is designed to be agnostic to ST platform and compatible with variable gene panels and resolutions, ensuring cross-technology generalizability.

Multimodal Spatial Domain Discovery and Biological Coherence

STORM produces highly resolved, biologically meaningful spatial domain maps. In Visium datasets comprising kidney and lung cancers, STORM multimodal embeddings yield maximal domain concordance (NMI/ARI up to 0.62/0.60 in lung cancer) and spatial homogeneity (HOM = 0.68), outperforming unimodal and prior foundation models (e.g., OmiCLIP NMI/ARI of 0.41/0.32, and all others <0.3/<0.2). Unlike gene-only approaches, STORM discriminates subtle architectural subdomains, including tertiary lymphoid structures (TLS), immune infiltration (INFL), and metabolically rewired tumor regions. Notably, STORM embeddings of pathologist-annotated “tumor” compartments revealed further metabolically and immunologically specialized subdomains not visually apparent, with gene signatures validated as prognostic in external TCGA-KIRC cohorts.

Figure 2: Multimodal spatial domain discovery in kidney and lung cancer samples, with fine parcellation of tumor and immune microenvironments and validation via prognostically relevant gene signatures.

STORM’s ability to differentiate functionally heterogeneous subregions—such as plasma cell-enriched versus Tfh/NK-enriched immune niches—has direct implications for immuno-oncology and spatial immunogenomics.

Virtual Spatial Transcriptomics Prediction from H&E

A central capability of STORM is the accurate imputation of spatial gene expression from H&E images, enabling practical ST at scale with routine clinical slides. Across 8 tumor types (85k+ spots), STORM surpasses 13 state-of-the-art baselines and all prior foundation models. In colorectal cancer, STORM achieves median Pearson correlation coefficient (PCC) of 0.633 (versus 0.304 for DeepPT, 0.266 for Virchow), with similar dominance across GBM (0.622), bladder (0.494), lung (0.545), and prostate (0.520). The accuracy is maintained even for rare or small panels, with STORM’s outperformance persisting up to high gene counts (2,000 genes).

Figure 3: Comparative performance of STORM in predicting gene expression from H&E, with gene-wise PCC distributions and spatial visualization for clinically actionable biomarkers.

Exemplar genes such as FGFR3 (bladder), ERBB2/HER2 (breast), MET (renal), COL1A1 (liver), and EGFR (lung) are reconstructed at high fidelity ($r=0.77$, $r=0.69$, $r=0.84$, $r=0.70$, $r=0.70$, respectively; Virchow typically 20–60% lower), establishing the utility of STORM for virtual molecular panels.

Cross-Platform and Histology Generalization

STORM is robust to under-represented tissue types and technical platform variation. On external datasets spanning lymphoma, mesothelioma, and ovarian cancer (absent or marginalized in pretraining), STORM retains generalization with median PCC improvements of 25–55% over the next best pathology foundation models. When evaluated on high-resolution, single-cell ST technologies (Xenium, Visium HD, CosMx), STORM maintains consistent performance (e.g., ovarian, PCC=0.295) and outperforms all scalable baselines, highlighting a platform-agnostic representation that does not overfit to platform or gene panel idiosyncrasies.

Prognosis Prediction and Multimodal Risk Stratification

STORM’s multimodal framework enhances patient-level prognostic modeling. In large colorectal and lung cancer cohorts (n>2,000 across training/validation/testing in real-world datasets such as PLCO, SURGEN, NLST), the joint integration of routine H&E with virtual ST outperforms single-modality, task-specific, and other foundation models. STORM delivers the highest C-indexes (0.701–0.741) and can risk-stratify patients into high and low-risk subgroups with HRs up to 3.5, significant even after adjusting for grade and stage (CRC multivariate HR=3.32, 95% CI 1.98–5.57, p<0.001; lung HR=2.38, 95% CI 1.67–3.38, p<10^{-6}).

Figure 4: Prognosis prediction workflow and evaluation, demonstrating robust improvement in risk stratification across CRC and lung cancer cohorts, validated in independent datasets.

Integration of clinical variables with STORM features further improves risk prediction, and subgroup analysis demonstrates added value for adjuvant therapy decision-making within TNM stage-matched cohorts.

Model Interpretability: Tissue, Cellular, and Molecular Axes

STORM provides interpretable risk predictions through attribution-based analysis at tissue, cell, and gene levels. Integrated Gradients identify high-/low-risk image regions congruent with adverse (micropapillary subtype, high tumor budding, infiltrative growth) and favorable (lymphocytic infiltration, pushing margin) histological phenotypes. Cell composition analysis (using HistoPLUS segmentation) detects expected enrichment patterns: cancer, fibroblast, and endothelial cell fractions are associated with increased risk, while lymphocytes and plasma cells denote low risk. At the gene level, analysis in lung cancer consistently retrieves robustly validated prognostic and regulatory genes (CD74, HLA-A/C, B2M, HSP90AA1/1B, EPAS1, EZR, XPO1) and key molecular pathways (IFN-γ, G2-M checkpoint, PI3K/AKT/mTOR, hypoxia, TNFα/NFκB, Myc, UPR, TGFβ, IL-6/JAK/STAT3).

Figure 5: Multilevel interpretability—tissue-ROI, cell, and gene-level analysis—establishing STORM’s predictions as grounded in pathologically and biologically substantive features.

Implications and Future Directions

STORM defines a new baseline for cross-modal integration in spatial omics, demonstrating that large-scale, masked pretraining on diverse transcriptomic and morphological inputs enables robust discovery and translational inference. The explicit spatial encoder and platform-agnostic design position STORM as a practical tool for both research and digital pathology, accelerating precision medicine via virtual biomarker inference, spatial biology-guided risk modeling, and subdomain-level mapping.

Practically, routine deployment of STORM on H&E WSIs can deliver high-resolution molecular annotation without incurring the cost/throughput barriers of experimental ST. This enables scalable biomarker imputation, improved risk stratification, and selection for adjuvant or targeted therapies based on latent molecular–morphological patterns that are currently inaccessible in clinical workflows. Theoretically, the hierarchical approach and robust cross-domain transfer of STORM motivate the extension of foundation model pretraining into further spatial omics modalities (e.g., proteomics, multiplex IF, spatial ATAC-seq) and broader tissue types.

Potential future developments include: (1) expansion to pan-cancer, pan-organ inference with even larger datasets, (2) patient-specific or longitudinal spatial biology modeling, and (3) deeper integration with clinical systems for real-time therapeutic decision support. The open-source release of the model architecture and code base will allow further academic validation and cross-cohort benchmarking.

Conclusion

The STORM model substantially advances the state-of-the-art in integrated spatial transcriptomics and histology analysis. Its demonstration of robust, generalizable, and interpretable molecular–morphological representation learning across spatial domain discovery, virtual gene expression prediction, and clinical prognosis positions STORM as a foundational resource for both spatial biology research and clinical translation. Its design establishes the blueprint for next-generation multimodal foundation models in computational pathology and precision medicine.