MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models

Abstract: Histopathological analysis is a cornerstone of cancer diagnosis, with Hematoxylin and Eosin (H&E) staining routinely acquired for every patient to visualize cell morphology and tissue architecture. On the other hand, multiplex immunofluorescence (mIF) enables more precise cell type identification via proteomic markers, but has yet to achieve widespread clinical adoption due to cost and logistical constraints. To bridge this gap, we introduce MIPHEI (Multiplex Immunofluorescence Prediction from H&E), a U-Net-inspired architecture that integrates state-of-the-art ViT foundation models as encoders to predict mIF signals from H&E images. MIPHEI targets a comprehensive panel of markers spanning nuclear content, immune lineages (T cells, B cells, myeloid), epithelium, stroma, vasculature, and proliferation. We train our model using the publicly available ORION dataset of restained H&E and mIF images from colorectal cancer tissue, and validate it on two independent datasets. MIPHEI achieves accurate cell-type classification from H&E alone, with F1 scores of 0.88 for Pan-CK, 0.57 for CD3e, 0.56 for SMA, 0.36 for CD68, and 0.30 for CD20, substantially outperforming both a state-of-the-art baseline and a random classifier for most markers. Our results indicate that our model effectively captures the complex relationships between nuclear morphologies in their tissue context, as visible in H&E images and molecular markers defining specific cell types. MIPHEI offers a promising step toward enabling cell-type-aware analysis of large-scale H&E datasets, in view of uncovering relationships between spatial cellular organization and patient outcomes.

• The paper introduces MIPHEI-ViT, a U-Net model integrating ViT foundation models to predict multiplex immunofluorescence signals from H&E images, bridging the gap between standard and advanced pathology techniques.
• The model demonstrates superior performance in predicting markers like Pan-CK, CD3e, and SMA compared to baselines, leveraging training on ORION and validation on HEMIT and IMMUcan datasets for generalization.
• MIPHEI-ViT offers a scalable, cost-effective approach for accessing detailed cellular analysis from H&E slides, enabling retrospective studies and potentially transforming pathology workflows by leveraging vast archival data.

MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models

MIPHEI-ViT presents a sophisticated approach for translating hematoxylin and eosin (H&E) stained images into multiplex immunofluorescence (mIF) signals. This paper addresses the logistical and cost challenges associated with widespread adoption of mIF in clinical settings by introducing a computational model capable of predicting mIF signals using standard H&E images. The model integrates Vision Transformer (ViT) foundation models within a U-Net architecture, thereby leveraging advancements in representation learning to enhance performance in histopathological image translation.

The importance of histopathological analysis is widely acknowledged in cancer diagnostics. H&E staining is a routine procedure used globally to assess tissue architecture and cell morphology, but it lacks the precision of mIF imaging, which identifies cell types and functional markers via specific protein expressions. This study bridges the gap between these methodologies by focusing on predicting mIF signals from conventional H&E slides, thus reducing the need for mIF-specific equipment and reagents.

Methodology and Experimental Setup

MIPHEI utilizes a novel U-Net model incorporating ViT foundation models as encoders. This architecture is tuned for histopathological image translation tasks, reinforcing multiscale feature extraction capabilities that are pivotal for capturing the complex relationships between nuclear morphologies and molecular markers. The foundation model employed—H-optimus-0—is finetuned using Low-Rank Adaptation (LoRA), enhancing adaptability while maintaining low computational overhead.

Training is performed using the ORION dataset, which consists of colorectal cancer tissue images with matched H&E and mIF data. Testing utilizes the HEMIT and IMMUcan datasets to examine generalization across different domains, ensuring robustness in real-world applications. The experimental evaluation combines pixel-level metrics with cell-type classification, leveraging a logistic regression classifier on extracted single-cell data, thus emphasizing biologically relevant outputs.

Performance and Benchmarking

MIPHEI demonstrated superior performance across several protein markers compared to baseline models, including Pan-CK, CD3e, and SMA, achieving substantial improvements in F1 scores over methods such as HEMIT. The paper highlights MIPHEI's robust generalization capabilities, achieving high correlation in cell-type counts across domains, even when presented with histological variations and technology shifts in mIF imaging.

Implications and Future Directions

Practically, MIPHEI's ability to predict multiplexed protein signals from H&E images offers a scalable approach to increase accessibility to detailed cellular analyses without the prohibitive costs and complexities of mIF technology. This method holds promise for the retrospective analysis of vast archival H&E datasets, providing insights into spatial cellular organization and its impact on treatment outcomes and survival.

Theoretically, the integration of ViT within U-Net architectures represents a compelling evolution in computational pathology, suggesting that future work might extend this approach to other molecular imaging modalities and undertake further refinement in predicting clinically relevant scores such as Immunoscore.

In conclusion, MIPHEI-ViT suggests considerable potential for transforming pathology workflows by harnessing advanced machine learning techniques to facilitate detailed cellular analysis from readily available H&E-stained slides. As more extensive datasets and broader domain adaptation strategies are explored, the model's applicability is poised to expand, contributing to enhanced biomarker discovery and hypothesis generation in oncology.