ReMAP-PET: Beyond Visual Understanding -- Learning Region-Guided Metabolic Alignment Semantics from Brain PET

Abstract: Positron Emission Tomography (PET) reveals brain metabolism and is clinically central to neurodegenerative disease assessment, yet existing 3D brain foundation models treat PET as generic volumetric data, missing the structured regional metabolic information that distinguishes it from structural neuroimaging. To address these limitations, we propose ReMAP-PET, a framework that moves beyond visual encoding by supervising a partially-tuned MedicalNet 3D ResNet-50 with brain regional standardized uptake value ratio (SUVR) profiles through joint regression and contrastive objectives, enabling the encoder to learn the metabolic semantics underlying PET modality. On 1015 paired PET--SUVR samples, ReMAP-PET achieves 0.070 SUVR MAE and 77.8% PET SUVR Recall@1, substantially outperforming five frozen pretrained baselines. We further connect the metabolic embedding to clinical language via contrastive alignment with frozen BioClinicalBERT and demonstrate end-to-end PET-to-report generation through SUVR-constrained verbalization. Linear probing on diagnostic classification and cognitive regression tasks confirms that the embeddings retain clinically relevant information without task-specific fine-tuning. Our results show that grounding PET encoders in regional metabolic semantics -- rather than treating PET as generic volumetric data -- yields representations that are structured, interpretable, and language-compatible, pointing to a new direction for metabolic-aware PET understanding.

• The paper introduces a region-guided framework that integrates PET visual features with 120-region SUVR metabolic supervision for precise clinical alignment.
• It employs a partial-tuning strategy on a 3D ResNet-50 PET encoder with contrastive loss to enhance cross-modal semantic retrieval.
• Empirical results show superior SUVR prediction (MAE = 0.070) and diagnostic classification (AUROC = 0.946), underscoring its translational impact.

Region-Guided Metabolic Representation Learning from Brain PET: An Analysis of ReMAP-PET

Motivation and Context

Positron Emission Tomography (PET), especially with fluorodeoxyglucose (FDG), provides unique access to brain metabolic activity crucial for early detection and tracking of neurodegenerative processes. Existing 3D foundation models predominantly treat PET data as generic 3D volumes, often ignoring structured regional measurements—specifically standardized uptake value ratios (SUVR)—that embody clinically relevant metabolic information. This deviation from physiological interpretability limits their translational impact in clinical and research settings.

The ReMAP-PET framework directly addresses this limitation by integrating region-level metabolic supervision and contrastive objectives, aiming to encode metabolic semantics into the representation space beyond what visual pretraining alone can deliver.

Figure 1: Existing PET methods ignore metabolic semantics (left); ReMAP-PET encodes both visual and metabolic structure, enabling interpretability and cross-modal functionality (right).

Methodological Framework

Architecture and Partial-Tuning Strategy

ReMAP-PET employs a MedicalNet 3D ResNet-50 as a PET encoder, partially tuned by updating only the terminal residual stage (layer4), leaving early layers frozen to retain general anatomical priors. Structured supervision is provided by 120-region SUVR vectors, which are encoded by a lightweight MLP trained from scratch. The learning objective combines SUVR profile regression with a bidirectional PET–SUVR contrastive loss; this joint objective is critical for shaping the embedding space.

Figure 2: ReMAP-PET pipeline—Stage 1 aligns PET and SUVR with joint losses; Stage 2 connects PET embeddings to clinical language; probes assess downstream clinical relevance.

Unlike prior “full fine-tuning” and other parameter-efficient methods (e.g., adapters, LoRA), this last-block adaptation is empirically validated as effective only for the ResNet family, not for ViT or U-Net-based backbones, likely due to ResNet’s architectural bottleneck properties.

Cross-Modal Alignment with Clinical Language

A secondary alignment layer connects metabolic embeddings with clinical language using small projection heads and frozen BioClinicalBERT. Text summaries are deterministically generated from SUVR profiles, ensuring that all semantic content is directly grounded in quantitative regional metabolism instead of free-form, potentially hallucinatory, clinical narratives.

Downstream Evaluation

The evaluation paradigm includes:

• Regression and retrieval metrics on SUVR profiles,
• Cross-modal (PET-to-SUVR, PET-to-text) retrieval,
• Linear probe-based diagnostic classification and cognitive regression, with strict encoder freezing, to reflect representational quality rather than task-overfitting.

  Figure 3: PET–SUVR cosine similarity: Frozen MedicalNet (left) is unstructured, while ReMAP-PET (right) yields strong cross-modal alignment (R@1 = 77.8%).

Empirical Results

Metabolic Structure Learning and Retrieval

ReMAP-PET achieves superior SUVR prediction (MAE = 0.070) and a large improvement in PET→SUVR Recall@1 (77.8%) relative to strong baselines, including BrainFM and fully frozen MedicalNet, where the latter achieves only 2.6%. Frozen or naively tuned encoders, even those with recent pretraining (ViT and U-Net based), fail to capture this structured metabolic alignment.

Figure 4: Predicted versus ground-truth SUVR profiles for a representative subject, demonstrating close regional alignment and high-fidelity metabolic reconstruction.

Clinical Interpretability and Downstream Probing

PET representations are highly linearly predictive of diagnostic status (AD vs. CN AUROC = 0.946), and achieved performance is at or modestly surpasses linear probes on the ground-truth SUVR. Cognitive regression metrics are competitive with state-of-the-art encoders but indicate diminishing returns over classical SUVR-based stratification.

The method’s structured metabolic embedding enables region-level PET–text retrieval and report generation with high factual region overlap (low metabolism overlap = 0.719, high metabolism overlap = 0.578), and the PET encoder’s metabolic predictions can be verbalized without introducing ungrounded or hallucinated findings.

Figure 5: Cortical surface plots for predicted (top) and ground-truth (bottom) SUVR values—model captures both high and low metabolic regions robustly.

Ablation and Mechanism Analysis

Loss component ablation reveals a critical dependence on contrastive alignment; pure regression achieves lowest MAE (0.055) but fails completely at cross-modal retrieval. Incorporation of the contrastive loss term results in substantial increases in retrieval performance with minimal compromise in regression accuracy.

Architectural Sensitivity of Partial Tuning

Applying the same “last-block-only” adaptation protocol to ViT-based and U-Net-based encoders yields negligible or negative gains, highlighting an architecture-specific effectiveness likely tied to ResNet’s semantically discrete bottleneck.

Theoretical and Practical Implications

ReMAP-PET advances PET representation learning in several crucial directions:

• Metabolic semantics encoding: Moves model supervision from visually generic features to physiologically grounded metabolic patterns.
• Transparent clinical interpretability: Emergent structured embeddings are directly mappable to clinical endpoints and textual summaries, suggesting better support for real-world deployment and regulatory interpretability.
• Parameter-efficient adaptation: Empirically supports partial-tuning for ResNets, raising caveats for architectural generalization.
• Medical vision–language alignment: Demonstrates that deterministic, measurement-constrained language generation can avoid pitfalls of hallucination in clinical text generation, an ongoing safety concern in medical AI systems.

Practically, this approach positions PET-based models for safer integration into diagnostic, reporting, or search pipelines where explainability is critical. The methodology also serves as a template for structured supervision in other functional imaging modalities.

Limitations and Future Directions

The authors recognize the following limitations:

• Training and validation are currently restricted to a single ADNI-derived FDG-PET cohort, with the necessity for multi-center, tracer-diverse testing.
• Textual supervision is template-based, limiting linguistic diversity and not capturing the variability of human reporting.
• The mechanism underlying the specificity of partial-tuning effectiveness to ResNet architectures remains empirical; a theoretical formalization is needed.
• Evaluation is based solely on linear probing; the effects of downstream fine-tuning or broader task integration remain unexplored.

Future work could target external validation, explore generalization to non-FDG tracers, task-specific fine-tuning, and theoretical analysis of adaptation bottlenecks.

Figure 6: Per-subject scatter of predicted vs. actual SUVR across 120 regions (left), and high overlap in top-5 low/high metabolic regions (right).

Conclusion

ReMAP-PET exemplifies a shift from visually generic to region-guided metabolic representation learning in brain PET, with clear benefits for interpretability, retrieval, and clinical probing. The approach demonstrates that constraining encoders using structured, physiologically meaningful supervision produces richer, more useful embeddings than conventional volumetric pretraining. The observed architecture-specific viability of partial tuning holds implications for the design and adaptation of future foundation models across neuroimaging modalities. This structured embedding paradigm offers a promising route for safer and more interpretable integration of PET-based AI tools in both research and clinical workflows.

Figure 7: Contrastive ablation: retrieval depends critically on contrastive alignment; pure regression yields low MAE but no cross-modal structure.