fMRI-Based Image Reconstruction

• fMRI-based image reconstruction is a method that decodes visual stimuli from brain activity by inverting cortical encoding processes using deep neural models.
• It utilizes a modular pipeline comprising fMRI signal encoding, feature mapping, and generative decoding to achieve both low-level and high-level visual fidelity.
• Recent advances address semantic misalignment and cross-subject variability through multi-modal guidance and explicit semantic reasoning, enhancing reconstruction accuracy.

Functional Magnetic Resonance Imaging (fMRI)-Based Image Reconstruction is a methodological framework in neural decoding that aims to recover or synthesize human-perceived visual stimuli directly from blood-oxygen-level-dependent (BOLD) activity measured by fMRI. This domain bridges neuroscience, machine learning, and computer vision, providing insights into the hierarchical and distributed representations of visual information in the human cortex and enabling emerging applications in brain–computer interfaces and clinical assessment.

1. Problem Definition, Historical Context, and Fundamental Objectives

fMRI-based image reconstruction seeks to invert the encoding process of the visual cortex, predicting a plausible visual image (natural scenes, faces, symbols, etc.) from measured fMRI responses. Early studies could decode simple geometric shapes or object categories, relying on linear regression and hand-crafted features. Contemporary research utilizes high-dimensional deep neural generative models and multimodal embedding spaces to improve both the spatial fidelity and semantic plausibility of reconstructed images (Guo et al., 24 Feb 2025).

Key objectives include:

• Capturing both low-level (edges, layout) and high-level (semantic content, object identity) visual aspects (Ozcelik et al., 2022, Ozcelik et al., 2023, Lu et al., 2023, Yang et al., 25 Jan 2026).
• Achieving generalization across unseen images and subjects, with robustness to inter-subject and inter-session variability (Quan et al., 2024, Beliy et al., 29 Oct 2025, Zangos et al., 3 May 2025).
• Interpreting the neuroscientific validity of learned representations and their mapping to cortical structures (Ye et al., 18 May 2025, Yang et al., 25 Jan 2026, Beliy et al., 29 Oct 2025).

2. Technical Methodologies: Pipeline and Model Architecture

The standard fMRI-based image reconstruction pipeline is modularized into three principal stages (Guo et al., 24 Feb 2025):

(a) fMRI Signal Encoding

• Spatial and temporal preprocessing: Slice-time correction, motion correction, session-level or trial-level z-scoring, and ROI-based voxel extraction are standard (Quan et al., 2024, Ozcelik et al., 2023, Lin et al., 2022).
• Dimensionality reduction and selection: Voxel selection using encoding performance (R²) (Qiao et al., 2018), spatial-spectral filtering (FreqSelect) (Ye et al., 18 May 2025), and shared embedding adapters for inter-subject alignment (Zangos et al., 3 May 2025) are prominent.

(b) Feature Mapping

• Linear or nonlinear regression links reduced-dimensional fMRI vectors to image or multimodal (vision-language) latent spaces. Regularized linear (ridge) regression dominates classical pipelines (Mozafari et al., 2020, Ozcelik et al., 2023, Ozcelik et al., 2022), while deeper mappings (with residual blocks or transformers) are used for complex, multi-subject or multi-modal setups (Lin et al., 2022, Quan et al., 2024, Beliy et al., 29 Oct 2025).
• Recent work demonstrates that mapping fMRI to the latent space of LLMs (LLMs, e.g., T5, LLaMA), rather than purely vision-based or joint spaces, aligns more closely with neural activity distributions and supports better compositional structure in generation (Huang et al., 17 Oct 2025, Yang et al., 25 Jan 2026).

(c) Generative Decoding

• Early approaches used optimization over pixels to invert predicted CNN features (Zhang et al., 2018).
• Linear-to-latent GAN pipelines (BigBiGAN, DCNN-GAN) utilize direct latent reconstruction followed by adversarial generation (Mozafari et al., 2020, Lin et al., 2019, Ozcelik et al., 2022).
• Two-stage pipelines introduce a coarse layout inference (e.g., via VDVAE or VAE-GAN), followed by semantic refinement using conditional latent diffusion models (LDMs). This modular design enables the integration of low-level perceptual and high-level semantic priors, improving naturalistic and compositional synthesis (Ozcelik et al., 2023, Lu et al., 2023, Ye et al., 18 May 2025).
• Multi-modal guidance: Advanced frameworks incorporate text, CLIP, and layout features as conditioning signals into diffusion backbones (e.g., Brain-Streams, SynMind, MindDiffuser, PRISM) (Joo et al., 2024, Lu et al., 2023, Huang et al., 17 Oct 2025, Yang et al., 25 Jan 2026), enabling disentangled control of semantic and structural fidelity.

3. Novel Approaches: Semantics, Multi-Modality, Transfer, and Cross-Subject Generalization

Recent advances target the following:

Explicit Semantic Reasoning and Hallucination Suppression

• SynMind (Yang et al., 25 Jan 2026) and PRISM (Huang et al., 17 Oct 2025) resolve semantic hallucinations—misalignment between reconstructed and target scene objects—by first parsing fMRI into rich, sentence-level, multi-granularity textual representations (via grounded Large Multimodal LLMs), which guide the diffusion model instead of relying on entangled visual embeddings. This "semantics-first" pipeline demonstrably improves high-level content faithfulness, with task-driven neurovisualization revealing broader, more meaningful cortical engagement.

Cross-Subject and Multi-Subject Pipelines

• Psychometry (Quan et al., 2024) deploys an Omnifit Mixture-of-Experts Transformer that enables aggregation of inter-subject commonality with subject-specific specialization, and enhances inference by retrieval-based memory augmentation.
• Brain-IT (Beliy et al., 29 Oct 2025) leverages brain-wide voxel clustering to enable functional inter-subject mappings, supporting "few-shot" cross-subject adaptation (1 hour of data per new subject achieves near full-data SOTA).
• Adapter alignment (AAMax) (Zangos et al., 3 May 2025) and other shared-space alignment techniques dramatically improve the cost-efficiency and scalability of fMRI-to-image pipelines, supporting real-world generalization even in low-data regimes.

Multi-Modal Guidance and Architectural Modularization

• Pipelines such as MindDiffuser and Brain-Streams (Lu et al., 2023, Joo et al., 2024) decompose the guidance signal into three streams—text (high-level semantics), visual features (mid-level semantics, e.g., CLIP embedding), and layout (low-level, e.g., VAE/diffusion latent)—mapping fMRI from distinct ROIs to each.
• These approaches exploit known neuroscientific dissociations between ventral (semantic) and early visual (perceptual/layout) cortex, operationalizing the "two-streams" hypothesis in model design.

4. Evaluation Metrics, Quantitative Results, and Comparative Benchmarks

Performance assessment combines low-level, structural, and high-level semantic alignment measures (Ozcelik et al., 2022, Ozcelik et al., 2023, Yang et al., 25 Jan 2026, Huang et al., 17 Oct 2025):

• Low-level: pixel-wise correlation (PixCorr), SSIM, and mean squared error (MSE).
• Mid/high-level: 2-way or n-way identification in deep network feature spaces (AlexNet, Inception, CLIP), LPIPS, EfficientNet-B, SwAV feature distances, FID.
• Semantic/human evaluation: forced-choice preference, object/attribute detection.

Selected recent results:

• MindDiffuser improves SSIM and CLIP similarity by 18–19% and 19% over prior SOTA (Lu et al., 2023).
• SynMind achieves an increase of ≈2% in Inception and CLIP scores and is preferred in 60.4% of human trials over MindEye2, with a reduction in semantic hallucinations (Yang et al., 25 Jan 2026).
• PRISM reports up to 8% reduction in perceptual loss (LPIPS) and outperforms CLIP-image or pure vision-latent spaces across all metrics (Huang et al., 17 Oct 2025).
• Cross-subject omnifit models such as Brain-IT and Psychometry now reach or surpass subject-specific pipelines with dramatically less training data and show high robustness to subject variability (Beliy et al., 29 Oct 2025, Quan et al., 2024).

5. Challenges, Limitations, and Neurocomputational Insights

Despite consistent improvements, several challenges and research opportunities remain:

• Data scarcity: fMRI-image paired datasets are small (typically <10K samples/subject), which impedes the training of deep or highly parameterized encoders (Guo et al., 24 Feb 2025, Lin et al., 2022).
• Cross-subject variability: Inter-individual anatomical and functional heterogeneity complicates model generalization, motivating transfer and alignment strategies (Zangos et al., 3 May 2025, Quan et al., 2024).
• Semantic misalignment: A persistent limitation is that reconstructions often appear visually plausible but semantically incorrect; disentangling semantic information and leveraging explicit text-based representations helps mitigate this (Huang et al., 17 Oct 2025, Yang et al., 25 Jan 2026).
• Interpretability: The spatial-frequency-aware FreqSelect module reveals learned frequency weights that recapitulate classic fMRI retinotopic tuning (emphasis on global shape in V1, transient mid-level features in V2/V4), supporting neuroscientific validity (Ye et al., 18 May 2025).
• Transferability: Multi-subject pipelines now achieve high-fidelity reconstructions with as little as 15 minutes to 1 hour of new subject data, a prerequisite for practical BCI applications (Beliy et al., 29 Oct 2025, Zangos et al., 3 May 2025).

6. Future Directions

Anticipated future research avenues, based on current trends and identified limitations:

• End-to-end fine-tuning of diffusion backbones with limited fMRI–image pairs.
• Extension to other modalities, including EEG/MEG, to overcome fMRI’s inherent temporal limitations and enable dynamic visual (video, imagined) decoding (Joo et al., 2024, Beliy et al., 29 Oct 2025).
• Incorporation of topographic priors, such as retinotopic maps and cortical magnification models, to enhance spatial alignment (Huang et al., 17 Oct 2025).
• Interpretable and explainable AI, using region-attribution and attention analysis to link decoded image content to specific brain areas, increasing neuroscientific utility and clinical confidence (Ye et al., 18 May 2025).
• Dynamic scene graphs and graph-neural-network intermediates for more natural handling of object relationships and spatial layouts in complex visual scenes (Huang et al., 17 Oct 2025).
• Efficient adaptation to low-resource regimes and cross-dataset transfer, enabled by lightweight, aligned adapters and retrieval-based strategies (Quan et al., 2024, Zangos et al., 3 May 2025).
• Real-time and closed-loop applications for clinical and communication assistive devices (Lu et al., 2023, Lin et al., 2022).

7. Representative Methods and Results Table

Below is a comparative table for selected recent approaches, reporting key reconstruction performance metrics on the Natural Scenes Dataset where provided. Numbers reflect the best available mean or top-performant values in the cited works.

Method	SSIM	CLIP ID (%)	Inception ID (%)	Semantic Hallucination Control
Brain-Diffuser	0.356	91.5	87.2	None
MindDiffuser	0.354	0.765*	–	Partial (structural, semantic)
Psychometry	0.340	96.8	95.8	Multi-subject Ecphory
Brain-IT	0.486	96.4	97.3	Dual-branch/cluster alignment
SynMind	0.407	96.9	97.8	Multi-granularity semantics
PRISM	0.464	94.7	97.3	Structured text bridge

*value denotes CLIP similarity, not identification rate.

• "A Survey of fMRI to Image Reconstruction" (Guo et al., 24 Feb 2025)
• "SynMind: Reducing Semantic Hallucination in fMRI-Based Image Reconstruction" (Yang et al., 25 Jan 2026)
• "Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI" (Huang et al., 17 Oct 2025)
• "Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformer" (Beliy et al., 29 Oct 2025)
• "Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity" (Quan et al., 2024)
• "MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion" (Lu et al., 2023)
• "FreqSelect: Frequency-Aware fMRI-to-Image Reconstruction" (Ye et al., 18 May 2025)
• "Efficient Multi Subject Visual Reconstruction from fMRI Using Aligned Representations" (Zangos et al., 3 May 2025)