EEG-to-Image Decoding

• EEG-to-image decoding is a technique that reconstructs visual stimuli from EEG signals using deep neural architectures and generative models.
• It overcomes challenges like low spatial resolution and high noise by integrating advanced preprocessing, CNNs, transformers, and cross-modal alignment.
• The approach has broad applications in brain-computer interfaces and neuroscience, yet faces hurdles in cross-subject variability and detailed image fidelity.

EEG-to-image decoding refers to the process of reconstructing visual stimuli from electroencephalogram (EEG) signals acquired while human subjects observe images. This task leverages the high temporal resolution and non-invasive nature of EEG, but must compensate for its low spatial resolution, high noise levels, and cross-subject variability. Recent advances have combined state-of-the-art deep learning encoders, large pre-trained generative models (particularly diffusion models), and cross-modal alignment techniques to enable the synthesis of semantically meaningful and structurally coherent images directly from raw EEG data. This article reviews foundational principles, key methods, evaluation metrics, datasets, and open challenges in EEG-to-image decoding.

1. Core Principles and Problem Formulation

EEG-to-image decoding aims to reconstruct (or retrieve) the perceptual content of a viewed image $x_I$ from the recorded EEG response $x_E$, where $x_E \in \mathbb{R}^{C_E \times T}$ for $C_E$ electrodes and $T$ time samples. The decoding pipeline is typically framed as finding a mapping $f: x_E \mapsto \hat{x}_I$ such that $\hat{x}_I$ is perceptually and semantically aligned with the true stimulus $x_I$.

Major advances have formalized this as a multimodal representation learning problem: EEG and image data are embedded into a shared latent space (frequently CLIP or diffusion prior space) using deep neural architectures specialized for each modality. This enables both zero-shot retrieval—matching EEG representations to large image banks—and generative reconstruction via pretrained models conditioned on EEG, with minimal or no explicit supervision (Li et al., 2024, Zhang et al., 10 Nov 2025, Zhang et al., 2024).

2. EEG Signal Processing and Feature Extraction

Robust preprocessing is essential given the low SNR and nonstationarity of EEG. Canonical steps include:

• Bandpass and notch filtering (e.g., 0.1–100 Hz and notch at 50/60 Hz) (Song et al., 2023, Choi et al., 2024, Xu et al., 2024).
• Epoching trials to stimulus onset (typically 0–1000 ms or 500 ms windows).
• Artifact correction using ICA or automatic rejection (Choi et al., 2024, Xu et al., 2024).
• Spatial referencing (common-average referencing) and normalization (z-score per channel).

Feature extraction spans:

• Temporal and spatial CNNs (e.g., ShallowConvNet, STConv, TSConv) (Song et al., 2023, Chen et al., 2024).
• Transformer encoders (spatiotemporal, channel-wise) (Bai et al., 2023, Zhang et al., 30 May 2025, Mishra et al., 2024, Abramov et al., 30 Oct 2025, Rezvani et al., 9 Jul 2025).
• Wavelet or time-frequency analysis (e.g., DWT, STFT) for multi-scale EEG features (Zhang et al., 30 May 2025, Ferrante et al., 2023).
• Advanced attention modules, including graph attention for spatial dependencies and channelwise gating (Chen et al., 2024, Zhang et al., 30 May 2025).

EEG encoders often output high-dimensional vectors (e.g., $d=512$–$1024$) to facilitate alignment with CLIP or diffusion model embedding spaces (Bai et al., 2023, Choi et al., 2024, Zhang et al., 2024).

3. Multimodal Representation Alignment and Cross-Modal Embedding

State-of-the-art pipelines align EEG and image stimuli into a shared latent space by:

• Contrastive (InfoNCE) objectives:
  • Symmetric losses align positive (matched) EEG–image pairs, using cosine similarity and temperature scaling (Song et al., 2023, Li et al., 2024, Zhang et al., 10 Nov 2025, Zhang et al., 2024, Rezvani et al., 9 Jul 2025).
  • Regularizers such as similarity-keeping enforce preservation of intra-modality relational geometry (Chen et al., 2024).
• CLIP alignment:
  • CLIP embeddings provide robust, multimodal feature spaces, facilitating EEG–image (and EEG–text) alignment and transfer learning (Bai et al., 2023, Choi et al., 2024, Zhang et al., 10 Nov 2025).
  • CLIP-based contrastive losses are often combined with MSE or alignment losses between EEG and CLIP representations (Bai et al., 2023, Choi et al., 2024, Zhang et al., 2024).
• Bidirectional semantic projectors:
  • Co-adaptive modules allow both EEG and image branches to map into a shared semantic space, mitigating static structural mismatches (Zhang et al., 10 Nov 2025).

Augmentation strategies such as cognitive prior augmentation—injecting variability via image and EEG perturbations—improve robustness and generalization (Zhang et al., 10 Nov 2025).

4. Generative Decoding: GANs, VAEs, and Diffusion Models

Generative models for EEG-to-image decoding fall into several paradigms:

Model Type	Pipeline Example	Alignment Method/Conditioning
GAN	cGANs with EEG code as generator input	Adversarial + perceptual loss (Mishra et al., 2024, Sabharwal et al., 2024)
VAE	Latent variational bottleneck, often hybridized	ELBO + L1 or adversarial (Sabharwal et al., 2024)
Diffusion	EEG embedding as cross-attention or input prior	CLIP alignment, IP-Adapters, LoRA (Bai et al., 2023, Choi et al., 2024, Zhang et al., 2024, Chen, 2024, Li et al., 2024, Abramov et al., 30 Oct 2025, Zhang et al., 30 May 2025)

Recent work converges on a two-stage diffusion framework (Bai et al., 2023, Choi et al., 2024, Chen, 2024, Li et al., 2024, Zhang et al., 2024):

1. Stage 1: EEG embedding (aligned to CLIP or diffusion priors) is refined, often via a diffusion prior trained with a denoising MSE loss.
2. Stage 2: The prior output conditions a pre-trained or frozen text-to-image diffusion model via cross-attention modules (IP-Adapters, adapters into U-Net, or LoRA blocks); optionally, additional conditioning (semantic prompts, captions, saliency maps) is integrated for spatial or semantic control (Abramov et al., 30 Oct 2025, Rezvani et al., 9 Jul 2025).

Some paradigms distinctly separate style and content conditioning, feeding parallel EEG-encoded features to different branches of the generator (Choi et al., 2024), or blend class and caption embeddings (Mehmood et al., 15 Jul 2025).

Spatial attention priors from saliency maps (e.g., ControlNet-based) have been shown to resolve structural EEG ambiguities and improve spatial fidelity (Abramov et al., 30 Oct 2025). Text-based mediation via LLM-generated semantic prompts further enhances interpretability and cognitive alignment (Rezvani et al., 9 Jul 2025).

5. Quantitative Evaluation Metrics and Benchmarks

EEG-to-image decoding is evaluated on both classification/retrieval and generative fidelity:

• Classification/retrieval: Top-1/Top-5 accuracy in $N$-way zero-shot image matching ($N$ up to 200) (Song et al., 2023, Chen, 2024, Zhang et al., 10 Nov 2025), semantic-based scores (e.g., WordNet similarity, CAT Score) (Zhang et al., 30 May 2025, Chen, 2024).
• Generative fidelity:
  • SSIM (Structural Similarity Index), Pixel Correlation, Inception Score (IS), Fréchet Inception Distance (FID) (Bai et al., 2023, Choi et al., 2024, Sabharwal et al., 2024, Zhang et al., 2024, Rezvani et al., 9 Jul 2025).
  • CLIP similarity, SwAV (distance in self-supervised vision embedding), and semantic alignment metrics (Choi et al., 2024, Li et al., 2024, Abramov et al., 30 Oct 2025).
  • Saliency alignment: Correlation, KL divergence, and similarity between reconstructed saliency maps and ground-truth attention maps (Abramov et al., 30 Oct 2025).
• CAT Score: Measures overlap of predicted and human-annotated semantic tags in generated images (Chen, 2024).
• 3D Point Cloud Reconstruction: N-way top-K matching via object classifier in 3D shape–color space (Guo et al., 2024).

Benchmark datasets underpinning these evaluations include Brain2Image/EEG-ImageNet (Bai et al., 2023, Choi et al., 2024), THINGS-EEG (Chen, 2024, Zhang et al., 2024, Li et al., 2024), Alljoined1 (Xu et al., 2024), and EEG-3D (Guo et al., 2024).

6. Major Findings, Biological Plausibility, and Applications

Recent SOTA models achieve:

• Top-1/Top-5 retrieval of up to 38–63%/80–89% in 200-way zero-shot settings (intra-subject), with cross-subject accuracy lagging by over 40 percentage points (Zhang et al., 10 Nov 2025, Zhang et al., 2024).
• SSIM values of 0.227–0.369, FID as low as 69.97 (Choi et al., 2024), CLIP similarity up to 0.90 (Abramov et al., 30 Oct 2025), and CAT Scores of ~440/1000 (Chen, 2024).

Biologically plausible analyses confirm that:

• Peak EEG decodability aligns with 150–500 ms post-stimulus, dominated by occipito-parietal activity (Song et al., 2023, Chen et al., 2024, Guo et al., 2024, Xu et al., 2024, Rezvani et al., 9 Jul 2025).
• Low- and high-frequency spectral bands contribute distinctively; spatial attentional modules reliably localize object-, scene-, and abstract-level representation over canonical cortical topographies (Chen et al., 2024, Rezvani et al., 9 Jul 2025).

Practical applications include non-invasive BCIs for hands-free image selection, clinical communication aids, and neuroscientific probing of human visual coding (Zhang et al., 10 Nov 2025, Zhang et al., 2024, Li et al., 2024). Some studies have extended EEG-visual decoding to 3D object reconstruction (Guo et al., 2024).

7. Open Challenges and Future Directions

Despite rapid progress, several challenges persist:

• Low spatial detail and noise: Fine-grained reconstructions (e.g., facial details) remain out of reach; generated images frequently exhibit abstraction or class confusion (Bai et al., 2023, Chen, 2024).
• Cross-subject generalization: Robustness to anatomical and cognitive variability is limited; inter-subject accuracies drop sharply (Zhang et al., 10 Nov 2025, Zhang et al., 2024).
• Standardization: Field-wide benchmarks, common preprocessing pipelines, and public datasets (e.g. Alljoined1) are catalyzing reproducibility but are not yet universally adopted (Sabharwal et al., 2024, Xu et al., 2024).
• Training stability and interpretability: GANs remain prone to mode collapse and VAEs to blurring; diffusion models are more stable but computationally intensive (Sabharwal et al., 2024). Interpretable attention and saliency mechanisms are increasingly deployed, but mapping neurophysiological features onto cognitive processes remains an open area (Rezvani et al., 9 Jul 2025, Abramov et al., 30 Oct 2025).
• Multimodal integration: Fusing EEG with eye-tracking, depth cues, or text context improves performance but increases pipeline complexity (Zhang et al., 2024, Abramov et al., 30 Oct 2025, Mehmood et al., 15 Jul 2025).

Future research avenues include unified EEG-vision-language pretraining, real-time and low-latency architectures, dynamic (video) decoding, and explainable AI for BCI auditing and closed-loop feedback (Sabharwal et al., 2024, Abramov et al., 30 Oct 2025, Guo et al., 2024, Mehmood et al., 15 Jul 2025).

EEG-to-image decoding stands as a frontier in neural decoding and cross-modal machine learning, integrating sophisticated representation learning, robust signal processing, and generative modeling to translate transient brainwave activity into visual content.