Auditory Brain Passage Retrieval: Cross-Sensory EEG Training for Neural Information Retrieval

Abstract: Query formulation from internal information needs remains fundamentally challenging across all Information Retrieval paradigms due to cognitive complexity and physical impairments. Brain Passage Retrieval (BPR) addresses this by directly mapping EEG signals to passage representations without intermediate text translation. However, existing BPR research exclusively uses visual stimuli, leaving critical questions unanswered: Can auditory EEG enable effective retrieval for voice-based interfaces and visually impaired users? Can training on combined EEG datasets from different sensory modalities improve performance despite severe data scarcity? We present the first systematic investigation of auditory EEG for BPR and evaluate cross-sensory training benefits. Using dual encoder architectures with four pooling strategies (CLS, mean, max, multi-vector), we conduct controlled experiments comparing auditory-only, visual-only, and combined training on the Alice (auditory) and Nieuwland (visual) datasets. Results demonstrate that auditory EEG consistently outperforms visual EEG, and cross-sensory training with CLS pooling achieves substantial improvements over individual training: 31% in MRR (0.474), 43% in Hit@1 (0.314), and 28% in Hit@10 (0.858). Critically, combined auditory EEG models surpass BM25 text baselines (MRR: 0.474 vs 0.428), establishing neural queries as competitive with traditional retrieval whilst enabling accessible interfaces. These findings validate auditory neural interfaces for IR tasks and demonstrate that cross-sensory training addresses data scarcity whilst outperforming single-modality approaches Code: https://github.com/NiallMcguire/Audio_BPR

• The paper introduces a novel method where auditory EEG signals serve as effective neural queries, outperforming both visual EEG and text-based baselines.
• It employs a dual encoder framework with contrastive learning to map EEG and text into a shared semantic space, with CLS pooling showing robust performance.
• The study reveals that cross-sensory training significantly mitigates data scarcity and boosts retrieval metrics, paving the way for inclusive brain-machine interfaces.

Auditory Brain Passage Retrieval via Cross-Sensory EEG Training

Introduction and Motivation

The paper "Auditory Brain Passage Retrieval: Cross-Sensory EEG Training for Neural Information Retrieval" (2601.14001) targets a profound challenge in information retrieval: the process by which users externalize internal information needs as explicit queries. Historically, this translation relies on textual or spoken input, which presents cognitive barriers and excludes users with visual or motor impairments. Brain Passage Retrieval (BPR) reframes the interface by directly mapping EEG signals—traditionally from visual reading tasks—to passage representations in a shared semantic space, sidestepping intermediate linguistic decoding. However, auditory EEG stimuli for BPR remain unexplored. This work sets out to systematically evaluate the efficacy of auditory EEG queries for passage retrieval, their competitiveness against visual EEG, and the impact of cross-sensory training in ameliorating severe data scarcity.

Problem Formulation and Architectural Overview

BPR is formulated as a dense retrieval problem over a passage corpus, where EEG signals synchronized to stimulus presentation serve as neural queries. The approach employs a dual encoder: one for EEG (trained from scratch), one for text (BERT-base, frozen). Both modalities are mapped to a common semantic embedding space. Cosine similarity drives query-document scoring and ranking.

Figure 1: EEG signals recorded during visual or auditory stimulus presentation serve as brain queries ($q_e$), which are encoded alongside text passages ($p$). Cosine similarity between encoded representations produces relevance scores for passage ranking and retrieval.

Four sequence-level aggregation strategies are systematically evaluated: MEAN, MAX, CLS token pooling, and MULTI-vector.

Datasets and Query Construction

Given the absence of multimodal EEG datasets with balanced content and paired query-document relationships suited for IR, the paper leverages the Alice dataset (auditory EEG, n=49) and the Nieuwland dataset (visual EEG, n=51), each capturing naturalistic comprehension of narrative texts. To generate paired data suitable for neural retrieval, the Inverse Cloze Task (ICT) is adapted: random query spans comprising 30% of passage words are extracted, and their context is masked (with probability $p_{mask}=0.9$) to force reliance on semantic—not lexical—features for retrieval.

Figure 2: Inverse Cloze Task (ICT) overview: EEG signals are recorded synchronously with text comprehension, and query spans $q_e$ (30% of total passage length) are extracted with corresponding neural responses; positive passages $P^+$ are created by masking with probability $p_{\text{mask}}$.

Crucially, minimal lexical overlap between datasets (Jaccard $\sim$0.18) creates a high-entropy cross-modal training regime, isolating neural semantic mapping from text memorization.

Training Protocols and Contrastive Learning Objective

The EEG encoder is a 1-layer, 4-head transformer; input tensors consist of word-level EEG segments, flattened and projected to a hidden dimension. Contrastive learning (InfoNCE) is used for cross-modal alignment: positives are EEG–passage pairs, negatives are in-batch distractors. Early stopping, gradient clipping, and mixed precision training mitigate overfitting and numerical instability.

Pooling method selection critically governs sequence-level representation. CLS, as a learnable token, enables global abstraction; MEAN and MAX induce statistical aggregation, while MULTI-vector retains fine-grained token correspondence.

Experimental Design

Three questions are addressed:

• RQ1: Do auditory EEG signals provide effective passage retrieval, and how does their performance compare to visual?
• RQ2: Does cross-sensory training (auditory+visual) enhance retrieval efficacy and is this pooling-dependent?
• RQ3: Can dataset amalgamation mitigate EEG data scarcity, and does architecture modulate benefits?

Evaluation metrics include MRR, Hit@k ($k=1,5,10$), and robustness is assessed via increasing passage masking (0–100%), simulating variable information scarcity typical in IR.

Results: Auditory EEG as Superior Retrieval Modality

Under individual training, auditory EEG—regardless of pooling—substantially outperforms visual EEG. For CLS pooling: auditory MRR=0.362, Hit@1=0.220, Hit@10=0.668; visual MRR=0.139, Hit@1=0.074, Hit@10=0.262. Auditory MRR is 160% higher than visual, with comparable boosts in Hit@1 and Hit@10.

Combined training further amplifies auditory performance: with CLS pooling, MRR=0.474, Hit@1=0.314, Hit@10=0.858, surpassing BM25 text baseline (MRR=0.428, Hit@10=0.542) and ColBERT in all metrics. This neural IR setup achieves text-competitive or superior results.

Figure 3: Cross-sensory training effects: individual (blue) vs. combined (orange) training—combined training yields marked improvements in MRR, Hit@1, and Hit@10 for both modalities, particularly auditory.

Performance is robust across masking, with neural approaches maintaining Hit@10 $\sim$0.65 at 75% masking, where text models degrade below 0.4—further evidence for semantic rather than lexical dependence.

Architecture-Dependent Cross-Sensory Adaptation

Cross-sensory training efficacy is pooling-dependent. CLS pooling consistently induces bidirectional gains: auditory MRR up 31% and visual MRR up 84%. MAX and MULTI favor visual gain at the expense of auditory. Thus, optimal aggregation must be stimulus-modality-aligned. Diversity in EEG corpora is, therefore, beneficial, but only when harmonized with the appropriate sequence aggregation mechanism.

Figure 4: Pooling method comparison (CLS, MEAN, MAX, MULTI): CLS achieves the most robust improvements across both modalities and masking ratios; other architectures offer asymmetric benefits.

The successful amalgamation of datasets with divergent stimulus protocols and low lexical overlap establishes neural-semantic generalization rather than shallow memorization.

Practical and Theoretical Implications

The empirical demonstration that auditory-evoked EEG can serve as competitive neural queries, even outperforming conventional text baselines, has major significance for accessible IR. It enables brain-machine interfaces for users with visual/motor disabilities, extending IR utility to conversational or audio-only contexts—podcasts, audiobooks, voice interfaces—where conventional input is infeasible.

Architecture-modality interactions imply that future BMI systems should incorporate adaptive aggregation pipelines responsive to input sensory channel. The demonstrated robustness under extreme masking suggests potential for low-information, highly inclusive neural IR interfaces.

Limitations and Future Directions

Limitations include the use of narrative comprehension (not active search or query formulation), two source texts, and lack of subject-specific adaptation. Extension to EEG captured during explicit information need realization, expansion to diverse languages and search paradigms, and integrating stimulus-adaptive pooling into real-time BMIs constitute natural future directions.

Conclusion

The paper presents a rigorous exploration of auditory EEG for passage retrieval and establishes that neural signals evoked by auditory stimuli can serve as effective, competitive, and even superior passage queries compared to both visual EEG and standard text baselines. Cross-sensory dataset fusion—when modulated by appropriate semantic aggregation—substantially amplifies performance and mitigates training data scarcity. These findings bear direct relevance for the design of inclusive brain-machine IR interfaces, with broad applicability in both conventional and accessibility-driven AI systems.