A Multimodal Approach to Estimating Vigilance Using EEG and Forehead EOG

Abstract: Objective. Covert aspects of ongoing user mental states provide key context information for user-aware human computer interactions. In this paper, we focus on the problem of estimating the vigilance of users using EEG and EOG signals. Approach. To improve the feasibility and wearability of vigilance estimation devices for real-world applications, we adopt a novel electrode placement for forehead EOG and extract various eye movement features, which contain the principal information of traditional EOG. We explore the effects of EEG from different brain areas and combine EEG and forehead EOG to leverage their complementary characteristics for vigilance estimation. Considering that the vigilance of users is a dynamic changing process because the intrinsic mental states of users involve temporal evolution, we introduce continuous conditional neural field and continuous conditional random field models to capture dynamic temporal dependency. Main results. We propose a multimodal approach to estimating vigilance by combining EEG and forehead EOG and incorporating the temporal dependency of vigilance into model training. The experimental results demonstrate that modality fusion can improve the performance compared with a single modality, EOG and EEG contain complementary information for vigilance estimation, and the temporal dependency-based models can enhance the performance of vigilance estimation. From the experimental results, we observe that theta and alpha frequency activities are increased, while gamma frequency activities are decreased in drowsy states in contrast to awake states. Significance. The forehead setup allows for the simultaneous collection of EEG and EOG and achieves comparative performance using only four shared electrodes in comparison with the temporal and posterior sites.

• The paper introduces a multimodal framework that fuses EEG and forehead EOG signals, achieving correlation coefficients of up to 0.85 in vigilance estimation.
• It demonstrates that forehead EOG provides superior single-modality performance and enhances EEG fusion results by reducing RMSE to 0.09.
• The study employs CCNF and CCRF models on simulated driving data, underscoring practical applications for wearable real-time vigilance monitoring.

Overview of Multimodal Vigilance Estimation using EEG and Forehead EOG

The paper presents a detailed exploration of estimating vigilance states through a combination of electroencephalogram (EEG) and electrooculogram (EOG) signals. This work is predicated on the importance of continuous monitoring of user mental states in contexts such as driving, where lapses in vigilance can result in serious safety risks. With a focus on real-world applicability, the authors employ a unique electrode placement that supports data collection through forehead EOG, thus ensuring feasibility and user comfort while maintaining robust signal acquisition.

In traditional applications, EEG is often utilized to detect transitions between wakefulness and sleep, serving as a critical neurophysiological marker. EOG, particularly when recorded from traditional electrode placements around the eyes, offers high signal-to-noise ratios for eye movement detection but may prove disruptive in practical applications. The novel methodology presented here utilizes forehead EOG, effectively capturing eye movement data with minimal discomfort.

Experimental Insights and Methodological Advances

The authors introduce a multimodal framework combining time-variant aspects of vigilance using continuous conditional neural fields (CCNF) and continuous conditional random fields (CCRF). This approach capitalizes on the complementary nature of EEG and EOG data, capturing temporal dependencies crucial for dynamic vigilance estimation. The data collected through a simulated driving system shared significant correlations between traditional VEO/HEO and new forehead EOG signal extraction methods. The chosen methodology demonstrates a high correspondence with traditional EOG signals through ICA-based separation approaches, achieving mean correlation coefficients of VEO $0.80$ and HEO $0.75$.

The research highlights the robustness of DE features extracted in the 2 Hz frequency resolution from various EEG data segments and the recognition power of those for vigilance estimation. The posterior EEG sites continue to show significant effectiveness for vigilance detection aligned with known physiological patterns such as theta and alpha frequency activity shifts.

Results and Implications

Throughout the experiments, an impressive outcome was secured from forehead EOG signals providing better single-modality performance than posterior EEG and when combined with other EEG sites through modality fusion, the results were greatly enhanced. The multimodal approach further increased performance metrics significantly with RBF SVR, CCRF, and CCNF models resulting in a mean correlation coefficient of up to 0.85 and RMSE reduced to 0.09 relative to single-modality implementations.

Also notably, the characteristic frequency pattern alterations for different vigilance levels align well with established literature findings supporting the effectiveness of the approach. The implications for practical applications become apparent in the potential for developing wearable, integrated BCI systems, accommodating efficient real-time vigilance monitoring crucial for high-vigilance demands scenarios like driving.

Future Directions

The exploration of generalizability regarding individual differences across subjects and sessions remains a pivotal challenge requiring additional focus. Future studies could explore applying longitudinal data and transfer learning methodologies to standardize and enhance model adaptations across diverse user bases.

Moreover, real-world application trials and usability studies on wearable devices integrating EEG and forehead EOG technology could substantiate the approach’s practical viability for industries reliant on sustained attentiveness. Enhancing adaptive feedback mechanisms in a closed-loop setup would also further contribute towards directly actionable and user-centered applications, potentially improving safety across various domains including transportation and workplace environments.