Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors

Abstract: Visual attention is highly fragmented during mobile interactions, but the erratic nature of attention shifts currently limits attentive user interfaces to adapting after the fact, i.e. after shifts have already happened. We instead study attention forecasting -- the challenging task of predicting users' gaze behaviour (overt visual attention) in the near future. We present a novel long-term dataset of everyday mobile phone interactions, continuously recorded from 20 participants engaged in common activities on a university campus over 4.5 hours each (more than 90 hours in total). We propose a proof-of-concept method that uses device-integrated sensors and body-worn cameras to encode rich information on device usage and users' visual scene. We demonstrate that our method can forecast bidirectional attention shifts and predict whether the primary attentional focus is on the handheld mobile device. We study the impact of different feature sets on performance and discuss the significant potential but also remaining challenges of forecasting user attention during mobile interactions.

• The paper introduces a method for forecasting user attention shifts during mobile interaction using a novel dataset combining device-integrated and wearable sensor data.
• Predictive models effectively forecasted attention shifts to and from the mobile device, with IMU data and application usage being key features, and improved primary focus prediction using gaze data.
• This research has significant implications for creating proactive, context-aware mobile interfaces that can adapt to user attention, enhancing user experience and potentially improving safety.

Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors: A Comprehensive Analysis

The paper at hand, titled "Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors," explores the ability to predict users' visual attention in mobile contexts. This research is framed within the broader domain of Human-Computer Interaction (HCI), focusing specifically on attentive user interfaces.

Context and Motivation

The ability to sustain visual attention on specific content is crucial in mobile interactions, yet such interactions are frequently interrupted by shifts of attention between the mobile device and the surrounding environment. Current user interfaces lack the capability to adapt preemptively to these shifts; they react only once the shift has occurred. This study addresses the need to predict these attention shifts, facilitating proactive adaptations that could significantly enhance user experiences.

Methodology and Dataset

To explore this, the researchers collected a novel dataset of over 90 hours of user interactions from 20 participants in a campus setting. This dataset included comprehensive recordings from device-integrated sensors and wearable cameras. The study's primary innovation lies in its method for predicting bidirectional attention shifts, leveraging features extracted from both the mobile device and an egocentric vision perspective.

The research team enumerated and analyzed several features impacting attention focus. These include sensory data from mobile devices like IMUs and application usage logs, and visual scene data from wearable cameras for tasks like object detection and depth estimation.

Predictive Modeling and Results

The study investigates three predictive tasks:

1. Prediction of Attention Shifts to/from the Mobile Device: Predictive models were evaluated for their ability to forecast attention shifts between the device and the environment. Features from the mobile device, particularly IMU data and application usage patterns, contributed most significantly to predicting shifts back to the device.
2. Primary Attentional Focus: This task involved predicting whether attention would primarily focus on the mobile device or elsewhere. Enhanced predictions were possible when integrating gaze data alongside standard sensory inputs.
3. Attention Span Prediction: Although preliminary results indicated that this task remains challenging with the current sensor array, it represents a crucial area for future research.

Implications and Future Directions

The implications of this work are vast. By enabling interfaces to predict when and where users will allocate their attention, mobile devices can preemptively adapt to user needs, minimizing latency and reducing the likelihood of errors in attention-critical environments. This capability holds promise for a variety of applications, from enhancing user engagement to notifying users of potential hazards when their attention drifts to the mobile device — particularly relevant in dangerous situations such as driving.

One notable limitation is the current hardware's cumbersome nature, which could skew natural user behavior. Future research might focus on refining the data acquisition approach to minimize intrusiveness, possibly through advances in mobile and wearable sensor technology.

Conclusion

This examination demonstrates the feasibility of attention forecasting, moving towards a future where mobile devices can intelligently anticipate and react to user attention dynamics. Given the complexities inherent in everyday environments, this research lays foundational work that can inspire and guide subsequent innovations in developing truly attentive and context-aware user interfaces.