Gaze-Informed Proactive AI Assistance for Children's Picture Exploration

Abstract: Proactive assistance with LLMs has received growing attention in the human computer interaction (HCI) community. However, most past work on proactive LLMs' assistance has focused on adult users and task-oriented settings, leaving open how such systems could support children, whose interests and needs are often expressed through gaze and other nonverbal behaviors rather than explicit requests. In this study, we focus on two key challenges of proactive assistance in children's picture exploration: when to provide assistance and what assistance to provide based on children's nonverbal behaviors. To address these challenges, we introduce Ollie, a gaze-informed proactive AI assistant that offers short narrative descriptions based on where a child is looking. Ollie uses children's gaze to estimate their attention, identify their current visual focus, and select a related picture region for the LLM to verbally describe. In a within-subject experiment, we compared gaze-informed assistance with random assistance. Results show that gaze-informed assistance kept children's attention on their current focus for a longer period of time, and guided them more effectively to related picture regions. Children, parents, and a participating kindergarten teacher viewed Ollie positively and consider that it better matched children's interests when compared with the random assistance. This work shows the feasibility of using gaze as an implicit input for proactive AI assistance for children and provides design implications for future child-centered AI systems.

• The paper demonstrates that real-time gaze data can trigger contextually relevant AI narration, aligning interventions with children’s implicit visual attention.
• It employs a two-state HMM and relevance-based AOI selection to achieve longer fixation durations (6.48s vs 1.67s) and higher follow-up fixation rates (54.06% vs 35.48%).
• The study underscores the potential of integrating gaze signals with LLM-driven narratives to support adaptive, child-centered educational technology.

Gaze-Informed Proactive AI Assistance for Children’s Picture Exploration

Introduction and Motivation

“Gaze-Informed Proactive AI Assistance for Children's Picture Exploration” (2607.00445) investigates how gaze can be leveraged as an implicit cue for fine-grained, child-centered proactive assistance during open-ended visual activities. While contemporary LLM-based conversational agents typically rely on explicit verbal or written prompts for interaction and are primarily targeted at adults in task-based settings, this work identifies a significant gap: supporting young children, whose interests and informational needs are often encoded in nonverbal behavior rather than direct requests. This is especially critical in activities such as picturebook exploration, where internal states like curiosity and engagement manifest in gaze rather than speech.

The core hypothesis is that real-time eye-tracking can inform not just the timing ("when") but also the content ("what") of proactive AI interventions for children. The proposed system, Ollie, analyzes gaze to provide LLM-generated narration that is contextually grounded in the child’s actual focus of attention and then guides the child to related, but previously unexplored, visual regions. This approach aims to address limitations of both random, predefined, and manually-triggered assistance by (i) capturing interest states non-disruptively, and (ii) enabling dynamic, interest-aligned support for open-ended, non-instructional visual tasks.

System Architecture and Gaze-Informed Pipeline

The Ollie system introduces a multi-stage gaze-informed pipeline for proactive AI assistance. The architecture combines continuous eye-tracking with relevance modeling and adaptive LLM-based narrative generation.

The first processing stage infers the child’s attentional state via a two-state HMM, distinguishing between "engaged" and "exploring" gaze patterns on predefined AOIs. The model utilizes a normalized fixation metric (FR-D) as the observation variable and updates parameters online for responsive state estimation. Upon identifying engagement with an AOI, the system selects a contextually relevant secondary AOI—unassisted thus far—by jointly minimizing a relevance distance that integrates semantic, spatial, and temporal attributes, with domain-specific weighting.

Narrative support is then generated by prompting an LLM with the full scene context, objects lists for both AOIs, and chaining hooks from prior narration rounds to ensure narrative continuity and stimulus variation. Spoken delivery is realized via TTS.

Figure 1: The gaze-informed assistance pipeline includes attention detection, relevance-based AOI selection, and LLM-based contextual narration.

This approach provides granular, context-aware interventions grounded in the real-time behavioral data of the child, directly addressing the dual challenge of when and what assistance to provide.

Experimental Design

A controlled within-subject experiment compared the gaze-informed condition with a random-assistance baseline, using Wimmelbook-style complex images segmented into AOIs. Primary AOIs were visually highlighted for three seconds on each narration in both conditions; secondary AOIs were only introduced by narration, avoiding confounds in attention-cueing analysis.

Children (N=22, age 5–8) interacted with the system under each condition. Gaze data were recorded and fixations quantified using PyGaze-based detection. AOI engagement, attention transitions, and verbal uptake were measured.

Figure 2: Example image applied for picture exploration in the experiment.

Results

Visual Attention Dynamics

In terms of direct gaze metrics, the follow-up fixation rate on the primary AOI exceeded 90% in both conditions (no significant difference), but fixation duration on the primary AOI was dramatically higher for gaze-based assistance (6.48s vs 1.67s, $p < .001$), indicating that gaze-informed narration fixes attention more robustly and for longer periods.

Figure 3: Follow-up fixation rates on the primary AOI demonstrated similar effectiveness in initial orienting for both conditions.

On the secondary AOI—introduced only via narration—the follow-up fixation rate was 54.06% (SD=20.92) in the gaze-informed condition versus 35.48% (SD=28.34) in the random baseline ($p < .05$), illustrating superior cross-region guidance by relevance-based selection.

Figure 4: Gaze-informed assistance elicits substantially higher follow-up fixation rates on the introduced secondary AOI than random assistance.

The relevance distance metric (semantic, spatial, and temporal) between AOIs was also significantly improved in the gaze-informed condition (0.43 vs 0.52, $p < .001$), confirming that gaze-based guidance targets AOIs that are more proximal and contextually meaningful. Despite these effects on attention allocation, fixation duration and latency to first fixation on the secondary AOI did not differ significantly by condition.

Figure 5: Comparable fixation durations on the secondary AOI across conditions, indicating a general brevity in attention to verbally-cued, non-visually-highlighted regions.

Temporal analysis of dominant AOI transitions revealed that in the gaze-based condition, attention shifts from the primary to the secondary AOI occurred more smoothly and overlapped (i.e., the secondary AOI captured attention even during the transition phase), while random assistance led to a more abrupt switch, suggesting qualitatively more coherent attentional guidance for gaze-based support.

Figure 6: Attention switch dynamics in the random assistance condition display sharper transitions from primary to secondary AOI.

Uptake and Subjective Impact

Analysis of post-exploration verbal descriptions demonstrated no significant difference by condition either in response-to-assistance overlap rate or description quality, though both exceeded chance and reflected broad individual variance.

Figure 7: Response-to-assistance rates capture the proportion of narration content reflected in children's verbal descriptions, with no significant difference across conditions.

Subjective feedback was notably strong: 15/22 children and 15/16 parents explicitly preferred the gaze-informed system, citing its greater alignment with the child’s interests and perceived agency. However, parents expressed concerns about screen time and limited conversational flexibility, and suggested use as an augmentation rather than a substitute for adult interaction. The kindergarten teacher considered gaze-based assistance most useful for free exploration or for language-diverse learners but emphasized the need for increased interactivity.

Theoretical and Practical Implications

This work demonstrates that gaze can function as a robust, low-friction signal for real-time estimation of children's engagement and interests in open-ended visual tasks, directly informing both the timing and content of proactive AI intervention. This finding supports integrating behavioral user modeling with LLM-based interaction systems in child-centered applications, mitigating the bottleneck of explicit language-based prompt formulation.

Practically, the pipeline advances situated, adaptive AI support for early learners, with demonstrated effects on both attention holding and context-appropriate redirection. The methodology and results generalize to other visually-intensive, open-ended domains for both children (e.g., drawing, museum exploration, AR guidance) and adults (e.g., medical imaging, visual analytics) where latent interest states are not readily verbalized.

Theoretically, these results extend the scope of mixed-initiative HCI and agency-supportive design by illustrating how implicit behavioral input can scaffold meaningful, child-led interaction with AI, while also providing systematic approaches for balancing situational proactivity with user control. Notably, increased attention holding did not translate to richer or more overlapping verbal output, highlighting complex relationships between gaze, comprehension, and verbal expression that merit further study.

Future Directions

Key limitations include the modest sample size, manual AOI definitions, and the restricted conversational interactivity of the initial Ollie system. Future directions should leverage gaze-driven online segmentation (e.g., with SAM (Kirillov et al., 2023) or similar), adaptive multi-turn dialogue, and cross-modal feedback loops (gaze as both trigger and confirmation of comprehension or engagement). Additionally, longitudinal deployment studies can clarify how child agency and motivation evolve with such interactive agents, particularly in collaborative or self-regulated learning contexts.

Conclusion

Gaze-informed proactive assistance represents a substantive advancement for child-centered AI, narrowing the gap between passive multimedia features and active, meaningful interaction by using behavioral data to align support with dynamic interests. By directly integrating gaze-driven models with LLMs, the work demonstrates scalable, context-sensitive AI narrators that sustain and channel attention more effectively than content-agnostic baselines, with clear preference among both children and parents for such context-sensitive, minimally intrusive support. Continued development along these lines is likely to inform next-generation educational technology and mixed-initiative AI systems for both child and general populations.