Gazeify: Gaze-Based Interaction Framework

• Gazeify is a comprehensive suite of technologies that estimates, interprets, and utilizes human gaze for interactive computing using methods like CNN-based appearance estimation and synthetic datasets.
• It employs multimodal techniques by combining RGB imagery, egocentric wearable sensors, and dynamic clustering algorithms to enable precise object referencing, conversational disambiguation, and AR applications.
• Quantitative benchmarks demonstrate improved gaze selection accuracy, low latency segmentation, and robust 3D gaze synthesis, enhancing interactions in HCI, robotics, and social analytics.

Gazeify describes a collection of technologies and frameworks for the estimation, interpretation, and utilization of human gaze in interactive computing systems, encompassing appearance-based estimation, multimodal referencing on wearable devices, vision-LLMs for intent understanding, and 3D gaze synthesis and redirection. Its principal applications span human–computer interaction (HCI), augmented reality (AR), social signal processing, and object referencing through both vision and conversational interfaces.

1. Core Methodologies in Gaze Estimation and Interpretation

Gazeify encapsulates diverse approaches to gaze estimation:

• Appearance-based gaze estimation utilizes RGB imagery and convolutional neural networks (CNNs) to predict eye pitch and yaw directly from eye-region crops, optionally integrating head pose information (e.g., via 6DRepNet). Synthetic datasets (e.g., MetaHuman-generated faces with ground-truth gaze vectors) are leveraged for improved generalization and sub-2° angular error rates in both controlled and naturalistic scenes (Herashchenko et al., 2023).
• Vision-LLMs (VLMs) enable unified multi-modal understanding: models such as GazeVLM fuse RGB and HHA-encoded depth with textual prompts to execute person detection, gaze target localization, and semantic object identification in static imagery. These systems establish state-of-the-art performance for tasks including object-level gaze detection, defined by mean average precision $AP_{ob}$ at IoU ≥ 0.5 for predicted versus ground-truth object bounding boxes (Mathew et al., 9 Nov 2025).
• Interpersonal gaze interpretation applies dynamic density-based clustering (e.g., DBSCAN) to gaze angle sequences, extracting regions of visual engagement (RVE) in a 3×3 grid. The salient cluster is renormalized for interaction-centric labeling, facilitating assessments of interpersonal attention in social scenarios such as deception detection and social skill analysis. Validation yields high per-frame F1 (≈ 0.85) against IR tracking and moderate correlation ($r=0.37$) with expert eye contact ratings (Tran et al., 2019).

2. Object Referencing and Multimodal Interaction on Wearables

Gazeify is instantiated in the “Gazeify Then Voiceify” pipeline for displayless smart glasses:

• System architecture comprises an egocentric RGB camera, IR eye tracker (90 Hz), microphone, and gesture controls. Processing is split between a Unity-based front end (audio/gesture I/O) and a Python back end (EfficientSAM segmentation, RTDETR detection, GPT-4o VLM for language tasks) (Zhang et al., 27 Jan 2026).
• Gaze selection algorithm: Upon gesture (e.g., pinch), gaze samples within a symmetric Δ≈0.5 s window are enriched with color, depth, spatial, and velocity features. Clustering isolates the maximal coherent group; its centroid seeds point-prompted EfficientSAM segmentation, yielding candidate object masks from which the highest-confidence is selected.
• Conversational disambiguation (Voiceify): After gaze-based selection, the VLM generates object semantic descriptions and spatial relations. Mis-selections enable free-form voice corrections; the model parses target, reference, and relational cues, filters candidate bounding boxes (IoU/center distances), and selects updated object instances via multimodal scoring.

3. Gaze Synthesis and Photorealistic Redirection

Gazeify technologies extend to controllable gaze image synthesis:

• 3D eyeball modeling and Gaussian splatting: High-fidelity meshes (e.g., 3DGazeNet) are combined with Gaussian Head Avatar architectures, mapping multiple anisotropic Gaussians to the eye surface as computational primitives for photorealistic rendering (Choi et al., 8 Aug 2025).
• Explicit 3D transformations: Desired gaze directions are decomposed into yaw ($\theta$) and pitch ($\varphi$); a rigid-body transform is applied to all Gaussians, supplemented by MLP-predicted offsets for anatomical alignment during expression changes.
• Adaptive periocular deformation: MLP-driven displacement and color-modulation of Gaussians handle subtle muscle movements, preserving iris detail and eyelid creasing for realism over varying gaze or head poses. Integration of multi-stream losses (RGB, mask IoU, landmarks, Laplacian smoothness) yields convergence and maintains identity consistency.

4. Quantitative Performance, Usability, and Validation

Benchmarking across Gazeify systems demonstrates distinct strengths:

Component	Metric	Value(s)	Context
Gazeify Then Voiceify	Gaze selection acc.	53% (SD=13%)	≥90% mask coverage
	Voice correction succ.	58.4% of errors	43.3% gaze, 15.2% false neg.
	Latency (segm/gen)	0.72/3.61 s	Segmentation/voice
GazeVLM	$AP_{ob}$	0.23 – 0.25	Object-level, IoU≥0.5
	AUC (GazeFollow)	0.929	Static images
ICE (Video)	F1 score	0.846 (σ=0.086)	IR-validated
	Corr. w/eye contact	$r$ = 0.37	Human ratings
Appearance-based Estimation	MAE (Columbia Gaze)	1.93° – 1.04°	Real/synthetic
3D Gaze Synthesis	Gaze MAE (ETH-XGaze)	5.01° – 5.70°	Head/face region

Usability studies report System Usability Scale mean SUS=73.7 ± 16.6, low NASA-TLX cognitive/physical demand, and positive subjective ratings on likability, usefulness, and ease-of-use. Participants highlighted natural rapid gaze selection but cited excess description verbosity and some VLM hallucination phenomena (spatial ambiguities, part–whole errors) (Zhang et al., 27 Jan 2026).

5. Applications and Impact

Gazeify systems support a spectrum of interactive and analytic applications:

• Collaborative referencing: In meetings or gaming, gaze/voice referencing enables hands-free selection and annotation of objects with situational awareness.
• AR shopping and assistance: Users can gaze at physical items for semantic retrieval or real-time shopping assistance.
• Social analysis: ICE enables automated interpersonal attention measurement, deception detection (distinguishing truth-tellers/liers by region-frequency vectors), and communication skill assessment in speed-dating contexts, outperforming facial emotion signals for skill prediction (Tran et al., 2019).
• Human–robot interaction (HRI): Robots equipped with Gazeify can infer intent, direct gaze-based object selection, and facilitate natural situational awareness.
• Video analytics: GazeVLM indexing of “who looked at what when” supports sports or security analytics.
• 3D animation and avatar control: Photorealistic redirection of gaze enables expressive avatars in telepresence, VR, and digital content creation.

6. Limitations and Prospective Directions

Existing Gazeify frameworks exhibit technical constraints:

• Static-frame limitation: Leading VLM-based approaches lack temporal modeling capabilities for dynamic gaze interpretation (Mathew et al., 9 Nov 2025).
• Hardware and real-time constraints: High computation footprints (e.g., Qwen2-VL-2B VLM, 3DGS rendering) challenge mobile or wearable deployment without aggressive optimization.
• Object vocabulary: Detection/naming accuracy depends on extensibility of class vocabularies (e.g., LVIS) to handle novel objects.
• Robustness: OpenFace-based interpersonal gaze encoding is sensitive to lighting, head orientation, and 1:1 interaction assumptions.
• Generalization: Appearance-based CNN methods require domain adaptation for edge cases (e.g., users wearing glasses), and may incorporate adversarial training, ONNX quantization, or temporal smoothing (e.g., Kalman filtering) for deployment (Herashchenko et al., 2023).
• Conversational AI limits: VLM hallucinations and verbosity remain open challenges in multimodal object referencing, necessitating further research into concise and contextually-grounded language generation.

Further extensions target multi-party interaction calibration, lightweight GPU-accelerated clustering, real-time system tuning, and enhanced analytic capabilities (e.g., group-level engagement graphs). Incorporation of synthetic data pipelines, domain randomization, and multi-modal fusion offer promising avenues for improving robustness and application breadth.

7. Historical Context and Terminology

The term “Gazeify” serves as an umbrella for gaze-centric modeling techniques. Early frameworks focused on appearance-based estimation and feature extraction for eye contact (Tran et al., 2019), evolving into modular CNN-based systems leveraging synthetic data (Herashchenko et al., 2023). Subsequently, multimodal referencing pipelines (e.g., Gazeify Then Voiceify) operationalized gaze selection, object segmentation, and conversational correction for wearable and AR devices (Zhang et al., 27 Jan 2026). Transformers and VLMs such as GazeVLM delivered unified, prompt-driven approaches for gaze detection and semantic object understanding (Mathew et al., 9 Nov 2025). The concurrent emergence of explicit 3D gaze synthesis and redirection architectures further expanded the technological landscape, providing identity-preserving, photorealistic avatar control (Choi et al., 8 Aug 2025).

In sum, Gazeify comprises a cohesive suite of methodological advances—appearance-based regression, density-clustering calibration, multimodal vision-language integration, conversational object referencing, and photorealistic gaze control—empowering a new generation of interactive systems for gaze-driven understanding, collaboration, and social analytics.