SpeakAssis: AI Speaking Assistance

• SpeakAssis is a class of real-time, AI-powered assistive systems that enhance spoken communication using multimodal sensing and dynamic feedback.
• It integrates advanced machine learning techniques and multimodal inputs—such as audio, video, and gaze—to deliver actionable coaching, assessment, and intervention.
• Deployments range from public speaking coaching to automated speaking assessment and accessible communication for speech impairments, demonstrating scalability and measurable improvement.

SpeakAssis refers to a class of real-time, AI-powered assistive systems designed to support spoken communication across a range of use cases, from public speaking feedback and language learning assessment to accessible augmentation for individuals with speech or language challenges. These systems are characterized by their integration of multimodal sensing (audio, video, gaze), advanced machine learning pipelines for real-time signal analysis, and direct intervention strategies (feedback, reformulation, or scoring), targeting both normative and impaired communicative contexts. The term "SpeakAssis" has appeared to denote both specific wearable assistants for eye-contact retraining (Du et al., 1 Feb 2026) and blueprints for broader speaking assistant architectures (Phan et al., 23 Jul 2025, Lou et al., 23 Oct 2025, Xu et al., 21 Mar 2025).

1. System Architectures and Sensing Modalities

SpeakAssis systems feature a range of input and output modalities tailored to application:

• Public Speaking Coaching: The system employs a head-mounted Pupil Core eye tracker (scene camera: 1280×720 px, 30 Hz; dual eye cameras: 120 Hz). Real-time gaze estimation is achieved with the Pupil Capture framework, while computer vision modules (YOLOv8n-face for face detection; Inception-ResNet-V1 for face identification) enable audience segmentation and gaze-target inference. The processing unit is a laptop running the entire pipeline, with audio prompts delivered to the speaker via a Bluetooth earbud—ensuring feedback is private and minimally disruptive (Du et al., 1 Feb 2026).
• Automated Speaking Assessment (ASA): Architectures rely on audio capture, often chunked (e.g., 30 s windows) for compatibility with a Whisper-small or large-scale speech encoder. Embedding extraction and aggregation (AVG or Transformer-based) precede a regression or metric-based scoring head. No ASR or transcription is needed in Whisper-only pipelines, reducing dependency and inference latency (Phan et al., 23 Jul 2025, Lo et al., 2024).
• Speech Impairment Assistance and AAC: Mobile-edge deployments incorporate robust ASR frontends (WhisperX, robust-wisper), specialized impairment recognition (CNN+LSTM, Transformer), LLM-driven language refinement conditioned on impairment type, and neural TTS output with style/state control (Lou et al., 23 Oct 2025, Xu et al., 21 Mar 2025). Contextual embeddings encode conversational partner identity and emotional tone, modulating LLM and TTS output for expressivity and agency.
• Multimodal and Multispeaker Dialog: Advanced systems integrate multi-speaker diarization (TellWhisper, Hyper-SD with TS-RoPE positional encoding), multimodal context fusion (vision, gaze, emotional state), and complex event decoding (who-said-what-when token sequences) for full-spectrum dialog understanding (Hu et al., 7 Jan 2026, Xu et al., 21 Mar 2025).

2. Algorithmic Backbone and Feedback Mechanisms

Core algorithms underpinning these systems include:

• Real-Time Gaze Capture and Feedback: At each frame $t$, faces are detected ($I_i$), gaze points $(x_t, y_t)$ are compared (minimum Euclidean distance), and gaze region is labeled as audience/non-audience. Eye contact proportion (EP) and gaze entropy (GDE, Shannon entropy over per-segment gaze distribution) are computed in sliding windows. Threshold-based prompting (e.g., $EP < r_p = 20\%$) triggers discrete spoken cues (“Please look at the audience”; “Look left more”) at defined intervals ($n=30$ s, $k=75$ s) (Du et al., 1 Feb 2026).
• ASA and Metric-Based Assessment:
  • Audio segmentation and embedding extraction via Whisper encoders.
  • Aggregation (average or Transformer) produces a global vector; linear or metric-based heads predict CEFR scores.
  • Prototypical networks (with cosine or squared Euclidean distance) optimize separation among proficiency clusters; class-rebalanced losses address data imbalance (Phan et al., 23 Jul 2025, Lo et al., 2024).
• Impairment-Aware Reasoning and Adaptive Refinement:
  • Impairment recognition modules yield a class label $C$ (“dysarthria,” etc.), which is fed jointly with the ASR transcript $T$ as conditioning signals to an LLM that rewrites or clarifies impaired speech ($T' = SpeechRefine(T, C)$).
  • TTS modules then synthesize output, parameterized by style/condition, with low end-to-end latency ($T_{total} \approx 1.01$ s for an 11.5 s utterance) (Lou et al., 23 Oct 2025).
• Contextual and Multimodal Integration:
  • AAC implementations combine context vectors ($C = \alpha E_p + \beta E_e + \gamma Enc(H)$), fusing conversational partner, emotional tone, and utterance history.
  • LLM decoders admit multimodal prompting and cross-modal conditioning, outputting candidate utterances for user selection and further neural TTS rendering (Xu et al., 21 Mar 2025).
• Speaker Attribution in Dialog:
  • Speaker-labeled rotary encoding in self-attention (TS-RoPE), along with hyperbolic speaker classification heads (Hyper-SD), enables real-time, overlapping speech parsing and diarization, critical for multi-party interaction systems (Hu et al., 7 Jan 2026).

3. Quantitative Evaluation and Benchmarking

Extensive empirical assessment demonstrates the efficacy of SpeakAssis methods:

System / Task	Improvement / Score	Reference
Public speaking (eye contact)	EP $+62.5\%$, GDE $+17.4\%$	(Du et al., 1 Feb 2026)
ASA (Whisper-based)	RMSE 0.384 (vs. baseline 0.44)	(Phan et al., 23 Jul 2025)
ASA (Metric-SSL)	ACC $92.6\%$, Macro-ACC $98.2\%$	(Lo et al., 2024)
Impaired speech (SIR accuracy)	$91.4\% \pm 1.7$ (F1 $0.92 \pm .02$)	(Lou et al., 23 Oct 2025)
AAC (TTS MOS)	$4.2/5$ (vs. $3.1$ generic)	(Xu et al., 21 Mar 2025)
Multi-speaker diarization	DER $-2.27\%$ absolute (AliMeeting)	(Hu et al., 7 Jan 2026)

In user studies, public speaking users found the system useful (Likert: 4.3/5), while 75% of audience rated assisted talks more engaging ($p < 0.0001$) (Du et al., 1 Feb 2026). For impaired-speech and AAC contexts, pathologists rated LLM/TTS output highly on clarity and authenticity.

4. Application Domains and Deployment

Principal application domains include:

• Live public speaking feedback: Wearable interventions during presentations.
• Automated Speaking Assessment: Large-scale, data-efficient CEFR scoring in education, using audio-only (ASR-free) or multi-modal inputs.
• Impairment assistance: Mobile AAC for dysarthria, stuttering, aphasia; adaptive prompt engineering based on impairment detection.
• Augmentative communication: Context-driven message construction and personalized TTS for users with complex needs.
• Meeting/dialog understanding: Multi-speaker diarization and ASR for note-taking, accessibility, and interface control.
• ASL and gesture-based IPA for deaf users: Controlled-vocabulary ASL recognition and interface with visual/haptic feedback in narrow-domain applications (Tran et al., 2024).

Scalable deployment is achieved via containerized inference, autoscaling (e.g., AWS ECS for tens to hundreds of instances), and edge adaptation (e.g., 4-bit quantization for LLMs), enabling subsecond feedback in production settings (Kumar et al., 2023, Lou et al., 23 Oct 2025).

5. Limitations, Usability Considerations, and Design Recommendations

Observed limitations and open issues include:

• Prompt Intrusiveness and Modal Interference: Audio feedback may distract or disrupt speaker flow, especially under cognitive load (Du et al., 1 Feb 2026); visual or haptic signals are proposed as lower-distraction alternatives.
• User Agency: Over-reliance on AI-generated content or suggestions can erode perceived ownership of utterances in multilingual and AAC deployments; adjustable granularity of suggestions is advised (Qin et al., 3 May 2025, Xu et al., 21 Mar 2025).
• Latency and Resource Constraints: Unoptimized LLM or TTS stages can exceed acceptable latency for real-time use; smaller models, quantization, and server-client partitioning are effective mitigations.
• Domain and Modality Coverage: Narrow-domain systems (e.g., ASL-enabled IPA) operate on constrained vocabularies. This is an intentional stepping-stone for trust and robustness, with progressive expansion via data-driven vocabulary selection (Tran et al., 2024).
• Personalization and Adaptivity: Voice banking, impairment adaptation, partner/emotion embeddings, and federated model updates are crucial for handling the diversity of user needs and contexts (Lou et al., 23 Oct 2025, Xu et al., 21 Mar 2025).
• Multimodal Sensing and Fusion: Reliable integration of gaze, emotional state, and conversational partner identity enhances expressivity and contextual appropriateness but increases engineering complexity.

6. Future Directions and Research Opportunities

Key research frontiers for SpeakAssis-style systems include:

• Subtle and Non-Disruptive Feedback Mechanisms: Development of near-eye displays, smart glasses, or haptic feedback to replace/augment audio cues in live coaching scenarios (Du et al., 1 Feb 2026).
• Personalized, Cross-Modal Embedding Spaces: Learning contextually-rich, user-adaptive representations for efficient fusion of gaze, audio, vision, and context in real-time dialog (Xu et al., 21 Mar 2025).
• Domain Expansion for Accessible Communication: Transition from domain-limited ASL gesture recognition toward general-purpose sign language understanding, leveraging user adaptation and multi-modal feedback (Tran et al., 2024).
• Edge-Optimized and Federated Learning: Model compression, quantization, and federated adaptation to meet privacy, latency, and update requirements at scale (Wagner et al., 31 Jan 2025, Lou et al., 23 Oct 2025).
• Integrated Assessment and Content Validity: Ensuring that automated scoring systems unify acoustic proficiency and topical content coverage, potentially through hybrid pipelines that combine audio and ASR-text inputs (Phan et al., 23 Jul 2025).
• User-Centered Agency Control: Interfaces that foreground user edits, clearly demarcate AI-driven versus user-provided output, and dynamically surface suggestions based on pacing and interactional context (Xu et al., 21 Mar 2025).

SpeakAssis thus denotes a convergent field of AI research synthesizing real-time multimodal signal capture, advanced deep learning models, and user-centric intervention strategies to support human communication across public performance, education, accessible technology, and conversational settings. The reviewed literature demonstrates significant progress in objective outcomes (e.g., eye-contact enhancement, ASA accuracy), user satisfaction, and system scalability, with ongoing work focused on greater personalization, multimodal integration, and agency preservation (Du et al., 1 Feb 2026, Phan et al., 23 Jul 2025, Lou et al., 23 Oct 2025, Xu et al., 21 Mar 2025, Lo et al., 2024, Hu et al., 7 Jan 2026, Tran et al., 2024).