SARAH: Spatially Aware Real-time Agentic Humans
Abstract: As embodied agents become central to VR, telepresence, and digital human applications, their motion must go beyond speech-aligned gestures: agents should turn toward users, respond to their movement, and maintain natural gaze. Current methods lack this spatial awareness. We close this gap with the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset. Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with speech while orienting the agent according to the user. Our architecture combines a causal transformer-based VAE with interleaved latent tokens for streaming inference and a flow matching model conditioned on user trajectory and audio. To support varying gaze preferences, we introduce a gaze scoring mechanism with classifier-free guidance to decouple learning from control: the model captures natural spatial alignment from data, while users can adjust eye contact intensity at inference time. On the Embody 3D dataset, our method achieves state-of-the-art motion quality at over 300 FPS -- 3x faster than non-causal baselines -- while capturing the subtle spatial dynamics of natural conversation. We validate our approach on a live VR system, bringing spatially-aware conversational agents to real-time deployment. Please see https://evonneng.github.io/sarah/ for details.
Summary
- The paper introduces a novel real-time, causal transformer-based VAE combined with a flow matching model to generate spatially aware full-body motion for virtual agents.
- It employs a gaze guidance mechanism using classifier-free guidance and a per-frame gaze score, enhancing dynamic agent orientation toward moving users.
- Quantitative evaluations demonstrate state-of-the-art spatial alignment and performance, achieving over 300 FPS and improved metrics on the Embody 3D dataset.
SARAH: Spatially Aware Real-time Agentic Humans
Motivation and Contributions
The "SARAH: Spatially Aware Real-time Agentic Humans" (2602.18432) paper addresses the challenge of synthesizing full-body motion for virtual agents in settings such as VR, telepresence, and digital human applications. Existing methods exhibit significant limitations, notably a lack of spatial awareness and real-time causal architectures, resulting in agents that are oblivious to user position and gaze preferences, and incapable of streaming operation.
SARAH introduces the first real-time, fully causal, streaming approach to dyadic conversational motion generation. The method explicitly models spatial awareness and controllable gaze intensity, enabling agents to orient toward moving users and adapt gaze in accordance with individual and contextual preferences. SARAH achieves this via a lightweight, causal transformer-based VAE with interleaved latent tokens for efficient streaming, and a flow matching model conditioned on user position, dyadic audio, and a tunable gaze score for classifier-free guidance.
Architecture Overview
SARAH’s architecture is divided into three primary modules:
- Causal Transformer-based VAE: Compresses motion into temporally-strided latent sequences. Latent tokens are interleaved at fixed strides, enabling real-time, autoregressive streaming without sacrificing temporal coherence. Both encoder and decoder use strictly causal self-attention; each latent only attends to past information.
- Flow Matching Model: Generates motion in the learned latent space, conditioned on user floor-projected head position and both participants' HuBERT-extracted audio features. The flow matching approach enables efficient single-step generative modeling, crucial for real-time streaming.
- Gaze Guidance Mechanism: Employs a classifier-free guidance scheme, allowing explicit control of gaze alignment intensity during inference via a gaze score computed as the dot product between the agent’s head orientation and the direction toward the user.
Figure 1: SARAH architecture depicts causal VAE with interleaved latent tokens feeding into a flow matching module, allowing conversational and spatially-aware motion synthesis, with optional gaze control.
Motion Representation
Traditional kinematic representations suffer from error propagation and ambiguity in joint-angle predictions. SARAH utilizes a fully Euclidean parameterization: each joint is represented as a 3D icosahedron, with global position obtained from the centroid and orientation recovered via SVD. Mesh shells around joints encode surface geometry, allowing precise end-effector control and improving training stability.
Figure 2: Each joint is modeled as a 3D icosahedron; positional and orientation features are extracted for Euclidean representation.
Gaze Control and Guidance
SARAH's gaze guidance mechanism allows dynamic control of eye contact during inference. The gaze score () is computed per frame as the dot product of the agent's facing direction and the vector toward the user. Guidance is implemented by concatenating the target gaze score with conditioning and applying classifier-free guidance with probabilistic modality dropout. At inference, can be tuned to steer the agent’s orientation, enabling adaptation to various conversational and cultural contexts.
Figure 3: The training data exhibits diverse gaze behaviors; gaze control is realized by computing a per-frame score and modulating the agent’s facing direction.
Figure 4: Visualization of agent’s facing direction as gaze score increases, demonstrating explicit controllable turning toward the user.
Dataset and Training
SARAH is trained on the dyadic subset of Embody 3D, the first dataset to capture dynamic spatial proxemics in conversation with rich demographic diversity and varied topics. Training utilizes a strict 80/10/10 data split and causal audio feature extraction via HuBERT-Large, circumventing information leakage during streaming inference.
Quantitative and Qualitative Evaluation
SARAH sets state-of-the-art performance benchmarks on multiple axes:
- Fréchet Gesture Distance (FGD): Distributional similarity between generated and ground truth meshes.
- FGD_acc: Measures acceleration for motion smoothness and dynamics.
- Foot Slide: Quantifies skating artifacts indicating physical implausibility.
- Wrist Var: Assesses gesture expressiveness via wrist velocity.
- Head Ang.: Mean dot product between agent facing direction and user position; quantifies spatial gaze alignment.
SARAH achieves competitive FGD (1.28) compared to retrieval baselines, while outperforming nearest-neighbor in spatial awareness (0.83 vs. 0.59) and running over 300 FPS—three times faster than non-causal baselines (MDM, A2P) that require future context. Qualitative results demonstrate SARAH’s responsiveness to user spatial motion and conversational context, with photorealistic avatar rendering.
Figure 5: SARAH produces realistic, spatially-aware conversational motion for photorealistic avatars, responsive to user movement.
Practical and Theoretical Implications
SARAH advances interactive AI agent capabilities, enabling real-time spatially-aware, controllable conversational agents deployable in VR and streaming environments. The architecture decouples learning from control, allowing fine-grained adjustment of gaze behaviors at inference without dataset bias constraints. The causal transformer VAE and flow matching operational paradigm pave the way for efficient motion modeling, robust to the demands of streaming and non-deterministic conversational dynamics.
The approach exposes the necessity for explicit spatial conditioning and control mechanisms in agentic motion synthesis. As virtual agents proliferate, practical applications span immersive entertainment, telepresence, education, and social robotics. The Euclidean surface-point representation enhances downstream tasks requiring physical plausibility and precise end-effector manipulation.
Limitations and Future Directions
SARAH’s dependence on training data biases can limit generalization to underrepresented spatial configurations and gaze behaviors. Current control is limited to gaze; gesture style and locomotion remain non-controllable. Extension to multi-party interactions will necessitate architectural innovation. Further research may explore hierarchical control schemas, integration of semantic gesture conditioning, and adaptation to broader interaction contexts, possibly leveraging LLMs for nuanced behavioral guidance.
Conclusion
SARAH establishes a new standard for real-time, spatially-aware conversational agent motion synthesis. Its causal transformer-based latent modeling, Euclidean motion representation, and gaze guidance mechanism unlock practical deployment in streaming environments with explicit controllability. Numerical results confirm state-of-the-art quality and efficiency, and SARAH's modular architecture is primed for future enhancements in multi-modal embodied AI.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper is about making virtual people (avatars) in VR and other apps move and act more like real humans during a conversation. Instead of just waving their hands to match speech, the avatar should also turn toward you, react when you move, and make natural eye contact—all in real time.
Key Objectives and Questions
The authors set out to build a system that can:
- Produce gestures that match the avatar’s speech.
- Be “spatially aware”: turn and react to where the user is and how they move.
- Let you control how much eye contact the avatar makes (some people prefer more, some less).
- Run in real time, like a live video call, without needing to “peek” at future frames.
In simple terms: Can we make a virtual person who listens, looks, and moves like a real person while you’re talking, and can we adjust their eye contact?
How They Did It (Methods)
To make this happen live, the team designed a fast, step-by-step system that only uses the past and present (not the future). Here’s the approach, explained with everyday ideas:
Real-time, “causal” model
“Causal” means the model never looks into the future. It only uses what has already happened and what’s happening now—like you reacting to a friend in real time as they speak and move.
“Streaming” means the system makes motion a little at a time, continuously, just like streaming a song without downloading the whole thing first.
Compress-and-generate pipeline
The system works in two parts:
- A motion compressor (a transformer-based VAE):
- Think of this like making a summary of the avatar’s movement over short time blocks, so the computer can handle it quickly.
- “Interleaved latent tokens” are these small summaries placed regularly along the timeline, so the model can generate motion smoothly, bit by bit, without losing track of what happened earlier.
- A motion generator (flow matching model):
- Imagine starting with random noise and then gently “pulling” it toward a natural motion, like guiding a dot through a maze to the right path.
- This generator uses the user’s position on the floor (where their head is) and both people’s audio features to guide the avatar’s movement.
- It runs in single steps, which makes it fast enough for real-time use.
Controllable eye contact (“gaze guidance”)
- The model learns natural gaze from real conversations (sometimes strong eye contact, sometimes looking away).
- At run time, you can turn a “gaze dial” to ask for more or less eye contact. The system nudges the avatar’s head direction to match your preference, but still keeps motion looking natural.
A simple, stable way to represent motion
- Instead of using complicated joint angles (which can be tricky and error-prone), the avatar’s body parts are represented with 3D points and a small 3D shape per joint (an icosahedron—think of a many-sided mini soccer ball).
- Using positions and directions directly (a “Euclidean” representation) makes training more stable and helps keep hands, feet, and head where they should be.
Data they used
- The team trained and tested on Embody 3D, a big dataset of two-person conversations where people walk around, change positions, and interact naturally.
- They use the user’s floor-projected head location (easy to get from a VR headset) and audio from both people to guide the avatar’s responses.
Main Findings and Why They Matter
Here are the key results:
- Real-time speed: The system runs at over 300 frames per second, which is much faster than many older methods (about 3× faster than some non-real-time ones). This means it’s fast enough for live VR.
- Natural, spatially aware motion: The avatar gestures with speech and turns toward the user, even without looking at future frames. It matches the “gaze alignment” of slower, non-real-time methods.
- Low visual errors: It keeps feet from “sliding” unrealistically over the ground and maintains expressive hand motion, which makes the avatar look more lifelike.
- Adjustable eye contact: The gaze control works. Turning the dial changes how much the avatar looks at the user while still keeping movement natural.
- Works on a live system: They tested it with an actual VR avatar, and it behaved well in real conversations.
Why this is important: When avatars act like real people—turning to face you, reacting as you move, and making comfortable eye contact—VR, telepresence, and digital human experiences feel much more believable and pleasant.
Implications and Impact
This research helps build avatars that feel present and responsive in real-time conversations. It could improve:
- VR companions and tutors that feel attentive.
- Telepresence meetings with avatars that react naturally, like real colleagues.
- Games and social apps with lifelike characters.
It also puts control in the user’s hands: you can set eye contact to a level that feels right for you or your culture. Looking ahead, the authors note that:
- The system learns from its training data, so rare behaviors might not be perfect.
- There’s room to add more controls (like specific gesture styles or movement speed).
- Extending this to group conversations (more than two people) will take more work.
Overall, this is a big step toward virtual humans that talk, look, and move like real partners—reliably and in real time.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a concise list of concrete gaps and unresolved questions that future work could address:
- Real-time evaluation details: FPS is computed from batch generation of 400 frames rather than true streaming; end-to-end latency, jitter tolerance, and responsiveness to abrupt user motion on commodity VR hardware are not quantified.
- On-device deployment: The model is trained and evaluated on A100 GPUs; energy, memory footprint, and performance on headset-class or mobile hardware (including optimization paths) are unreported.
- Long-horizon stability: The autoregressive imputation strategy may accumulate errors over extended interactions; no analysis of drift, temporal coherence degradation, or recovery under prolonged sessions.
- Conditioning scope: The method conditions only on the user’s floor-projected head position and dyadic audio; the impact of incorporating full-body user pose, user gaze direction, hand gestures, and environmental context remains unexplored.
- Eye vs. head gaze: “Gaze” is approximated via head orientation; explicit eye modeling (saccades, vergence, blinks), decoupling of eye-head coordination, and perceptual effects of these signals are not addressed.
- Personalization: Gaze guidance is globally controllable but not personalized; no mechanism to learn or adapt a user-specific gaze profile over time, nor procedures for quick calibration or online feedback integration.
- Multi-dimensional control: Only gaze intensity is controllable; controls for gesture style, amplitude, timing, interpersonal distance, locomotion speed, and turn-taking behavior are missing.
- Turn-taking and discourse structure: The model uses audio features but does not explicitly model turn-taking cues, prosodic structure, or semantic/pragmatic content (e.g., text or conversation state), limiting alignment of gestures to discourse-level events.
- Robustness to sensing noise: Sensitivity to noisy audio (e.g., overlapping speech, room reverberation) and noisy/laggy head tracking is not evaluated; failure modes under sensor dropouts or latency spikes are unknown.
- Environment and physical constraints: Contact, balance, and collision avoidance are not enforced; behavior around obstacles, tight spaces, and varying floor geometries is not studied.
- Locomotion planning: While full-body motion is generated, step planning, footfall timing, and path planning relative to moving users or environmental constraints are not explicitly modeled or evaluated.
- Dataset bias and generalization: The method “inherits training data biases”; generalization to rare proxemic configurations, new room geometries, cultural gaze norms, and unseen interaction styles is not quantified.
- User safety and comfort: No human-subject evaluation of comfort, social acceptability, and cultural appropriateness—especially under extreme gaze settings (e.g., g=1.0)—or guidelines for safe ranges of eye contact.
- Multi-party interactions: Extending to >2 participants (social formations, shifting foci of attention, group turn-taking) and associated architectural changes is explicitly left for future work.
- Cross-avatar retargeting: How the Euclidean icosahedron representation transfers across rigs with different joint topologies, bone lengths, and hand/finger articulations is not discussed; fidelity of fine-grained hand motion is unclear.
- Physical realism metrics: Evaluation focuses on foot slide and kinematic metrics (FGD/FGD_acc); broader physical plausibility (ground reaction forces, momentum, balance) and biomechanical realism are not assessed.
- Semantic appropriateness: There is no evaluation of gesture semantics (iconic, deictic, beat gestures) or alignment with spoken content; integrating text or semantic signals and measuring appropriateness remains open.
- Causality vs. anticipation: The causal model matches gaze alignment without future access, but the limits of causal reactivity (e.g., anticipating rapid user trajectory changes) and potential benefits of minimal lookahead are not studied.
- Parameter sensitivity: The effects of stride s, latent dimensionality, KL β, CFG strength, and history buffer length on quality, stability, and speed are not systematically analyzed.
- Guidance extremes and failure modes: For high guidance (g≈1.0), FGD worsens; robustness to out-of-distribution guidance targets, automatic calibration of g per context, and strategies to avoid unnatural behavior are unexamined.
- Comparative benchmarks: Baselines are retrained on Embody 3D with adaptations; standardized benchmarks, cross-dataset evaluation, and user studies for perceptual quality would strengthen claims.
- Real-world deployment specifics: Integration details (tracking pipelines, audio capture/processing, synchronization), bandwidth constraints, and end-to-end system reliability in live VR telepresence are not reported.
- Ethical considerations: Potential impacts of gaze manipulation (persuasion, discomfort), fairness across demographics, and disclosure/consent mechanisms for adaptive social behaviors are not addressed.
Practical Applications
Practical Applications of SARAH: Spatially Aware Real-time Agentic Humans
Below are actionable, real-world applications derived from the paper’s findings and innovations. Each item specifies sector(s), potential tools/products/workflows, and key assumptions or dependencies that could impact feasibility.
Immediate Applications
The following applications are deployable now, leveraging SARAH’s causal, real-time, streaming architecture and controllable gaze guidance.
- Virtual telepresence and collaboration in VR/AR (software, enterprise)
- Use case: Real-time avatars that orient to a moving participant, maintain natural gaze, and gesture in sync with speech for meetings, training, and remote collaboration.
- Tools/products/workflows: Unity/Unreal plugin (SARAH SDK) with “UserTrajectoryInput” from the headset, on-device or edge inference server, a UI “gaze intensity” slider per user.
- Assumptions/dependencies: Access to headset head-position tracking; dyadic audio streams (or robust speaker separation); sufficient compute on-device or low-latency edge; integration with avatar renderers (e.g., MHR).
- Customer service kiosks and digital receptionists (retail, finance, hospitality)
- Use case: Screen-based digital humans that turn toward approaching customers, modulate eye contact for comfort, and respond conversationally.
- Tools/products/workflows: Camera-based user position estimation, SARAH motion engine, policy presets for eye contact intensity (e.g., “low-gaze” setting for privacy-conscious contexts).
- Assumptions/dependencies: Single-user focus; environmental sensors (RGB-D or stereo camera) for head localization; clear audio capture; policies for consent and recording.
- VR games and social platforms: spatially aware NPCs and companions (gaming, entertainment)
- Use case: NPCs that react to player movement and speech in VR worlds, with tunable gaze and natural gestures.
- Tools/products/workflows: Behavior-tree integration, real-time audio capture pipeline, gaze guidance API tied to difficulty or comfort settings.
- Assumptions/dependencies: Generalization to diverse game motions; limited multi-user support; content moderation for voice inputs.
- Virtual tutors and trainers (education)
- Use case: Tutors that face learners, maintain appropriate gaze based on learner preference, and gesture naturally during instruction.
- Tools/products/workflows: LMS integration; gaze comfort presets; analytics logging for engagement (e.g., dwell time, orientation).
- Assumptions/dependencies: Cultural and age-dependent gaze norms; audio quality and latency; controlled classroom/VR environment.
- Social skills training and therapy (healthcare)
- Use case: Adjustable gaze intensity to help users practice eye contact and conversational proxemics (e.g., social anxiety support).
- Tools/products/workflows: Clinical/therapeutic app with clinician-configurable gaze settings; session logging for progress; privacy controls.
- Assumptions/dependencies: Need for clinical validation; individualized calibration; sensitive handling of personal data.
- Human–Computer Interaction research and user studies (academia)
- Use case: Controlled manipulation of gaze intensity and spatial alignment to study proxemics, engagement, and comfort in VR/AR.
- Tools/products/workflows: Experimental toolkit exposing gaze score and motion outputs; auto-logging of FGD, foot-slide, head-angle metrics; reproducible pipelines.
- Assumptions/dependencies: IRB/ethics approvals; awareness of dataset bias in proxemics and gaze behaviors.
- Social robots with expressive head/torso motion (robotics)
- Use case: Mapping SARAH’s head orientation and upper-body gestures to robot kinematics for in-person interactions.
- Tools/products/workflows: ROS node that ingests user head position and audio; kinematic adapters; safety envelope constraints.
- Assumptions/dependencies: Mechanical limits and degrees of freedom; collision and safety policies; controlled indoor settings; foot motion less relevant.
- Automated talking-head and avatar animation for content creation (media/entertainment)
- Use case: Reduce keyframing by generating engaging, speech-aligned, spatially aware motion for talking characters and digital humans.
- Tools/products/workflows: DCC pipeline plugin (e.g., for Blender/Unity/Unreal); batch generation from recorded audio plus rough user trajectory; gaze presets per scene.
- Assumptions/dependencies: Style control beyond gaze is limited; quality depends on audio clarity; may require post-processing for specific artistic styles.
Long-Term Applications
These applications require further research, scaling, or development, including expanded datasets, multi-party support, broader controls, and policy frameworks.
- Multi-party conversation and group dynamics (software, enterprise, education)
- Use case: Agents that manage gaze and orientation across multiple participants in meetings or classrooms.
- Tools/products/workflows: Multi-subject tracking; attention allocation strategies; group-aware conditioning pipelines.
- Assumptions/dependencies: New datasets capturing multi-party proxemics; architectural modifications for multi-user conditioning and turn-taking.
- Full-body social robots with locomotion and physical interaction (robotics)
- Use case: Humanoids that not only orient and gesture but also reposition, maintain spacing, and respect proxemic norms in shared spaces.
- Tools/products/workflows: Whole-body control stack integrating SARAH’s motion latents with foot contact planning and balance; onboard perception; safety certification processes.
- Assumptions/dependencies: Hardware capabilities; robust perception; safety and liability frameworks; real-world generalization beyond lab settings.
- Rich behavioral control: style, affect, and cultural adaptation (software, HCI, localization)
- Use case: Controllable parameters for gesture style, emotional tone, cultural gaze norms, and situational etiquette.
- Tools/products/workflows: Expanded control signals beyond gaze (style tokens, affect sliders); region- and culture-specific presets; user profiling modules.
- Assumptions/dependencies: New labeled datasets; ethical handling of cultural stereotypes; UX testing to avoid bias and misrepresentation.
- Integrated conversational agents combining SARAH with LLMs (software, customer service, education)
- Use case: End-to-end agents that listen, reason, speak, and move naturally with synchronized verbal and non-verbal cues.
- Tools/products/workflows: Pipeline tying ASR/TTS/LLM dialogue with SARAH motion; prosody-to-gesture alignment; latency-optimized streaming stack.
- Assumptions/dependencies: Tight synchronization across modalities; robust diarization/speaker separation; cost-effective deployment at scale.
- Clinical applications: autism and social communication therapies (healthcare, policy)
- Use case: Evidence-based interventions using controllable gaze and proxemics in VR to improve social communication skills.
- Tools/products/workflows: Clinical trial platforms; standardized outcome measures; clinician dashboards; integration with EHRs where appropriate.
- Assumptions/dependencies: Regulatory approvals; clinical validation; privacy-preserving data storage and device compliance (HIPAA, GDPR).
- Privacy-preserving, on-device inference and compliance (policy, software)
- Use case: Deploy models entirely on headsets or local edge to avoid transmitting raw audio/video; enforce transparency and consent.
- Tools/products/workflows: Optimized model compression; secure enclaves; auditable telemetry; configurable consent prompts.
- Assumptions/dependencies: Hardware acceleration on consumer devices; adherence to regulations (GDPR/CCPA); clear organizational policies.
- Standards and best practices for digital-human proxemics (policy, industry consortia)
- Use case: Sector-wide guidelines for safe, respectful gaze and distance behaviors across contexts (education, healthcare, retail).
- Tools/products/workflows: Benchmark suites (FGD, foot slide, gaze metrics); recommended presets by context; certification programs.
- Assumptions/dependencies: Cross-cultural research; stakeholder input (users, clinicians, educators); dynamic updating as norms evolve.
- Synthetic data generation for training perception and interaction models (academia, software)
- Use case: Use SARAH to generate large-scale, labeled datasets of realistic dyadic proxemics and gaze for model training and evaluation.
- Tools/products/workflows: Data generation pipelines; automatic labeling of head angles/gaze scores; domain randomization for environments.
- Assumptions/dependencies: Validation to avoid synthetic bias; careful alignment with real-world distributions; transparency in dataset documentation.
- Previsualization and virtual production at scale (media/entertainment)
- Use case: Director-driven control over gaze and spatial alignment during pre-vis; later swap in high-fidelity motion or refine styles.
- Tools/products/workflows: Integration with virtual production toolchains; batch control scripts; export to standard motion formats.
- Assumptions/dependencies: Expanded style and emotion controls; robust project-level workflows; collaboration with DCC ecosystems.
- Accessibility and inclusive design across devices (policy, software)
- Use case: System-level settings to adjust eye contact or proximity for users with neurodiversity or cultural preferences.
- Tools/products/workflows: OS-level gaze comfort controls; per-app overrides; accessibility audits and labels.
- Assumptions/dependencies: Broad device support; inclusive UX research; interoperability standards.
Notes on general dependencies and feasibility across applications:
- The current model is optimized for dyadic interactions and conditions primarily on user head position plus dyadic audio; multi-party and full-body conditioning will require architectural and dataset extensions.
- Real-time performance depends on hardware and deployment strategy; while the paper demonstrates streaming on VR headsets and 300+ FPS on high-end GPUs, consumer devices may need model compression and edge offloading.
- Audio capture quality and speaker separation/diarization are critical for robust alignment; noisy environments may necessitate beamforming and noise suppression.
- The system inherits dataset biases (e.g., gaze norms, proxemic patterns); careful evaluation and mitigation are needed for cross-cultural, age-diverse, and accessibility-sensitive deployments.
Glossary
- Acausal methods: Generative models that use both past and future context, making them unsuitable for real-time streaming. Example: "have focused on acausal methods, e.g. vanilla diffusion"
- Adaptive Layer Normalization (AdaLNZero): A conditioning mechanism that injects conditioning (e.g., timestep) via adaptive layer norm; AdaLNZero is a zero-initialized variant popular in diffusion/flow transformers. Example: "To incorporate the noise timestep, we use AdaLNZero."
- Autoregressive: A generation paradigm that predicts the next token/block conditioned only on previously generated outputs. Example: "real-time, autoregressive streaming"
- Causal attention: Attention masking that restricts each position to attend only to past positions to ensure causality. Example: "We apply causal self-attention: each frame attends only to past frames, and each μk/σk token attends to preceding frames and earlier latent tokens."
- Causal masking: An attention mask enforcing that tokens attend only to earlier tokens during generation. Example: "For real-time streaming, we enforce strict causality via causal attention masking."
- Causal VAE (Variational Autoencoder): A VAE whose encoder/decoder use causal attention so that encoding/decoding proceeds without access to future frames, enabling streaming. Example: "We propose a causal VAE architecture to support streaming inference."
- Classifier-Free Guidance (CFG): A sampling technique that mixes conditional and unconditional predictions to control adherence to conditioning signals. Example: "we introduce a gaze scoring mechanism with classifier-free guidance to decouple learning from control"
- Diffusion forcing: A technique to adapt diffusion models for causal generation by enforcing consistency with past frames. Example: "or diffusion forcing"
- Distillation: Compressing multi-step generative samplers into single-step models for real-time performance. Example: "The video diffusion community has adopted distillation to compress multi-step models into single-step models for real-time streaming"
- Dyadic: Pertaining to two-person interactions or data involving two participants. Example: "Given a user's position and dyadic audio, our approach produces full-body motion"
- End-effector: The terminal part of a kinematic chain (e.g., hand or foot) whose position/orientation is often constrained or controlled. Example: "enables precise end-effector control."
- Euclidean motion representation: Representing pose directly in Cartesian space (positions/orientations) rather than joint angles, improving stability. Example: "a fully Euclidean motion representation that improves training stability and enables precise end-effector control."
- Flow matching: A generative framework that learns a velocity field to transport noise to data in a single or few steps. Example: "We adopt a transformer-based flow matching model for real-time, causal motion generation."
- Foot Slide: A metric measuring sliding artifacts when feet should be stationary on the ground. Example: "Foot Slide, the fraction of frames where feet are near the ground (<5\,cm) yet moving horizontally (>3\,cm/s), indicating skating artifacts;"
- Forward kinematics: Computing global joint positions from local joint rotations and root transforms along a kinematic chain. Example: "using forward kinematics and linear blend skinning to obtain meshes"
- Fréchet Gesture Distance (FGD): A distributional metric (Fréchet distance) comparing generated and real gesture pose distributions. Example: "FGD (Fréchet Gesture Distance), which measures distributional similarity between generated and ground-truth poses via the Fréchet distance over the vertex positions of the mesh;"
- Gaze score: A scalar measuring alignment between the agent’s facing direction and the direction toward the user, used for controllable gaze. Example: "To enable controllable gaze at inference, we compute a gaze score , where is the agent's facing direction and points toward the user"
- Head Ang.: A metric for gaze alignment defined as the mean dot product between facing direction and direction to user. Example: "Head Ang., the mean dot product between the agent's facing direction and the vector toward the user, quantifying gaze alignment ($1$\,=\,facing user, \,=\,facing away)."
- HuBERT features: Self-supervised speech representations used as conditioning for motion generation. Example: "we extract HuBERT features~\cite{hsu2021hubert} from each audio stream"
- Icosahedron: A 20-faced polyhedron used here to encode joint position and orientation via its vertices. Example: "We represent each joint as a 3D icosahedron."
- Imputation: Filling in (or fixing) parts of the latent/noisy sequence with known or previously predicted values to enforce temporal consistency. Example: "we enforce temporal consistency through imputation."
- Inverse kinematics: Recovering joint angles that achieve desired end-effector positions/orientations. Example: "Mesh positions are then recovered via inverse kinematics."
- KL divergence: A regularization term in VAEs that encourages the posterior to match a prior distribution. Example: "We optimize the VAE with reconstruction and KL losses:"
- Linear blend skinning: A standard mesh deformation method that blends joint transformations to produce vertex positions. Example: "using forward kinematics and linear blend skinning to obtain meshes"
- Monadic: Pertaining to single-person settings without an explicit interlocutor or spatial partner. Example: "Existing gesture generation methods are predominantly monadic: they synthesize motion for a single speaker conditioned on audio or text, with no awareness of an interlocutor"
- Oculesics: The study/use of eye movement and gaze in communication. Example: "Oculesics (eye gaze and contact~\cite{kendon1967some}) and proxemics (interpersonal distance~\cite{argyle1965eye}) play crucial roles"
- Proxemics: The study/use of interpersonal distance and spatial relations in social interaction. Example: "Oculesics (eye gaze and contact~\cite{kendon1967some}) and proxemics (interpersonal distance~\cite{argyle1965eye}) play crucial roles"
- RoPE (Rotary Positional Encoding): A positional encoding scheme that rotates query/key vectors by position-dependent phases. Example: "We then use rope for temporal positional encoding."
- Streaming inference: Generating outputs online in real time as inputs arrive, without future context. Example: "with interleaved latent tokens for streaming inference"
- SVD (Singular Value Decomposition): A matrix factorization used here to recover joint orientation from surface points. Example: "we recover the global orientation via SVD against a reference icosahedron."
- Temporal stride: The fixed interval between interleaved latent tokens/blocks in a sequence model. Example: "we interleave latent tokens at a fixed temporal stride ."
- Vector Quantization (VQ): Discretizing continuous embeddings into codebook indices for efficient autoregressive modeling. Example: "vector-quantized motion tokens"
- VQ-VAE (Vector-Quantized Variational Autoencoder): A VAE whose latent space is quantized into a discrete codebook. Example: "It employs separate VQ-VAEs for arm and hand movements"
- Wrist Var: A metric for gesture expressiveness based on average wrist velocity. Example: "Wrist Var, the average wrist velocity measuring gesture expressiveness;"
- x_1-prediction: A training objective variant that predicts a specific target (here denoised data at flow time) instead of velocity/noise. Example: "Using -prediction, we train:"
Collections
Sign up for free to add this paper to one or more collections.
