Text-to-Trajectory Paradigm: Bridging Language and Motion

• Text-to-Trajectory Paradigm is a computational framework that maps natural language descriptions into time-resolved continuous or discrete motion paths by bridging semantic and physical spaces.
• It employs dual encoders and generative decoders to ensure semantic alignment and physical plausibility through methods like contrastive losses, reconstruction, and flow matching.
• Its applications span robotics, autonomous navigation, animation, and human-computer interaction, demonstrating its practical impact on multi-modal control and task planning.

The text-to-trajectory paradigm encompasses a family of computational frameworks that map natural language input—typically free-form textual descriptions, commands, or constraints—into explicit, time-resolved continuous or discrete trajectories. These trajectories may represent spatial paths, kinematic states, motion plans, or higher-level task sequences, and are grounded in a joint semantic-physical space defined by the architecture and embeddings of the underlying model. Bridging a crucial gap between symbolic language and control-oriented representations, text-to-trajectory models have advanced in domains including animation, robotics, cinematography, human–system interaction, autonomous navigation, video generation, audio spatialization, procedural tool use, and reinforcement learning.

1. Problem Formulation and Core Modeling Principles

Text-to-trajectory models formally define two principal domains:

• The text space $\mathcal{T}$, comprising natural-language descriptions, semantic constraints, or context instructions.
• The trajectory space $\mathcal{M}$ (or $𝒳$ or similar notation), representing sequences of states/positions, typically as $N \times d \times T$ tensors (e.g., $N$ points in $d$-dimensions tracked over $T$ frames).

The learning objective is generally to learn mappings $f_{\mathrm{text}}: \mathcal{T} \to \mathbb{R}^d$ and $f_{\mathrm{traj}}: \mathcal{M} \to \mathbb{R}^d$ into a shared or jointly supervised latent space, such that semantically and physically coherent (text, trajectory) pairs are proximal, and decoding from text embeddings produces physically plausible, contextually appropriate motion (Galoaa et al., 11 Dec 2025, Qian et al., 30 Dec 2025, Lee et al., 2024).

Across instantiations, this paradigm can be factored into:

• A semantic encoder (Transformer, CLIP, BERT, etc.) for text,
• A trajectory encoder (often Transformer- or MLP-based) for paths/motion,
• A decoding or generation process, generative or deterministic, that outputs trajectories conditioned on text (or text+context).

In domains interfacing with task-planning or logic (e.g., dialogue pipelines or RL constraints), text-to-trajectory may extend to mapping language into stepwise action plans or procedural traces, with tool definitions or intermediate semantic representations as auxiliary artifacts (Xu et al., 15 Jan 2026).

2. Model Architectures and Learning Pipelines

Several structural approaches have emerged:

• Autoencoding with Joint Embedding Alignment: Lang2Motion uses a transformer-based auto-encoder trained with dual supervision from textual captions and visual trajectory overlays, using CLIP-based text and image encoders to enforce alignment in a shared latent space. At inference, the trajectory encoder is replaced by the text encoder, allowing generation of plausible object trajectories purely from language (Galoaa et al., 11 Dec 2025).
• Latent Motion Reasoning (LMR): LMR decomposes generation into two manifolds—“reasoning” (coarse, semantic, global trajectory) and “execution” (high-frequency, kinematic details)—employing a dual-granularity tokenizer. The two-stage process mirrors cognitive motor planning, addressing the semantic-kinematic impedance by separating intent from instantiation; decoder heads are conditioned autoregressively, bridging high-level description and low-level motion (Qian et al., 30 Dec 2025).
• Manifold Flow Decoupling: MMFP first learns a low-dimensional manifold of physically valid motions by deterministic autoencoding, independent of task or text. Conditioned normalizing flows are then trained to map text encodings to latent embeddings, with flow matching and paraphrase-robustness regularization enabling data-efficient many-to-many text-motion mapping (Lee et al., 2024).
• Classical Planning with LLM-Driven Intents: Text2Traj factors the pipeline into text→plan→trajectory, combining LLM-prompted intent and action generation with geometric planners (such as PRM/DWA) to instantiate collision-free, task-grounded paths. This modular structure is domain-agnostic, generalizing to settings such as retail behavior, indoor navigation, and traffic simulation (Asano et al., 2024).
• Multi-Modal and QA-Driven Trajectory Generation: In LMTraj, both history and future trajectories are cast as textual QA pairs; scene images are captioned into text, coordinates are tokenized into string format, and a tokenizer specialized for numerals ensures accurate semantic/physical encoding. LLMs (e.g., T5) are trained using these prompts for both deterministic (beam) and stochastic (temperature) prediction, demonstrating strong multi-modal forecasting (Bae et al., 2024).

The following table details representative architecture patterns:

Paradigm	Semantic Encoder	Trajectory (Motion) Encoder	Decoding Mechanism	Alignment Supervision
Lang2Motion	CLIP Text/Image	Transformer Auto-encoder	MLP Autoregressive	Dual contrastive, direct recon
LMR	Transformer (text→lat.)	Dual-Granularity Tokenizer	Autoregressive (discrete/cont.)	CLIP-alignment, masked LM
MMFP	Sentence-BERT + MLP	Deterministic Autoencoder	Conditional flow-matching in latent	Flow-matching losses, paraphrase
Text2Traj	GPT-4 style LLM	No learned traj encoder	Classical PRM + DWA planners	Human-verified plausibility
LMTraj	Caption (BLIP-2), T5	Numeric Tokenizer	T5 QA decoder	Cross-entropy, multi-task QA

3. Objective Functions and Losses

Text-to-trajectory models frequently employ composite objectives, multitask alignment, and auxiliary supervision:

• Reconstruction Losses: $L_{\text{recon}}$ (per-point $L_1$), $L_{\text{vel}}$ (velocity matching), $L_{\text{range}}$ (spatial extent) ensure that generated trajectories replicate physical properties of reference data (Galoaa et al., 11 Dec 2025).
• CLIP-Based and Contrastive Losses: $L_{\text{text}}, L_{\text{image}}$ incentivize semantic embeddings of text, rendered trajectory overlays, and their autoencoder latents to agree in $d$-dimensional joint space; contrastive variants enforce pairwise alignment at batch level (Galoaa et al., 11 Dec 2025, Qian et al., 30 Dec 2025, Lee et al., 2024).
• Latent Flow Matching and Optimal Transport: In flow-based models, training targets velocity fields that interpolate samples in latent space from canonical priors to demo-based codes, regularized for robust text mapping (including paraphrase robustness) (Lee et al., 2024).
• Dual-Stage Decoding Losses: LMR optimizes both for kinematic reconstruction from execution tokens and semantic alignment via global CLIP similarity and masked text prediction from reasoning tokens, reflecting the decoupling of physical and semantic objectives—which are shown to be topologically orthogonal in latent space (Qian et al., 30 Dec 2025).
• QA and Language Modeling Losses: In QA-formulated predictors, cross-entropy is minimized over the output token sequence for each prompt/answer (“What trajectory does pedestrian $i$ follow?”) and regularized to cover auxiliary reasoning tasks (destination, direction, grouping, mimicry, collision) (Bae et al., 2024).

4. Evaluation Methodologies, Datasets, and Metrics

Evaluation across works generally involves retrieval, generation, and compatibility/semantic metrics:

• Retrieval Metrics: Recall@$k$ quantifies the ability to retrieve correct trajectories given text or vice versa; Lang2Motion achieves Recall@1 = 34.2%, Recall@5 = 71.3% on MeViS, outperforming video synthesis baselines by +12.5 points (Galoaa et al., 11 Dec 2025).
• Trajectory Accuracy: Standard measures include Average Displacement Error (ADE), Final Displacement Error (FDE), Jaccard Overlap (AJ), Occlusion Accuracy (OA), and Dynamic Time Warping (DTW) on recognition or motion forecasting tasks (Galoaa et al., 11 Dec 2025, Bae et al., 2024, Yow et al., 2024).
• Semantic Alignment and Coverage: CLIP similarity scores, global/local CLIP-T, CLaTr-Score (contrastive language-trajectory) feature distances, and joint latent FID/PRDC (Galoaa et al., 11 Dec 2025, Zhang et al., 16 Oct 2025, Courant et al., 2024).
• Ablation and Generalization Studies: Loss component ablations (e.g., removing CLIP losses in Lang2Motion causes Recall@1 to drop to 8.4%), multi-domain zero-shot transfer (e.g., NTU RGB+D action recognition accuracy 88.3%, kinetic skeletons 41.6%) (Galoaa et al., 11 Dec 2025).
• Qualitative User Studies: Manual validation of generated paths/captions (Asano et al., 2024), as well as domain-specific acceptability (e.g., robot manipulation, safe RL violations) (Yow et al., 2024, Dong et al., 2024).

Task-specific datasets include MeViS (general object motion/caption pairs), HumanML3D and KIT-ML (motion/language), E.T. (camera-trajectory with text in film), pedestrian datasets (ETH/UCY, SDD, GCS), and domain-generated synthetic tool-use traces (Galoaa et al., 11 Dec 2025, Qian et al., 30 Dec 2025, Courant et al., 2024, Bae et al., 2024, Xu et al., 15 Jan 2026).

5. Applications and Modalities

The text-to-trajectory paradigm underpins a broad range of practical domains:

• Robotics and Manipulation: Language-guided trajectory design (MMFP, ExTraCT), including both synthesis and explainable correction, supports low-data robot learning and generalizable on-the-fly policy adaptation (Lee et al., 2024, Yow et al., 2024).
• Video and Animation: Explicit control of arbitrary object or camera motion enables precise, semantically-grounded animation, as in DIRECTOR and E.T., which map script-level descriptions into camera/subject paths for virtual cinematography (Courant et al., 2024).
• Human Modeling and Tracking: Generative modeling and recognition for human action, including pose, gesture, and style transfer, leveraging shared manifolds and latent editing (Lang2Motion, LMR) (Galoaa et al., 11 Dec 2025, Qian et al., 30 Dec 2025).
• Autonomous Driving and Traffic: Joint text and scene representations drive interpretable trajectory prediction for agents or ego-vehicles (e.g., using DistilBERT text encoders combined with BEV image backbones), improving upon classical regression (Keysan et al., 2023).
• Spatial Audio and Sound Synthesis: Explicit prediction of 3D locomotor paths for moving sound sources from text descriptions, enabling spatial audio synthesis and fine-grained auditory scene simulation (Liu et al., 26 Sep 2025).
• Procedure Induction, Planning, and Tool-Use: GEM synthesizes API schemas and multi-turn tool-use trajectories directly from narrative corpora, supporting data generation and training for agentic LLMs in multi-step domains (Xu et al., 15 Jan 2026).
• Safe RL and Constraint Reasoning: Natural-language constraints are translated into trajectory-level cost signals and decomposed to support safe exploration and constraint satisfaction in RL, as with TTCT (Dong et al., 2024).

6. Limitations and Future Directions

Reported challenges and open problems include:

• Sequence Length and Physical Fidelity: Trajectory generation is typically limited to fixed horizons (e.g., $T=30$ in Lang2Motion), and relies on tracking accuracy and data bandwidth (Galoaa et al., 11 Dec 2025).
• Generalization and Extrapolation: Most methods excel in interpolation or within-distribution transfer, with diminished ability to extrapolate physically or semantically beyond observed training data (e.g., unobserved motion directions in MMFP) (Lee et al., 2024).
• Compositionality and Multi-Agent Control: Few current paradigms handle multi-agent or multi-object scenes concurrently; incorporation of richer relational reasoning, user-in-the-loop editing, and complex compositional prompts (as in TGT) is an area of active work (Zhang et al., 16 Oct 2025).
• Constraint and Temporal Reasoning: TTCT and related RL approaches highlight the complexity of accurately decomposing trajectory-level, non-Markovian, and relational constraints from sentence meaning (Dong et al., 2024).
• Modal and Multimodal Integration: Despite progress in fusing textual, visual, and other modalities (audio, map), cross-modal scaling and alignment remain sensitive to prompt engineering, tokenizer design, and upstream captioning quality (noted in LMTraj, Text2Move) (Bae et al., 2024, Liu et al., 26 Sep 2025).
• Physics and Causality: Many paradigms are data-driven, lacking explicit physical priors or causal modeling of dynamical environments; future research may blend learned models with mechanistic or physics-based priors to improve robustness and transfer (Lee et al., 2024).

7. Experimental Impact and Cross-Domain Generalization

Empirical studies consistently highlight the superiority of text-to-trajectory models in semantic alignment, controllability, and transfer:

• Lang2Motion achieves faster sub-second inference and up to +33–52% improvement in physical trajectory accuracy versus state-of-the-art video baselines (ADE = 12.4 vs. 18.3–25.3), while supporting robust latent editing, style transfer, and semantic interpolation (Galoaa et al., 11 Dec 2025).
• LMR reduces HumanML3D FID by 71% over prior discrete models and enhances R-Precision, with user studies preferring the generated motions even against ground-truth 40% of the time (Qian et al., 30 Dec 2025).
• MMFP attains lowest MMD (≈0.007 for level-3 granularity) and highest classification accuracy (>99%) in low-data language-to-trajectory settings, outperforming both diffusion and VAE baselines (Lee et al., 2024).
• LMTraj’s QA/tokenization approach achieves lower ADE/FDE than numerical regressors (0.22 m/0.32 m in stochastic mode), and is effective in zero-shot (Bae et al., 2024).
• In RL, TTCT-trained agents meet task constraints with 2–4× lower violation rates than cost-based baselines, and show zero-shot transfer to new constraint-shifted environments, with violation-prediction ROC AUC of 0.98 (Dong et al., 2024).

Text-to-trajectory frameworks thus establish new state-of-the-art across diverse tasks, demonstrating broad utility and defining a blueprint for future systems that integrate human language with explicit, actionable plans and motions.