View-Invariant Latent Action (VILA)

• VILA is a view-agnostic latent representation that disentangles genuine action dynamics from view-specific noise using self-supervised, contrastive, and adversarial methods.
• It employs diverse strategies such as BiGRU encoders, inverse/forward dynamics, and NeRF-based modeling to ensure consistent, cross-view feature extraction.
• Empirical benchmarks show that VILA significantly enhances action recognition, robotic control, and 3D articulation performance across multiple datasets.

A View-Invariant Latent Action (VILA) is a latent representation of an action or motion that is independent of the observing viewpoint and encodes dynamics or state information robustly across changing camera perspectives. VILA has become a central concept in action recognition, robotic control, articulated 3D modeling, and self-supervised representation learning, with formalizations and architectures that span self-supervised contrastive learning, adversarial domain adaptation, LTI system characterization, motion retargeting, and NeRF-based 3D modeling. Recent frameworks employ VILA both as the objective—ensuring representations are consistent across views—and as an operational bottleneck for extracting invariant, actionable, and transferable knowledge.

1. Formal Definitions and Motivating Problem

The notion of a View-Invariant Latent Action originated from the need to disentangle genuine action dynamics from their view-specific observations. In skeleton-based action recognition, for example, each clip $\hat X_i^u \in \mathbb{R}^{T\times 3N}$ (scene $i$, view $u$) undergoes transformations—such as a rigid alignment followed by sequence encoding—to produce $z_i^u$, a latent feature meant to satisfy:

• All $u$ for a given scene $i$: $z_i^u \approx z_i^v$ for all $v \neq u$
• For different scenes $i \neq j$: $z_i^u \not\approx z_j^v$

This is achieved through alignment, contrastive learning, and clustering mechanisms such that intra-action variance attributable solely to viewpoint is minimized, while inter-action discrimination is preserved or enhanced (Men et al., 2023).

In control or reinforcement learning, a VILA-type latent $z_t^v$ summarizes the scene transition $(o_t^v, o_{t+k}^v)$ and is derived so that equivalent physical actions executed from distinct viewpoints yield (by model construction and loss terms) the same $z_t^v$, enforcing a dynamics-grounded, view-independent representation (Jeong et al., 6 Jan 2026).

2. Architectural Strategies and Mathematical Frameworks

VILA is instantiated via several architectural paradigms:

• Self-supervised skeleton action encoding: 3-layer BiGRU encoders, FC projection heads, and temporal coarse alignment generate $z_i^u$ that undergo view-alignment and contrastive regularization (Men et al., 2023).
• Contrastive binarized code extraction (LTI-theoretic): Dynamics-based Invariant Representation (DIR) streams leverage pole estimation of LTI systems, followed by sparse coding and binarization for view-invariant code extraction; optionally fused with RGB-based context streams (Zhang et al., 2023).
• Inverse/forward dynamics models: Vision encoders $E$ convert multi-view observations to state features, with an Inverse Dynamics Model mapping pairs ($s_t^v, s_{t+k}^v$) to compact latent actions $z_t^v$; a Forward Dynamics Model then predicts future features, yielding a latent space robust to appearance variation but sensitive to action dynamics (Jeong et al., 6 Jan 2026).
• View adversarial autoencoders: Gradient Reversal Layers and view-predictor discriminators drive the latent representations to be maximally unpredictable by viewpoint while retaining motion discriminability (Li et al., 2018).
• Orthogonal decomposition and motion retargeting: Latent spaces are factorized into "Character" (view/subject) and "Motion" (action dynamics) via projection onto orthogonal dictionaries; swapping and retargeting enforce the invariance of the motion code (Yang et al., 2022).
• Hypernetwork-driven NeRFs: A learnable code $z_s$ per state modulates implicit neural radiance fields for articulated 3D modeling, with the code shared across all camera views for a given state (Swaminathan et al., 2024).

3. Objective Functions and Losses for View Invariance

Central to the formulation of VILA are loss functions that penalize viewpoint-specificity and encourage dynamic/action consistency:

• Contrastive Losses: InfoNCE-style objectives maximize the similarity of latent pairs from the same action across views versus other actions, typically with additional focalization:

$$L_{fc}(X_i^u) = -\sum_{v\neq u} \left[w_+ \log S(X_i^u, X_i^v) - w_- \log \sum_{j\neq i} S(X_i^u, X_j^u)+S(X_i^u, X_j^v)\right]$$

where $w_+$, $w_-$ focus weight on hard positives/negatives (Men et al., 2023).

• Adversarial Losses: Gradient reversal for the encoder coupled with a softmax view classifier, enabling the encoding to suppress any view-discriminative gradient (Li et al., 2018).
• Supervised Action-Aligned Contrastive Loss:

$$\mathcal{L}_{\mathrm{W\text{-}NCE}} = -\sum_{i=1}^B\sum_{j\neq i}w_{ij}\log \frac{\exp(\mathrm{sim}(z_i, z_j)/\tau)}{\sum_\ell \exp(\mathrm{sim}(z_i, z_\ell)/\tau)}$$

with $w_{ij}$ weighting pairs with similar ground-truth action sequences (Jeong et al., 6 Jan 2026).

• Manifold and structure alignment: Losses that align cosine similarity matrices of the latent and the corresponding action label space (e.g., global geometry structure loss) (Jeong et al., 6 Jan 2026, Swaminathan et al., 2024).
• Motion retargeting and triplet losses: Ensuring that "motion" codes extracted from different skeletons or subjects encode only dynamics, not skeletal or view factors, via cycle-consistency and triplet separation (Yang et al., 2022).

4. Empirical Evidence and Benchmarks

VILA frameworks consistently achieve or surpass state-of-the-art performance on major benchmarks:

Dataset/Setting	SOTA Accuracy	VILA Variant	Reference
N-UCLA skeleton	99.9% (RGB+3D)	Contrastive DIR+CIR	(Zhang et al., 2023)
NTU-RGB+D 60	99.9% (RGB+3D)	Contrastive	(Zhang et al., 2023)
IXMAS	99.9% (cross-view)	JSRDA (shared+private + joint-sparse + subspace DA)	(Liu et al., 2018)
RoboSuite/Lift task	94.7% (unseen views)	VILA RL, finite-camera, action-aligned	(Jeong et al., 6 Jan 2026)
3D NeRF articulation	↑PSNR, ↓Chamfer vs PARIS	LEIA	(Swaminathan et al., 2024)

Ablation analyses confirm the necessity of both the view-invariant embedding enforcement (contrastive/adversarial/focalization losses) and explicit view-alignment or retargeting steps. For example, on N-UCLA, accuracy climbs from 74.4% (GRU-reconstruction only) to 88.3% (FoCoViL full), with clustering purity rising from 0.413 to 0.605 (Men et al., 2023). On real-robot tasks, VILA-based policies realize 63.3–85.0% success from unseen viewpoints, compared to 0–13.3% for conventional vision policies (Jeong et al., 6 Jan 2026).

5. Connections to Related Paradigms

VILA-based learning unifies several threads:

• Dynamical system theory: Modeling joint trajectories as impulse responses of linear time-invariant systems, whose invariant system poles are view-independent regardless of 3D-to-2D projection (Zhang et al., 2023).
• Adversarial domain adaptation: Explicit suppression of domain (here, viewpoint) information through adversarial training, as a generalization of domain-invariant feature learning (Li et al., 2018).
• Cross-modality fusion: Fusing skeleton-based VILA encodings with contextual RGB features for improved discrimination in settings with ambiguous pose inputs (Zhang et al., 2023).
• Action retargeting/transfer: Decomposing latent variables into dynamic and static factors, allowing for cross-subject, cross-view, and even cross-domain transfer in the action recognition setting (Yang et al., 2022).

In 3D scene modeling, VILA connects with hypernetwork conditioning of NeRFs, using a single code per articulation state across all viewpoints, diverging from methods that rely on part-based or motion-driven codebooks (Swaminathan et al., 2024).

6. Limitations and Prospects

Known limitations include reliance on multi-view data or ground-truth action sequences for hard alignment, dependency on accurate skeleton or pose estimation, and, in some cases, lack of modeling of scene context or object interactions (Yang et al., 2022, Jeong et al., 6 Jan 2026). Collection of multi-camera data or accurate synthetic views (as with ZeroNVS) can be non-trivial in real-world setups.

Future avenues identified by the literature include:

• Moving beyond camera-pose invariance to generic invariance (lighting, object/background, embodiment) (Jeong et al., 6 Jan 2026)
• End-to-end robot fine-tuning or sim-to-real loops
• Replacing supervised alignment with self-supervised temporal/dynamics consistency signals
• Integrating VILA approaches with explicit modeling of interaction and object context (Yang et al., 2022, Zhang et al., 2023).

7. Summary Table: Principal Approaches to VILA

Approach	Core Mechanism	Key Loss/Formulation	Reference
FoCoViL	View-aligned GRU encoding + focal contrastive	Focalized InfoNCE	(Men et al., 2023)
JSRDA	mSDA + sparse coding + DA	Shared sparse codes; coupled DA	(Liu et al., 2018)
Contrastive DIR	LTI pole extraction + binarization; SimCLR	Sparse code + contrastive	(Zhang et al., 2023)
ViA (motion retargeting)	Encoder-decomp + retarget + autoencoder	Orthogonal proj., cycle, triplet	(Yang et al., 2022)
View-adv autoencoder	Cross-view prediction + view-discriminator	Adversarial GRL, cross-view decoding	(Li et al., 2018)
VILA for RL	Inverse/forward dynamics + action-aligned InfoNCE	Prediction, action-guided contrastive	(Jeong et al., 6 Jan 2026)
LEIA for 3D articulation	State-conditioned hypernetwork NeRF	Per-state code, manifold reg., reg.	(Swaminathan et al., 2024)

Each instantiation centers the learning of a compact, action-centric code whose semantics are robust to sensor pose and, in many cases, domain or subject variability, supporting high-fidelity recognition, policy transfer, and generative modeling across the action understanding spectrum.