ViSA: 3D-Aware Real-Time Video Shading

• The paper introduces ViSA, a framework that integrates explicit geometric modeling, neural rendering, and transformer-based temporal smoothing to achieve photorealistic video relighting and avatar synthesis.
• It employs dual encoders and 3D-aware tri-plane as well as 3D Gaussian splatting techniques to ensure temporally consistent outputs and accurate view and lighting control.
• Quantitative evaluations show ViSA’s superiority in reduced lighting error, improved stability, and accelerated real-time performance, making it impactful for VR, telepresence, and gaming applications.

ViSA (Video Shading Architecture) encompasses a family of real-time, 3D-aware systems for photorealistic video relighting and avatar synthesis. These frameworks directly address traditional limitations in video-based editing and avatar generation—such as slow inference, lack of view/lighting control, texture artifacts, and motion discontinuities—by uniting explicit geometric modeling, neural rendering, and temporally consistent generative models. Recent implementations span portrait video relighting based on tri-plane NeRF variant architectures (Cai et al., 2024) and 3D-aware avatar creation with autoregressive video diffusion guided by 3D Gaussian splatting (Yang et al., 8 Dec 2025).

1. System Architectures and Pipelines

Two recent ViSA systems share the core requirement of complete 3D awareness, but differ in their target domain and architectural details:

A. Portrait Video Relighting Pipeline (Cai et al., 2024):

• Dual-Encoder Backbone: Each video frame $F_i$ is processed by an Albedo Encoder $E_A$ and a Shading Encoder $E_S$.
  • $E_A$ predicts an albedo tri-plane $T_{A_i}$ (three 32×32 feature planes, 256 channels) using DeepLabV3 ResNet-50, CNN, and ViT blocks.
  • $E_S$ predicts a shading tri-plane $T_{S_i}$ conditioned on $T_{A_i}$ and a lighting code $L$ (9D spherical harmonic coefficients) via StyleGAN2-based CNN.
• Temporal Consistency Network (TCN): Two 4-layer transformer branches (albedo and shading) receive a window of prior tri-planes, perform self/cross-attention, and output residuals $\Delta T_{A_i}$, $\Delta T_{S_i}$, leading to temporally smoothed tri-planes $\tilde{T}_{A_i}$, $\tilde{T}_{S_i}$.
• NeRF-Style Volumetric Rendering: The tri-planes condition NeRF-integration to produce photorealistic RGB outputs under arbitrary view and lighting.
• Super-Resolution: A StyleGAN2 head (from EG3D) upsamples to $512^2$.

B. Upper-Body Avatar Creation Pipeline (Yang et al., 8 Dec 2025):

• Stage 1: One-Shot 3D Gaussian Reconstruction.
  • Inputs: A single reference image $I_{ref}$.
  • Feature Extraction: Semantic features via frozen DINOv2, low-level visual features from hierarchical VAE encodings, and learnable human shape priors (per-vertex embeddings for SMPL-X).
  • 3D Lifting: Features are assembled per-vertex, passed through a 5-layer transformer to predict 3D Gaussian splat attributes (position offset, scale, quaternion, color SH coefficients, opacity, and a dense 3D feature vector).
  • Rendering: Standard 3DGS renderer to produce $I_{ren}$ and feature maps $F_{cond}(t)$ for animation.
• Stage 2: Real-Time Autoregressive Video Diffusion Shader.
  • Static Conditioning: Reference image latent embedding, precomputed and reused via attention KV caches, ensures persistent identity.
  • Dynamic Conditioning: At each frame $t$, $F_{cond}(t)$ from the animated 3DGS model is concatenated with the diffusion noise, guiding denoising.
  • Autoregressive Rollout: Each frame is generated by a distilled, causal transformer; output latents are autoregressively appended for temporal modeling.

2. Mathematical Foundations and Rendering Models

A. Tri-plane Neural Rendering (Cai et al., 2024):

• Classic NeRF Volume Rendering:
  • For any camera ray $r(t) = o + t d$,

  $$C(r) = \int_{t_n}^{t_f} T(t)\, \sigma(x(t))\, c(x(t), d) \; dt$$

  where $T(t)$ is accumulated transmittance, $\sigma(x)$ is volume density, and $c(x, d)$ is the view-dependent radiance. - Radiance factorization:

  $c(x, d; L) = A(x) \odot S(x; L)$

  where $A(x)$ and $S(x; L)$ are trilinear-sampled values from the smoothed albedo/shading tri-planes.

• Lighting Representation: Spherical harmonics coefficients $L \in \mathbb{R}^9$, supporting expressive low-frequency global illumination over Lambertian surfaces and enabling both cast/soft shadow effects.

B. 3D Avatar Gaussian Splatting (Yang et al., 8 Dec 2025):

• SMPL-X as Canonical Prior: Per-vertex tokens $T_i$ are constructed by combining image-sampled semantic/visual features and learned priors, projected from canonical 3D to image space.

• Gaussian Splat Attributes Prediction: Tokens are transformed to predict detailed per-vertex attributes: 3D offset, scale, quaternion, multi-band SH color, opacity, and dense feature descriptors.

• Feature-Based Conditioning for Diffusion: Dense 3D features $F_{cond}(t)$ are channel-wise concatenated with diffusion model latents, aligning generated frames tightly to the geometric and appearance priors.

3. Temporal Consistency and Conditioning Strategies

A. Transformer Temporal Smoothing (Cai et al., 2024):

• The TCN applies multi-head self-attention on the sequence of tri-planes, plus cross-attention between albedo and shading. It outputs corrections $\Delta T$, which are added to smooth per-frame predictions, mitigating flicker and enforcing inter-frame coherence.

• Losses combine reconstruction (RGB, albedo, shading), short-term and long-term temporal losses (using LPIPS in warped space, occlusion downweighting), and adversarial losses.

B. Identity and Temporal Conditioning (Yang et al., 8 Dec 2025):

• Static conditioning leverages precomputed KV caches for all attention layers from $I_{ref}$, with shifted Rotary PE for spatial alignment through pose deformations.

• Dynamic 3D feature conditioning directly injects per-frame 3DGS features, outperforming both sparse keypoint and rendered RGB conditioning.

• Temporal context is preserved by caching histories in the autoregressive transformer, with explicit self-rollout during training to prevent exposure bias.

4. Quantitative Performance and Comparative Evaluation

ViSA is benchmarked against both single-image relighting, optimization-driven avatar fitting, and direct video synthesis approaches:

Method	Lighting Error (LE) ↓	Instability (LI) ↓	ID ↑	LPIPS Flicker ↓	Time (s) ↓
B-DPR	0.9093	0.3041	0.5222	0.1015	200
B-SMFR	1.0929	0.3352	0.4479	0.0626	200
B-E4E	0.6384	0.1963	0.2892	0.0306	0.2
B-PTI	0.8220	0.2630	0.4728	0.1080	30
ViSA	0.7710	0.2533	0.5396	0.0159	0.03

• Real-Time Relighting: Achieves $\approx 33$ fps on RTX 4090 (Cai et al., 2024).

• Reconstruction Quality: LPIPS=0.240, DISTS=0.128, pose=0.036, ID=0.702 (comparable to optimization-based approaches, but at real-time speed).

• Avatar Creation: PSNR/SSIM/LPIPS/İPS-self for self-reenactment: 22.1/0.87/0.043/0.037, outperforming GUAVA (18.6/0.86/0.072/0.040) and Champ, with qualitative improvements in texture fidelity and temporal coherence (Yang et al., 8 Dec 2025).

• Autoregressive Inference: Real-time performance at 15 fps (A100), with significant latency reduction when using feature (vs. RGB) conditioning.

5. Implementation Considerations and Engineering Strategies

• Tri-plane Factorization: 3×32×32, 256 channels. Efficient trilinear sampling via custom CUDA kernels; 96 stratified points per ray in NeRF integration.

• Encoder Architectures: Albedo uses ResNet-50 (ImageNet-pretrained) and ViT (12 layers, 768 dims); Shading uses 5 conv + 4 StyleGAN2-modulated layers.

• Diffusion Model: Small, causal transformer, KD-distilled for few-step per-frame denoising.

• Super-resolution: StyleGAN2 EG3D head up to $512^2$.

• Temporal Transformers: 4 layers, 512-dim, 8 heads per branch.

• Training: Multi-stage for both modules, with 32M iterations on 8×V100 (Cai et al., 2024) or 32×H20 GPUs for ~5 days (Yang et al., 8 Dec 2025). Separate schedules for encoders and transformers.

• Optimization: Feature injection for diffusion and eliminating redundant VAE encodes enables a 34% faster inference (Yang et al., 8 Dec 2025).

6. Limitations and Future Directions

• Failure Modes:

  • Extreme occlusion or rare poses can lead to structure artifacts or discontinuities.
  • Incomplete training data for complex lighting (hard shadows, dramatic environment illumination) yields inconsistent or inaccurate shading.
  • Fine hair-body interactions (e.g., loose, moving hair) challenge the representation capacity of both 3DGS and tri-plane architectures.
• Scalability: Current avatar pipelines primarily target upper-body; full-body modeling, environmental integration, and end-to-end audio-driven animation are open problems.
• Performance Frontiers: Further compression of the auto-regressive model or adoption of 1-step consistency distillation are proposed to push beyond 15 fps real-time synthesis.
• Lighting Generalization: Incorporation of HDR or learnable environment-light priors is an active research direction to address pervasive relighting artifacts.

7. Broader Context and Comparative Analysis

The ViSA paradigm represents a convergence of explicit geometric priors (e.g., SMPL-X, 3DGS, tri-planes) and modern neural field/diffusion strategies, setting new state-of-the-art for real-time, high-fidelity, temporally stable video shading. By combining feed-forward encoders, spherical harmonics-based lighting, and transformer-based temporal networks (Cai et al., 2024), as well as integrating efficient 3D feature conditioning in autoregressive generative chains (Yang et al., 8 Dec 2025), ViSA offers a robust alternative to both computation-heavy optimization and temporally unstable direct video generation. This platform has significant implications for virtual reality, gaming, and telepresence, with future extensions toward more comprehensive avatarization—including legs, environment-aware relighting, and naturalistic, audio-driven animation (Yang et al., 8 Dec 2025, Cai et al., 2024).