JacobianAvatar: Temporally Consistent Semi-rigid Avatar Reconstruction from a Monocular Video

Abstract: Generating realistic human avatars in complex motions--such as clothing dynamics--requires modeling of global and local deformations which remains challenging in monocular settings. We address this problem by leveraging neural Jacobian fields (NJFs) for representing semi-rigid deformations. We train self-supervised neural networks for predicting Jacobian matrices that give the pose-dependent deformations, by solving a Poisson equation. However, monocular input presents several difficulties such as self-occluded regions and invisible surfaces. To address these issues, we introduce three key components: a constrained Poisson solver, signed distance-based Jacobian regularization, and a deformation-guided residual flow loss, which together suppress boundary artifacts, recover frequently occluded regions such as armpits and thighs, and enforce temporal consistency during motion. Experiments on benchmark and in-the-wild videos demonstrate that our method generates temporally stable and geometrically coherent avatars, outperforming state-of-the-art approaches.

• The paper introduces hierarchical neural Jacobian fields for modeling pose-dependent deformations, enabling temporally consistent semi-rigid avatar reconstruction.
• It employs a hybrid explicit-implicit representation that decouples texture and geometry, achieving photorealistic rendering and robust performance under occlusions.
• Quantitative results on datasets like SynWild and MonoPerfCap demonstrate superior Chamfer Distance, PSNR, SSIM, and LPIPS metrics compared to state-of-the-art methods.

Temporally Consistent Semi-rigid Avatar Reconstruction via Neural Jacobian Fields

Introduction

The paper "JacobianAvatar: Temporally Consistent Semi-rigid Avatar Reconstruction from a Monocular Video" (2606.31115) presents a novel framework for reconstructing animatable human avatars exhibiting semi-rigid deformations from monocular video input. The central advancement lies in explicitly modeling pose-dependent and temporally coherent non-rigid dynamics, such as clothing wrinkles, through hierarchical neural Jacobian fields. The approach leverages a hybrid mesh-plus-3D Gaussian Splatting (3DGS) representation to produce high-fidelity geometry and photorealistic rendering, robustly handling frequent self-occlusions and sparse observations inherent in monocular setups.

Methodological Contributions

The JacobianAvatar pipeline consists of several technical innovations:

• Canonical Mesh Refinement: The system initializes human geometry using a template mesh, refined via differentiable rendering guided by normal maps and segmentation masks. This process aligns the canonical mesh and pose parameters for optimal downstream deformation modeling.
• Hierarchical Neural Jacobian Fields (NJF): Deformation gradients conditioned on pose are learned per face in a mesh using two complementary NJFs—a coarse field for global articulation and a fine field for high-frequency localized deformations. The mapped mesh is obtained by solving a screened Poisson equation, regularized to ensure stability under partial, monocular observations.
• Hybrid Explicit-Implicit Representation: Appearance is modeled via mesh-anchored neural networks for normal-conditioned color, augmented by a mesh-embedded 3DGS module for photorealistic view synthesis. Texture and geometry are decoupled to avoid texture-copying artifacts common in prior NeRF/SDF-based methods.
• Geometric and Temporal Constraints:
  • Constrained Poisson Solver: Incorporates screening terms to preserve vertex positions for invisible/occluded regions, minimizing large artifacts specific to monocular inputs.
  • Signed Distance-based Jacobian Regularization: Employs canonical SDF priors to enforce surface normal consistency and suppress global shape distortion.
  • Deformation-guided Residual Flow Loss: Temporal correspondence is enforced by minimizing the residual between projected vertex flows and dense optical flow estimations, ensuring sub-pixel temporal stability across sequences.

Qualitative Analysis

The qualitative evaluation demonstrates superior geometry recovery and rendering fidelity relative to SOTA explicit (ExAvatar, GoMAvatar) and implicit (Vid2Avatar, LSAvatar, FacAvatar) avatar methods.

Figure 1: Qualitative comparison of normal maps; neural Jacobian field modeling recovers fine wrinkles and avoids texture-copying artifacts observed in SDF-based competitors.

Figure 2: Mesh reconstruction comparison with SDF baselines; JacobianAvatar yields cleaner surfaces free of Marching Cubes extraction artifacts.

Figure 3: Rendering quality comparison on MonoperfCap; sharp textures and accurate geometry are maintained in unseen views, especially for facial regions and local clothing details.

The ablation study elucidates the contribution of each architectural component, showing that removal of the coarse-to-fine hierarchy, screened Poisson regularization, or SDF-based prior leads to degradation in mesh quality, temporal consistency, or susceptibility to view-specific overfit.

Figure 4: Ablation analysis on SynWild; each loss or module independently affects geometry fidelity and artifact suppression.

Numerical Results and Claims

JacobianAvatar exhibits strong numeric results, frequently outperforming established baselines in geometric accuracy and rendering metrics. Notably:

• On the SynWild dataset, the method achieves a Chamfer Distance of 2.46 ($\times1000$), normal error of 0.091, and F1 scores at 1cm and 2cm of 0.397 and 0.681, respectively, exceeding both explicit and implicit SDF and 3DGS methods on all measures.
• For rendering quality on MonoPerfCap, PSNR reaches 31.83, SSIM attains 0.978, and LPIPS drops to 1.67, consistently surpassing competitors.
• The framework maintains robust performance in the presence of severe occlusions, sparse views, and temporal discontinuity, substantiating the bold claim that explicit mesh deformation via hierarchical NJFs yields state-of-the-art semi-rigid geometry and temporally stable avatars from monocular input.

Implications and Future Perspectives

Practically, JacobianAvatar advances the generation of realistic avatars for telepresence, VR, and content creation, particularly in scenarios constrained by limited, in-the-wild video. The explicit mesh deformation approach enables topology preservation and artifact suppression crucial for downstream animation and editing pipelines. Theoretically, the decoupling of geometry and texture, combined with geometric priors, signals a paradigm shift away from pure implicit field-based solutions toward hybrid, geometrically grounded avatar modeling.

The limitation regarding loose clothing and facial expression modeling—stemming from reliance on template LBS weights and naked-body priors—suggests avenues for integration with garment-aware skinning, physics-based simulation, or facial MLP modules. Future research should explore context-conditioned deformation fields, cross-modal neural priors (e.g., diffusion-based geometry completion), and scalable optimization frameworks that generalize across diverse clothing topologies and complex dynamic scenes.

Conclusion

JacobianAvatar introduces a robust pipeline for temporally consistent, high-fidelity semi-rigid avatar reconstruction from monocular video by leveraging hierarchical neural Jacobian fields, constrained optimization, and geometric regularization. Empirical results highlight substantial improvements in both geometric and appearance accuracy, supporting the claim that explicit NJF-based deformations are superior for non-rigid human modeling in sparse-view settings. This framework establishes a new technical benchmark for monocular avatar reconstruction and invites further pursuit of topologically flexible, context-aware deformation paradigms in 3D human digitization and synthesis.