4D Human-Scene Reconstruction from Low-Overlap Captures

Abstract: Existing volumetric capture of dynamic human performance achieves high fidelity with dense camera arrays. However, in real-world scenarios, only a handful of low-overlap cameras are available, which degrades the output quality and leaves large areas unobserved. Recent 4D reconstruction methods have focused on low-overlap settings, yet they still produce noticeable artifacts in under-observed regions. Video diffusion models have emerged as another option, but they show geometrically inconsistent results for humans. To address these limitations, we propose StudioRecon, a pipeline that reconstructs 4D human scenes from sparse, low-overlap cameras by decoupling background and humans. We densify background supervision by synthesizing hundreds of camera-controlled novel views with a video diffusion model. We also robustly initialize deformable Gaussian humans with cross-view identity association and triangulated multi-view keypoint fitting. Finally, our recursive enhancement module with motion-adaptive consistency injection harmonizes the composed output, thereby further avoiding remaining artifacts. We achieve state-of-the-art novel view synthesis across four real-world datasets and demonstrate applications such as novel trajectory rendering and human replacement.

• The paper presents the StudioRecon pipeline that decouples static background and dynamic human reconstruction using video diffusion and SMPL priors.
• It employs a robust multi-view association achieving 97.8% identity accuracy and mitigates occlusions via hybrid 3D triangulation.
• The recursive enhancement module enforces temporal coherence, reducing warp-based errors by 23% while delivering artifact-free reconstructions.

4D Human-Scene Reconstruction from Low-Overlap Captures: A Technical Perspective

Overview and Motivation

The work entitled "4D Human-Scene Reconstruction from Low-Overlap Captures" (2607.09125) addresses a core limitation in contemporary dynamic scene capture: the reliance on dense, overlapping camera arrays for high-fidelity human-scene reconstructions. This paradigm is impractical for in-the-wild scenarios—such as home settings or sports venues—where only few, widely-separated cameras are available. The contribution is the StudioRecon pipeline, designed to achieve high-fidelity 4D reconstructions of humans and their environments, even under severe view sparsity and occlusion. The core insight is a strategic decoupling of background and dynamic human reconstruction, exploiting different priors for each: video diffusion models for static backgrounds, and parametric 3D body models (SMPL) for humans.

Figure 1: Overview of the StudioRecon pipeline composed of sparse-to-dense view synthesis, multi-view human pose estimation, decoupled background/human Gaussian reconstruction, and recursive enhancement.

Pipeline Architecture and Key Components

The StudioRecon method is partitioned into four fundamental stages:

1. Sparse-to-Dense View Synthesis: Given a handful of widely-separated cameras, the pipeline first employs a camera-controlled video diffusion model (GEN3C) to synthesize dense novel viewpoints, greatly densifying background supervision. Static scene content is thus extended well beyond available observations.
2. Multi-View Human Pose Estimation: Independently in each view, humans are detected and tracked via a monocular pose estimator (CoMotion). Cross-view identity correspondence is robustly solved using a hybrid affinity metric combining 3D spatial proximity and pose similarity, followed by multi-view triangulation for joint SMPL parameter recovery and scale alignment.
3. Decoupled Gaussian Representation Optimization: Background and human components are instantiated as separate sets of 3D Gaussians. Background Gaussians are initialized from point clouds derived from thresholded segmentation and depth, and supervised using only the synthesized static views (excluding masked human regions). Actor Gaussians are initialized from the canonical SMPL mesh and temporally deformed via LBS, with dynamic attributes learned from all input videos.
4. Recursive Enhancement Module: To harmonize composited background and human renderings and remove residual artifacts or geometric inconsistencies, a single-step 2D diffusion model (Difix3D+) is applied framewise. A motion-adaptive consistency injection mechanism—blending warped enhanced outputs from prior frames using per-pixel EMA with flow-based occlusion/confidence—ensures temporal coherence, mitigating flicker.

   Figure 2: Overview of the recursive enhancement module, which blends previous enhanced outputs via optical flow using a motion-adaptive per-pixel EMA before single-step diffusion refinement.

Technical Contributions

Decoupled Priors for Reconstruction

Static backgrounds, due to their extended visibility but sparse direct supervision, are best modeled using the priors baked into large-scale generative video diffusion models. In contrast, human geometry—highly articulated, occlusion-prone, and temporally correlated—is most appropriately regularized using parametric body models. Empirical results demonstrate that joint optimization (entangling backgrounds and humans) introduces artifacts under sparse supervision, which are systematically avoided via decoupling.

Cross-View Human Association

Key to robust multi-person capture in severely occluded, low-overlap settings is the hybrid cross-view identity association. This approach outperforms pure appearance or pose matching, yielding 97.8% accuracy across diverse, challenging scenes with zero cross-identity confusion.

Figure 3: Cross-view identity association illustrated over four low-overlap camera views; color-matched boxes indicate successful person matching.

Recursively Coherent Enhancement

The integration of motion-adaptive consistency injection with single-step diffusion enhancement delivers temporally stable, artifact-free results. This mechanism significantly reduces frame-to-frame drift (warp-based error drops by 23%) while contributing to improved human perceptual experience.

Figure 4: Schematic of the motion-adaptive consistency injection: optical flow is computed via RAFT to warp previous outputs, which blend with current input before diffusion-based refinement.

Empirical Results

Quantitative benchmarks across four challenging real-world datasets (EgoHumans, Harmony4D, Mobile Stage, SelfCap) consistently place StudioRecon at the apex of all major metrics (PSNR, SSIM, LPIPS), with the most substantial gains in perceptual quality (LPIPS). In both 360° and 180° test layouts, the pipeline substantially outperforms dense 4DGS, MonoFusion, and SpacetimeGS baselines—with improvements up to 5dB PSNR and over 50% lower LPIPS in many cases.

Figure 5: Qualitative comparison on challenging 360° scenes demonstrating superior background sharpness and human reconstruction quality for StudioRecon.

Further, the enhancement module, when applied to competitive baselines, yields consistent improvements in perceptual fidelity, though these baselines fall short of the pipeline’s overall performance due to geometric inaccuracies that cannot be recovered post hoc.

Figure 6: Additional qualitative comparison: sharper foreground/background separation and detail preservation across activities.

(A)blation studies further confirm submodule necessity: removing cross-view association, diffusion-based dense view supervision, or recursive enhancement each yields measurable performance drops and increased artifacting.

Limitations

Despite strong results, reconstruction of high-frequency facial and hand details remains suboptimal due to fundamental observational limits. Objects held by actors are systematically unmodeled, as the pipeline operates exclusively over SMPL-defined geometry. Shadows composited into the static background remain temporally static—an artifact of the snapshot-based background modeling—resulting in physically incorrect geometry-light interactions over time.

Figure 7: Limitations: dynamic objects (left, e.g. basketball) are absent from the reconstruction (right) since they are not part of the body model.

Implications and Future Directions

The implications of this decoupled, prior-driven 4D human-scene modeling approach are multifold. Practically, it enables high-fidelity, photorealistic human-scene free-viewpoint video from minimal capture infrastructure—transformative for sports analytics, telepresence, and consumer volumetric media creation. Theoretically, the work establishes a foundation for further integration of specialized priors (beyond generic diffusion or parametric body models), such as articulated object tracking or scene understanding modules for dynamic object and shadow modeling. Rapid progress in controllable video diffusion and articulated human modeling suggests that the generalized version of this pipeline could become the de facto solution for sparse-view 4D capture.

Conclusion

"4D Human-Scene Reconstruction from Low-Overlap Captures" delivers a rigorously engineered, empirically validated pipeline that unlocks reliable, high-fidelity reconstruction in scenarios previously deemed under-constrained. By decoupling background and human models, leveraging video diffusion and SMPL-based priors, and enforcing temporal coherence through motion-adaptive consistency injection, the approach advances the practical frontier of in-the-wild human-scene capture. The remaining limitations—dynamic object modeling and shadow handling—suggest immediate directions for research in hybrid generative geometric modeling.