Real-Time Physics Simulation with Dynamic Mesh-Gaussian Reconstructions

Abstract: Integrating dynamic 3D reconstructions into physics simulation requires fixed mesh topology for efficient collision detection, but state-of-the-art methods like DG-Mesh produce varying topology optimized for geometric quality. We investigate whether topology conversion can enable physics integration while preserving reconstruction fidelity. We propose a dual-representation framework combining fixed-topology meshes for physics with Gaussian splatting for rendering, achieving 4.65$\times$ speedup over varying-topology baselines through runtime vertex buffer updates. We evaluate two conversion strategies, temporal correspondence tracking and template-based projection, against native fixed-topology methods (MaGS) on the DG-Mesh dataset. Our evaluation reveals that both conversion approaches incur 65-80% geometric degradation, producing results inferior to MaGS despite DG-Mesh's superior initial quality. This demonstrates that high-quality reconstruction and physics-compatible topology represent fundamentally distinct objectives that cannot be reconciled through post-processing. Our findings inform future development of physics-aware reconstruction methods and our framework enables real-time simulation with any fixed-topology approach.

• The paper introduces a dual-representation framework that decouples high-quality rendering from physics simulation by converting dynamic mesh topologies into fixed structures.
• The paper evaluates two conversion strategies—temporal correspondence tracking and template-based projection—demonstrating up to a 4.65× speedup with a trade-off of 65–80% geometric degradation.
• The paper highlights limitations such as cumulative drift and loss of fine detail, emphasizing the need for future physics-aware reconstruction algorithms.

Real-Time Physics Simulation with Dynamic Mesh-Gaussian Reconstructions

Introduction and Motivation

"Real-Time Physics Simulation with Dynamic Mesh-Gaussian Reconstructions" (2606.00444) addresses a persistent challenge in integrating dynamic 3D scene reconstructions with physics simulation: the incompatibility between high-fidelity, varying-topology meshes and the fixed-topology requirements of modern physics engines. Neural rendering advances such as DG-Mesh (Liu et al., 2024) and dynamic 3D Gaussian Splatting (3DGS) [kerbl3Dgaussians] achieve superior geometric quality and photorealism, but their time-varying mesh topologies hinder real-time physics integration. In contrast, native fixed-topology approaches like MaGS (Ma et al., 2024) facilitate efficient collision detection, but at the cost of reduced geometric fidelity.

This work systematically evaluates whether post-hoc topology conversion can reconcile high-quality reconstruction with physics-compatible topology, and proposes a dual-representation framework enabling efficient simulation and photorealistic rendering.

Dual-Representation Framework and Conversion Strategies

The proposed architecture decouples physics and rendering by maintaining two synchronized representations: a fixed-topology mesh ($\mathcal{M}_{\text{phys}}$) for physics, and a high-fidelity Gaussian splatting representation ($\mathcal{R}_{\text{render}}$) for visualization. This enables leveraging best-in-class rendering while preserving the simulation efficiency required by engines like Genesis [genesis2024], which depend on static vertex and connectivity structures for spatial acceleration.

To enable physics integration of varying-topology DG-Mesh outputs, two conversion strategies are introduced:

• Temporal Correspondence Tracking: Propagates template mesh vertices through time via surface projection, periodically re-anchoring to the template to mitigate drift. This approach attempts to retain local correspondence, but introduces cumulative geometric errors.
• Template-Based Projection: Projects a static template mesh onto each frame independently using barycentric interpolation. This eliminates temporal drift but often suppresses geometric detail due to fixed vertex density mismatches.

Native MaGS meshes, in comparison, maintain fixed topology by joint optimization—directly aligning mesh vertices and Gaussian primitives, subject to deformation priors (e.g., ARAP, SMPL).

Performance and Geometric Accuracy Evaluation

A direct varying-topology physics baseline demonstrates the prohibitive computational overhead associated with per-frame collision structure rebuilds: 31.8 FPS vs. 147.8 FPS for fixed-topology meshes, with a 4.65$\times$ speedup achieved via the dual-representation vertex buffer update mechanism.

Quantitative evaluation on the DG-Mesh dataset (six dynamic scenes, 200 frames each) measures geometric fidelity using Chamfer Distance (CD) and Earth Mover's Distance (EMD):

Method	CD ($\downarrow$)	EMD ($\downarrow$)
DG-Mesh (Original)	0.697	0.130
Temporal Tracking (Ours)	1.152	0.185
Template Projection (Ours)	1.254	0.200
MaGS (Native)	1.108	0.113

Both conversion strategies induce 65–80% geometric degradation (CD) relative to original DG-Mesh, yielding results inferior to MaGS despite MaGS’s lower initial quality. Temporal tracking yields a 4% higher CD and 64% higher EMD than MaGS; template projection is 13% worse in CD and 77% worse in EMD.

Qualitative results highlight visible artifacts and loss of thin structures in conversion methods.

Figure 1: Qualitative comparison at frame 20 on the horse sequence. Conversion strategies (b,c) introduce smoothing and artifacting, while MaGS (d) preserves geometric fidelity among fixed-topology approaches.

Theoretical and Practical Implications

The results establish that mesh-topology conversion is fundamentally lossy: high-fidelity reconstruction and physics-compatible topology are irreconcilable objectives when attempted via post-processing. DG-Mesh optimizes for geometric detail unconstrained by topology, while MaGS prioritizes topological stability, jointly optimizing mesh and Gaussian parameters.

Practically, the dual-representation framework with efficient buffer updates advances real-time performance and supports analysis of fixed-topology strategies in simulation. It allows decoupling rendering from physics and informs design of future reconstruction algorithms.

From a theoretical perspective, the work provides empirical evidence that mesh correspondence and topology preservation are bottlenecked by geometric complexity mismatches. Temporal tracking accumulates drift; template projection sacrifices local detail. Conversion approaches fail to inherit DG-Mesh’s quality, invalidating the assumption that post-hoc alignment can deliver both optimal visual fidelity and simulation compatibility.

Limitations and Future Directions

Limitations include:

• Conversion strategies do not handle topological events (contact, tearing).
• Drift and over-smoothing in conversion worsen with sequence length and complexity.
• Evaluation is limited to geometric metrics, omitting physics-specific quantitative evaluation.
• Framework is applied only to rigid-body simulation; soft-body and non-rigid physics remain unexplored.

Future research should focus on developing dynamic reconstruction methods that natively optimize for high-fidelity, fixed-topology meshes. This may involve explicit regularization targeting collision fidelity, adaptive mesh refinement, or joint deformation modeling. Differentiable simulation and mesh-Gaussian hybrid optimization may be integrated during training to align objectives.

Conclusion

This work demonstrates that post-hoc topology conversion of varying-topology, high-quality mesh reconstructions fundamentally degrades geometric fidelity, producing results inferior to native fixed-topology methods. The dual-representation framework achieves real-time simulation rates, enabling deployment of fixed-topology approaches in physics engines. The findings underscore the need for physics-aware reconstruction algorithms that directly optimize for mesh consistency and simulation-compatible representations, rather than relying on lossy post-processing. The proposed infrastructure provides a foundation for evaluating and integrating such methods in robotics, virtual reality, and embodied AI applications.