PhysAgent: Automating Physics-Based 4D Synthesis via Trajectory-Grounded Multi-Agent Feedback

Abstract: Achieving fully automated, physically plausible 3D motion synthesis is a core objective in graphics and generative AI. However, configuring complex environmental force fields still relies entirely on manual expert intervention, creating a severe bottleneck for large-scale simulation data generation. Existing automated methods primarily focus on material optimization and exhibit severe modality gaps and technical flaws when applied to the vastly more complex force field optimization space: naive LLMs lack underlying simulation feedback, causing severe physical inaccuracies, while traditional Score Distillation Sampling (SDS) suffers from sluggish gradients, local optima entrapment, and a mathematical inability to dynamically switch discrete force fields. To address this, we propose PhysAgent, the first simulator-in-the-loop multi-agent framework that leverages multimodal inputs for automated, physically grounded 4D synthesis. By decoupling intrinsic materials from extrinsic dynamics, PhysAgent utilizes a Semantic Agent equipped with an externalized Force Field Skill module to master simulation rules and generate valid initializations. Subsequently, the Refine Agents, driven by Trajectory-Grounded Multi-Agent Feedback, leverage vision foundation models to extract dense point trajectories from rendered frames. By converting these explicit motion trajectories into structured textual descriptors, the agent harnesses LLM commonsense reasoning to execute zero-shot macroscopic leaps, effectively escaping local optima and dynamically switching discrete force fields. Extensive experiments demonstrate that PhysAgent rapidly generates stable, diverse physical scenes from arbitrary multimodal prompts, significantly outperforming existing baselines in both generation diversity and physical accuracy.

• The paper introduces a multi-agent, simulator-in-the-loop framework that automates both object material and force field configuration.
• It leverages closed-loop optimization with Semantic and Refine Agents to enhance semantic alignment and physical fidelity.
• Empirical results demonstrate significant improvements over baselines in dynamic plausibility, perceptual consistency, and overall performance.

PhysAgent: Multi-Agent Closed-Loop Automation for Physics-Based 4D Synthesis

Motivation and Limitations of Prior Art

Automated 4D physics-based scene synthesis remains constrained by its dependency on manually configured environmental force fields, a bottleneck that impedes both scalability and physical realism in generative pipelines. Previous methods focus extensively on optimizing intrinsic object properties but either disregard the extrinsic dynamics induced by complex environmental forces or impose computationally expensive, unstable frameworks dependent on laborious manual intervention. Score Distillation Sampling (SDS)-based pipelines, while capable of learning material properties, are encumbered by slow convergence, local optima entrapment, and limited expressivity for discrete force configurations. Naive generative approaches, including LLMs without simulation-in-the-loop grounding, fail to ensure physical plausibility, as they lack any feedback from actual physical evolution. The limitations are summarized in (Figure 1).

Figure 1: Comparison of physics-based 4D synthesis paradigms, with PhysAgent achieving full-parameter automation and physical compliance via continuous visual feedback.

PhysAgent Architecture and Simulation-In-the-Loop Paradigm

PhysAgent introduces a simulator-in-the-loop, multi-agent system engineered for physically compliant 4D synthesis with full automation of both object materials and environmental force fields. The system architecture (Figure 2) explicitly decouples intrinsic and extrinsic properties: object geometries and materials are statically initialized from image inputs using feed-forward models, while dynamic force configuration is driven entirely from user text prompts. The Semantic Agent is responsible for force field initialization and is augmented by an externalized Force Field Skill Library. This module encodes the taxonomy and operational constraints (translation, scaling, impulse, torque) underpinning all supported force types, thus formally restricting action-space exploration to physically meaningful subspaces.

Figure 2: Overview of the PhysAgent closed-loop paradigm using multi-agent reasoning, simulator feedback, and iterative trajectory alignment for physical compliance.

Following initialization, the Material Point Method (MPM) simulator generates 4D dynamics, with extrinsic force fields acting at the grid phase to evolve the scene. Crucially, the rendered frames are processed by a perception pipeline comprising segmentation, depth estimation, and dense point tracking, which facilitate extraction of structured kinematic trajectories from image-space outputs.

A closed-loop optimization regime is realized through two interacting agents: a Refine Planning Agent (driven by LLM commonsense reasoning) that interprets trajectory deviations relative to the target prompt, and a Refine Execution Agent that modifies force field parameters via zero-shot, discrete-state updates. This paradigm avoids the slow, myopic updates of SDS-style approaches and can dynamically switch between force field regimes—a capability unattainable with gradient descent over non-differentiable or discrete parameters.

Numerical Results and Empirical Performance

PhysAgent demonstrates strong empirical improvement on both semantic alignment (CLIPsim) and human preference metrics compared to recent baselines. Qualitative results (Figure 3) show that, under diverse prompt-induced force fields, PhysAgent generates highly plausible and temporally smooth dynamic responses while maintaining physical fidelity across object categories.

Figure 3: Diverse physically grounded 4D dynamics generated by PhysAgent for multiple text-conditioned scenarios.

Direct qualitative comparisons (Figure 4) highlight scenarios (alternating wind, compound pulling) where baseline approaches exhibit implausible, artifact-ridden dynamics due to their inability to optimize discrete, prompt-driven environmental forces. PhysAgent consistently sustains plausible object deformation and motion.

Figure 4: Comparison of dynamic scene quality between PhysAgent and baseline methods for complex force field prompts.

Ablation studies further reveal the necessity of the Force Field Skill Library and Refine Agents. Removal of either leads to significant degradation in statistical and perceptual metrics. Notably, the closed-loop multi-agent regime demonstrably rectifies both magnitude and directional mismatches in dynamic evolution, as substantiated by before/after visualizations (Figure 5).

Figure 5: Impact of Refine Agents, showing correction of kinematic deviation via closed-loop multi-agent optimization.

Theoretical and Practical Implications

PhysAgent’s multi-agent, simulator-in-the-loop approach represents a substantive advance in bridging the simulation/semantics gap. By recasting optimization as causal alignment through hierarchical agent reasoning and iterative perception-action loops, the framework overcomes the mathematical intractability of discrete force field configuration present in gradient-based optimization.

The methodology’s explicit decoupling of material and force configuration renders it highly modular and compatible with existing material-centric generative methods, enabling unified optimization across both intrinsic and extrinsic physical domains. The use of vision foundation models for trajectory extraction sidesteps reliance on ground-truth 3D annotation, facilitating real-world deployment where only monocular or multimodal data are available.

On the practical side, the framework makes full-parameter 4D physics-based synthesis accessible for applications in embodied AI, simulation augmentation, and controllable content generation without expert physics intervention. Limitations persist in handling highly localized or multi-stage compound force interactions, and scenarios with severe occlusion or low-texture surfaces remain challenging for robust perception modules.

Future Prospects

Extensions of PhysAgent are expected to focus on increasing temporal and spatial granularity of force field mapping, augmenting perception modules to operate beyond object-centric segmentation, and further integrating agent reasoning within an end-to-end physics-vision-language foundation model. Scaling to open-world, real-time 4D synthesis remains a long-term objective necessitating advances in efficient simulation, adaptive perception, and tighter semantic grounding.

Conclusion

PhysAgent establishes a robust and efficient framework for physics-based 4D synthesis via trajectory-grounded, multi-agent closed-loop optimization (2606.08688). It automates the translation of multimodal prompts to dynamic force field configuration, achieving significant gains in physical realism and semantic alignment over prior art. By leveraging structured skill libraries, vision-based trajectory feedback, and hierarchical agentic reasoning, PhysAgent offers a scalable paradigm for fully automated, compliant 4D generative simulation.