Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning

Abstract: Vision-as-inverse-graphics, the concept of reconstructing an image as an editable graphics program is a long-standing goal of computer vision. Yet even strong VLMs aren't able to achieve this in one-shot as they lack fine-grained spatial and physical grounding capability. Our key insight is that closing this gap requires interleaved multimodal reasoning through iterative execution and verification. Stemming from this, we present VIGA (Vision-as-Inverse-Graphic Agent) that starts from an empty world and reconstructs or edits scenes through a closed-loop write-run-render-compare-revise procedure. To support long-horizon reasoning, VIGA combines (i) a skill library that alternates generator and verifier roles and (ii) an evolving context memory that contains plans, code diffs, and render history. VIGA is task-agnostic as it doesn't require auxiliary modules, covering a wide range of tasks such as 3D reconstruction, multi-step scene editing, 4D physical interaction, and 2D document editing, etc. Empirically, we found VIGA substantially improves one-shot baselines on BlenderGym (35.32%) and SlideBench (117.17%). Moreover, VIGA is also model-agnostic as it doesn't require finetuning, enabling a unified protocol to evaluate heterogeneous foundation VLMs. To better support this protocol, we introduce BlenderBench, a challenging benchmark that stress-tests interleaved multimodal reasoning with graphics engine, where VIGA improves by 124.70%.

• The paper introduces VIGA, an innovative framework that iteratively refines scene programs through interleaved multimodal reasoning and perceptual feedback.
• It operates via a dual-agent system (Generator and Verifier) in a closed analysis-by-synthesis loop, effectively addressing spatial grounding challenges in VLMs.
• Quantitative evaluations on benchmarks like BlenderGym and SlideBench demonstrate significant performance gains in 3D editing, error recovery, and model generalization.

Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning: An Expert Summary

Conceptual Overview and Motivation

The paper introduces VIGA (Vision-as-Inverse-Graphics Agent), an agentic framework designed to reconstruct and edit scenes in graphics engines through an iterative, interleaved multimodal reasoning loop. VIGA reframes the long-held goal of “vision-as-inverse-graphics”—recovering a symbolic, editable program from visual inputs—as the problem of agentic code-and-perception. Rather than relying on single-shot program synthesis from images, VIGA alternates between generating candidate scene programs and actively verifying discrepancies via visual inspection (“write$\rightarrow$run$\rightarrow$render$\rightarrow$compare$\rightarrow$revise”). This tightly coupled loop enables fine-grained spatial and physical grounding, overcoming the inherent limitations of current VLMs which fail at precise code generation due to underspecified spatial semantics.

VIGA is architected as a two-role agent with a Generator (for program/code synthesis) and a Verifier (for perceptual inspection and feedback). Each role draws on a curated skill library, and both operate over a context memory structure that maintains plans, code diffs, and render histories to enable long-horizon reasoning while controlling context bloat. Notably, the system is model- and task-agnostic: it does not require model fine-tuning or auxiliary modules, functioning as a unified protocol that evaluates heterogeneous foundation VLMs across diverse tasks, with a particular focus on open-ended 3D/4D scene synthesis and editing.

Agentic Loop: Analysis-by-Synthesis Architecture

At its core, VIGA operates a closed analysis-by-synthesis (AbS) loop wherein each iteration proceeds as follows:

1. Generation: The Generator consults context memory to plan and propose incremental edits or complete programs for the scene. This involves semantic tool calls (e.g., make_plan, get_scene_info) orchestrated through a structured prompt interface.
2. Execution: The program is executed in the graphics engine (e.g., Blender), producing an updated render.
3. Verification: The Verifier inspects the scene using viewpoint-control and inspection tools (e.g., set_camera, investigate, set_keyframe), identifies dominant discrepancies, and formulates actionable feedback.
4. Memory Update: Plans, code diffs, and feedback are appended into context memory via a sliding window mechanism, maximizing retention of pertinent information while preventing context overload.
5. Iteration: The above cycle repeats until convergence or a stopping criterion is met.

   Figure 1: The core VIGA pipeline alternates code synthesis and multi-view visual verification, enabling closed-loop iterative scene refinement via memory and skills.

This explicit separation of generation and verification, mediated by tailored tool interfaces and evolving memory, scaffolds complex multimodal reasoning. The agent achieves task generality by eschewing domain-specific routines: all validation is performed via model-internal perceptual feedback rather than heuristic rules or external predictors.

Qualitative Capabilities Across Modalities

VIGA demonstrates broad cross-modal applicability:

• 3D Scene Reconstruction: Given a single input image, the agent constructs a matching 3D scene module with high fidelity across multiple viewpoints and compositional arrangements.

  Figure 2: VIGA reconstructs 3D scenes from single images, recovering layout, geometry, and realistic materials across viewpoints.

• 4D Physical Scene Generation: Starting with a static frame and goal description, VIGA synthesizes dynamic scenes that are physically plausible and temporally coherent, reflecting real-world physics and agent interactions.

  Figure 3: VIGA generates 4D scenes from static input, imbuing objects with plausible dynamics and motion in accordance with textual specifications.

• Iterative Scene Editing: For image-guided editing, VIGA identifies granular visual discrepancies—such as pose, camera, lighting, or content—and refines scene code in subsequent steps, enhancing visual-code grounding and scene accuracy.

  Figure 4: Example VIGA trajectories with tool-driven analysis of visual discrepancies and corresponding iterative code refinement.

This multimodal generality is accomplished with no task-specific redesign, underscoring the framework’s flexibility.

Quantitative Performance and Benchmarks

BlenderGym

Experiments on BlenderGym reveal that VIGA substantially improves over both one-shot and memory-less baselines for 3D editing tasks. Numerical highlights include:

• Photometric Loss (PL) Reduction: Improvements exceeding 67% in material editing and over 289% overall for open-source models (Qwen3-VL-8B), with notable gains for commercial models (GPT-4o) as well.
• N-CLIP Semantic Alignment: Consistent improvement across categories, indicating superior scene correspondence with targets.

These results evidence that iterative feedback and structured memory serve as effective scaffolds for VLMs that are otherwise weak in spatial reasoning.

SlideBench

On SlideBench for 2D slide generation, VIGA delivers marked improvements in execution success rates and reference-based metrics, especially for weaker models:

• Executable Code Recovery: >5x improvement in overall scores for Qwen3, demonstrating robust error handling and higher code reliability compared to one-pass methods.

BlenderBench

VIGA introduces BlenderBench, a new evaluation suite focusing on tasks beyond the reach of prior benchmarks (camera adjustment, compositional editing):

• VLM Score and PL: VIGA achieves a 2-3x improvement in composite reasoning scores, with substantial gains on long-horizon compositional tasks (Task 3).
• Robustness and Generalization: The impact of memory and iterative verification is especially dramatic for smaller VLMs (Qwen3), which show up to +251.56% VLM Score improvement on complex edits.

  Figure 5: BlenderBench advances beyond BlenderGym by evaluating camera control, multi-step and compositional editing, stress-testing agentic multimodal reasoning.

  

  Figure 6: Example of VIGA’s verification trajectory in hard 3D editing: locating and diagnosing target objects in visually complex scenes.

Theoretical and Practical Implications

VIGA advances the paradigm of vision-as-inverse-graphics by integrating agentic iteration, explicit perceptual verification, and execution-grounded symbolic modeling. The strong empirical results demonstrate that memory scaffolding, tool composition, and closed-loop AbS architectures can surmount the limitations of current foundation models in fine-grained spatial grounding. The framework functions as both a practical solution for scene program synthesis (e.g., digital twins, robot training pipelines) and a diagnostic testbed for evaluating spatial intelligence in VLMs.

The theoretical implication is the elevation of multimodal reasoning from passive image-text inference to active program synthesis and verification, where both perceptual input and action capabilities are interleaved and contextualized over long horizons.

However, challenges remain. The quality of verification is bounded by the intrinsic capabilities of base VLMs and inspection tools, which may fail on fine geometric precision or nuanced physical cues. Scaling context memory and tool orchestration will be essential as tasks grow in complexity.

Future Directions

Anticipated avenues include:

• Integration with Advanced Generative Assets: Plug-and-play with evolving tools (Meshy, Tripo, SAM-3D) for richer scene synthesis.
• Long-Horizon Memory Innovations: Development of more sophisticated memory protocols beyond sliding windows to support arbitrarily complex reasoning and history tracking.
• Robotic Embodied Agents: Transfer of AbS loop from digital scenes to real-world robotics, enabling closed-loop visual-program grounding and interaction.
• Foundation Model Training: Exploitation of execution-grounded scene programs as structured data for next-generation multimodal agent learning.

Collectively, VIGA’s analysis-by-synthesis approach is poised to drive richer, more reliable cross-modal intelligence in both artificial agents and evaluative protocols.

Conclusion

VIGA introduces an agentic, program-centric framework for vision-as-inverse-graphics, with empirical evidence that iterative AbS loops—grounded in tool-driven generation, verification, and contextual memory—dramatically enhance the spatial reasoning abilities of current VLMs. The paradigm unifies scene synthesis, editing, and multimodal verification across 2D/3D/4D tasks, laying a practical and theoretical foundation for future advances in interpretable multimodal AI and agentic visual understanding.