MiXR: Harvesting and Recomposing Geometry from Real-World Objects for In-Situ 3D Design

Abstract: Recent developments in 3D generative AI enable users to create bespoke 3D models from text or image prompts. However, these approaches provide limited control over spatial structure, making them ill suited for tasks requiring precise geometric composition. We present MiXR, an XR system for in-situ compositional modeling that enables users to create new 3D models by harvesting geometry from their environment. Users extract segments from captured objects and assemble new artifacts through direct 3D manipulation, while generative AI synthesizes a coherent model from the user-defined composition. This hybrid workflow allows users to define spatial structure explicitly while delegating geometric refinement to generative models, enabling them to specify spatial intent that is difficult to express through verbal prompts alone. In a controlled user study ($N=12$), participants using MiXR rated their designs as significantly closer to the target, felt more in control, and experienced lower cognitive workload compared to a generative composition baseline.

• The paper presents a novel hybrid AR/XR system, MiXR, that enables direct capture and segmentation of real-world 3D proxy meshes for precise spatial composition.
• It integrates a three-mode workflow (Generate, Segment, Compose) with AI-driven geometric refinement, demonstrating superior performance over prompt-based methods in accuracy and user control.
• Empirical evaluations reveal higher design realization, reduced workload, and greater spatial fidelity, despite limitations from fixed-resolution voxel grids.

MiXR: Harvesting and Recomposing Geometry from Real-World Objects for In-Situ 3D Design

Motivation and Context

MiXR introduces a novel paradigm for 3D design in XR, directly addressing the limitations of prompt-based 3D generative AI systems. Existing methods for text- or image-driven 3D generation are characterized by one-shot pipelines, minimal iterative control, and an inability to preserve granular spatial relationships essential for artifact-level fidelity. Empirical studies demonstrate that verbal prompts invariably fail to capture fine spatial intent, especially regarding part placement, orientation, and scaling. This deficiency relegates creative practice to prompt optimization, undermining user-driven compositional design.

In contrast, MiXR leverages the affordances of AR/XR by exposing spatial representation and manipulation in situ. It aligns with HCI literature on human-in-the-loop workflows that pivot away from opaque prompt-only interfaces towards intermediate artifact manipulation and iterative feedback. Unlike prior AR/XR systems, which focus on object augmentation or digital capture, MiXR enables in-situ harvesting of real-world object geometry, segment-level extraction, direct spatial recomposition, and AI-driven refinement—consolidating the entire pipeline within a unified spatial interface.

System Architecture and Workflow

MiXR is structured as a three-mode spatial workflow: Generate, Segment, and Compose. Users capture 3D proxy meshes of physical objects using SAM3D (Team et al., 20 Nov 2025) with a combined screenshot and verbal prompt. Each 3D asset is retained as a workspace item with its latent representation, facilitating subsequent compositional manipulation.

Segmentation is executed via manual paint-based selection in AR. This approach supplants automated semantic segmentation methods (e.g., SAM [Segment Anything]) in favor of direct user control, accommodating idiosyncratic design intent beyond semantic boundaries. Each segment is defined at vertex-level granularity, ensuring precise part extraction.

In Compose mode, users spatially arrange segmented parts using axis-aligned rotation, translation, and scaling gizmos optimized for direct hand interaction. This spatial assembly encodes the user's compositional intent—including part placement, relative scale, and orientation—before invoking generative AI-based geometric refinement.

Refinement utilizes the intermediate SLAT (Structured Latent) representations from SAM3D. User-defined transformations and selections are applied to SLAT voxel grids, followed by latent union and decoding. The decoder synthesizes a unified mesh, harmonizing surface detail and texture without retraining, handling gap-filling and texture integration across the composed volume.

Implementation details include WebXR deployment on Google Chrome for the Samsung Galaxy XR headset, leveraging XRBlocks (Li et al., 29 Sep 2025) for spatial controls and a cloud backend (NVIDIA A40) yielding ∼15-second latency.

Technical Evaluation

MiXR's composition pipeline is critically evaluated for robustness under systematic translation, rotation, and scaling sweeps between paired source objects from Objaverse [13142-13153]. Key findings:

• Translation: Increasing spatial separation degrades mesh IoU from a baseline of 0.78 to 0.67 and elevates Chamfer distance (practical stability within 2.5 object extents).
• Rotation: Minimal impact across full angular sweep; mesh IoU and Chamfer distance remain stable, reflecting decoder tolerance to reorientation.
• Scale: Uniform up-scaling exhibits sharp non-linear Chamfer growth and gradual mesh IoU decline, indicating practical fidelity limits around 3–4x scaling.

The architectural constraint—fixed-resolution $64^3$ voxel grid—means quality decays sharply for compositions expanding spatial support. These results set operational bounds for reliable compositional fidelity in MiXR.

User Study: Quantitative and Qualitative Results

A controlled within-subjects study (N=12) compares MiXR against a generative composition baseline (VLM-mediated Gemini [Gemini 3] + SAM3D), across abstract (building blocks) and real-world (organic objects) tasks.

Quantitative Outcomes

• MiXR is rated significantly higher on all post-task Likert items: design realization (M=6.29 vs. 3.29, p<.001), output match (M=6.08 vs. 3.08, p<.001), perceived control (M=5.92 vs. 3.04, p<.001), learnability (M=6.42 vs. 4.46, p=.002), and scalability (M=5.75 vs. 2.83, p<.001).
• NASA TLX workload scores favor MiXR: lower mental demand (M=2.38 vs. 3.67, p=.020), superior performance (M=1.96 vs. 5.00, p<.001).
• Forced-choice preferences: unanimous selection of MiXR for control, accuracy, spatial complexity handling, and confidence in matching targets.

Qualitative Themes

• Direct manipulation supports intuitive spatial reasoning, outperforming prompt-based interaction for spatial precision.
• Physical effort with MiXR is perceived as engaging, contrasted with the monotony and frustration of prompt iteration in the baseline.
• Participants report heightened attachment to artifacts composed directly, attributing ownership and authorship to the hands-on process.
• Hybrid workflows are envisioned, combining generative generation with direct spatial manipulation for refinement.

Practical Applications and Implications

MiXR enables embodied, context-aware, and semantically rich 3D artifact creation, illustrated in:

• Personalized Fabrication: Functional designs leveraging meaningful geometry (e.g., a custom book support with corgi geometry).
• Creative Reuse: Repurposing household items as compositional primitives (e.g., chair from mat and mug segments).
• Accessibility Design: In-situ ergonomic artifact creation (e.g., umbrella grip conforming to user's hand geometry).

The system facilitates an unprecedented degree of compositional control for novice users, overcoming language bottlenecks in spatial specification. Importantly, MiXR operationalizes a hybrid interaction model—leveraging language for object instantiation and direct manipulation for spatial arrangement—which is a design pattern likely to propagate across future AI-enabled XR design workflows.

Limitations and Future Directions

Key limitations include reliance on SAM3D’s single-image reconstruction quality, lack of motion/articulation and material properties in captured meshes, absence of dimensional constraint tooling for precision fabrication, and dependency on fixed-resolution voxel grids for composition. The study cohort is limited to novices; extrapolation to professional practice requires further validation.

Future research directions entail system extension to multi-model generative backends, integration of physical property modeling, dimensional constraint feedback, and support for articulated structures. Enhanced measurement and error correction tooling will further augment fidelity and utility.

Conclusion

MiXR establishes a hybrid, in-situ compositional workflow for 3D design in XR, enabling direct harvesting and recomposition of real-world geometry. Empirical findings substantiate that spatial manipulation of segments yields superior control, spatial fidelity, and satisfaction over prompt-driven composition, regardless of geometric complexity. The architectural separation of spatial arrangement (user) and geometric refinement (AI) addresses fundamental bottlenecks in intent translation and artifact alignment in 3D generative systems. These results have broad implications for the evolution of human-AI co-design in spatial computing, suggesting future tools will integrate interactive intermediate artifact manipulation as core to the creative pipeline.