Prop-Chromeleon: Adaptive Haptic Props in Mixed Reality through Generative Artificial Intelligence

Abstract: Mixed Reality (MR) aims to blend digital and physical worlds, but the absence of haptic feedback often breaks visual-tactile consistency. We introduce Prop-Chromeleon, a MR system based on generative AI that dynamically transforms everyday objects into adaptive passive haptic props through user-provided text prompts. Our AI pipeline performs generation and anchoring of virtual assets that align with the shape of physical props, allowing us to study how virtual content generation behaves under geometric and prompt-based constraints. We evaluate Prop-Chromeleon's effectiveness through a generation study using varied object shapes and user prompts, combining quantitative shape similarity metrics with qualitative prompt fidelity analysis. Our user study further showcases Prop-Chromeleon's improvements in perceived realism, immersion, and enjoyment compared to static baselines. These results show that shape-aware generation can support both believable haptic interaction and creative engagement in MR.

• The paper introduces Prop-Chromeleon, a generative AI pipeline that dynamically transforms physical props into geometrically aligned 3D models in MR.
• It integrates depth sensing, text-to-image diffusion, and single-view 3D reconstruction to achieve high shape fidelity and prompt-driven semantic alignment.
• User studies and technical evaluations show significant improvements in haptic realism, immersion, and interaction quality compared to static MR baselines.

Prop-Chromeleon: Generative Adaptive Passive Haptics for Mixed Reality

Introduction

The paper "Prop-Chromeleon: Adaptive Haptic Props in Mixed Reality through Generative Artificial Intelligence" (2605.00804) presents an MR platform enabling on-demand, text-prompt-driven transformation of physical objects into adaptive, shape-aligned virtual models. The approach is underpinned by a generative AI pipeline that incorporates depth sensing, text-to-image diffusion, and single-view 3D reconstruction. This system bridges digital and physical modalities by precisely overlaying and tracking generated 3D models geometrically constrained by real-world props, a paradigm shift from static 3D asset libraries and manual mapping. Through quantitative and user studies, the authors demonstrate significant improvements over conventional MR approaches relying on static baselines, with Prop-Chromeleon yielding marked gains in perceived realism, immersion, and interaction quality.

System Architecture and Pipeline

Prop-Chromeleon is implemented as a wearable stereo MR system (Quest 2 + ZED Mini) that facilitates real-time video passthrough and depth acquisition. The user provides a text prompt describing the desired transformation of a tangible prop.

The processing pipeline consists of:

1. Depth and RGB Capture: The ZED Mini provides synchronous depth and color input.
2. Depth-Guided Text-to-Image Generation: By integrating Stable Diffusion with ControlNet, depth constraints are injected into T2I diffusion, ensuring that output image geometry matches the physical prop.
3. Image-to-3D Reconstruction: The diffusion image is upsampled to a watertight mesh using TripoSR, which balances speed and 3D shape fidelity.
4. Spatial Anchoring & Alignment: Vuforia's Model Target is used for 6-DoF tracking, overlaying the generated mesh directly atop the physical object during manipulation, ensuring haptic-visual consistency during real-world interaction.

   Figure 1: The Prop-Chromeleon processing pipeline combines depth and object tracking, and integrates the user's prompt by combining Stable Diffusion and ControlNet to generate a 3D object that is overlaid and aligned with the physical prop.

User interaction is prompt-driven and tightly coupled to prop geometry: once the prompt is submitted, the system generates a 3D model that is visually aligned to the underlying object, supporting both pick-and-place and hands-on manipulation in MR.

Figure 2: Prop-Chromeleon from the user's point of view. Left: The user inputs a "Master Yoda" text prompt to transform the Paddington bear. Right: Prop-Chromeleon generates a 3D model that matches the prompt and is overlaid on the physical prop for haptic interaction with the 3D model.

Evaluation: Technical and Human-Centric

Generation Study

A large-scale technical study systematically probed the pipeline's geometric fidelity and semantic compliance. Using 50 diverse ShapeNetCore objects and human-inspired as well as neural-generated prompts (structured into [noun], [adj noun], [noun with description], and [noun performing action]), 800 prompt–object pairs were evaluated.

Prompt selection was informed by real user behavior, revealing natural prompt patterning when participants were asked to transform reference objects in various creative ways.

Figure 3: User-generated prompts from preliminary study.

Shape similarity between input proxy and generated mesh was quantified with Chamfer and Hausdorff distances, resulting in overall Chamfer = 0.322 and Hausdorff = 0.456 (normalized to the unit sphere). Outlier removal only modestly improved these metrics, indicating that the generative pipeline robustly preserves core geometry across the prompt-object space.

Visual inspection further confirms that shape deviations are minor and localized to fine detail, with generated meshes maintaining the global contour and affordances of the physical prop.

Figure 4: Comparison between generated meshes (blue) and original object shapes (white) produced by our pipeline. The Hausdorff and Chamfer distances quantify the geometric differences between the original and generated shapes.

Semantic evaluation (presence of main elements, color/texture, and theme) via triple-blind ratings yielded high inter-rater reliability (Fleiss’ κ > 0.74). Results indicate an overall prompt-fidelity rate exceeding 90% with custom prompts: inclusion of main elements = 95.5%, color/texture = 97.0%, thematic adherence = 94.3%. General, less-conditioned prompts performed substantially worse (elements: 64.5%, theme: 63.5%), highlighting the centrality of aligning prompt semantics with physical geometry for optimal outcome.

User Study

A within-subjects user study (N=12) compared Prop-Chromeleon's aligned generation against a baseline condition lacking geometric alignment (virtual model only globally scaled/rotated).

Participants interacted with a diverse set of everyday objects (e.g., toy bear, sneaker, bottle) and assessed transformations for haptic realism, immersion, and expressiveness. Example prompt–object–generation sets illustrate the range of semantic transformations and the fidelity of shape/appearance matching.

Figure 5: Prompts, original physical objects, Prop-Chromeleon generations, and baseline generations used for Task 1 in our user study.

Users manipulated physical props while perceiving the virtual transformed object, supporting haptic exploration, endpoint alignment, and affordance-driven tasks.

Figure 6: Participants interacting with different condition groups in the user study, Left: Participant moving the prop in the shoe-Robotic Dog scene, baseline condition; Middle: Participant touching the prop's hat brim in the Paddington Bear-Cute Pig scene, Prop-Chromeleon condition; Right: Participant engaging with prop in the Paddington Bear-Cute Pig scene, Prop-Chromeleon condition;

Quantitative results using the multidimensional Haptic Experience (HX) scale show statistically significant improvements across realism (R1–R3), harmony (H3, H5, I1), and involvement (H2, I2) factors for Prop-Chromeleon versus baseline (Wilcoxon signed-rank, p < .01 for most dimensions). Expressivity did not differ, reflecting the shared static/passive nature of both haptic experiences.

Figure 7: Questionnaire results for individual questions, and haptic factors. We found a significant difference for all questions and factors except E5: The haptic feedback reflects varying inputs and events'', E1:The haptic feedback all felt the same'', and PA4: ``Expressivity''.

Semi-structured interviews corroborate that haptic–visual congruence, especially at shape boundaries and affordances (e.g., touching a bear's hat), was critical for realistic and enjoyable interaction. Unexpected or playful generations (e.g., semantic misalignment or creative reinterpretations) were welcomed as part of the creative value proposition.

A second, fully open-ended creative exploration task highlights the broad generative space and reveals application potential in both playful and design-oriented authoring.

Figure 8: Examples of user-generated prompts and their corresponding generated results from Task 2.

Theoretical and Practical Implications

Prop-Chromeleon operationalizes generative passive haptics as a problem of conditional generation under dual constraints: physical geometry and user-specified semantics. This reframes MR haptics from static asset mapping/manual annotation to scalable, automated, user-driven authoring. Systematic evaluation exposes several theoretical and practical implications:

• Geometry–Semantics Trade-off: Maximum haptic realism is achieved when prompt semantics are compatible with the underlying prop geometry; more divergent prompts introduce greater expressivity but correlate with increased visual–tactile mismatch.
• Prompt Grounding: Human prompt generation is intrinsically influenced by physical form. Systems aiming for high-fidelity or believable transformations should expose, and potentially constrain, available prompt spaces to the geometric affordances of existing props.
• Scalability and Flexibility: The pipeline supports arbitrary, per-instance creative interaction with minimal manual setup, outperforming prior approaches that rely on static, curated libraries or tedious mapping.
• Safety and Perceptual Risks: Arbitrary remapping of affordances, particularly for tool-like or hazardous props, can lead to misuse. Future systems must integrate category-level safety constraints or affordance-preserving pipelines to avoid unsafe reinterpretations.

Limitations and Future Directions

Despite robust shape fidelity and high user preference, Prop-Chromeleon's current design presents several bottlenecks:

• Latency: End-to-end processing, dominated by offloaded diffusion and reconstruction inference, introduces interactional delays ($\sim$20 s). Real-time applications will require on-device acceleration or hybrid caching strategies.
• Texture/Affordance Discordances: While geometry is tightly constrained, tactile sensation mismatches (e.g., a metallic prop feeling wrong for a visually generated cactus) disrupt immersion for some users. Integrating physical property estimation (texture, thermal feedback) into prompt conditioning is an open avenue, possibly leveraging multimodal large vision-LLMs.
• Occlusion and Tracking: Visual occlusion conflicts (e.g., virtual props rendering over user hands) occasionally undermine spatial presence, suggesting the need for improved occlusion-aware rendering and multi-object/hand tracking integration.
• Complex Deformation and Multi-Object Contexts: Extension to deformable objects, dynamic retargeting, or room-scale transformations remains future work. Incorporating multi-view reconstruction or real-time SLAM could expand feasible interaction envelopes.

Conclusion

Prop-Chromeleon (2605.00804) constitutes a rigorous, technically grounded demonstration of generative AI for adaptive passive haptics in MR. Through controlled technical and user-centric investigations, the authors show that combining depth-guided generative diffusion with online 3D reconstruction and robust tracking yields significant advances over static approaches—both in quantitative geometric metrics and, more critically, in subjective haptic experience. The work frames new directions in MR research, spotlighting the delicate geometry–semantics interplay underlying effective generative haptic feedback, and underscores the promise of generative pipelines as the foundation for customizable, scalable, and creative MR interaction systems.