HandMade: Spatial Prompting for Generative 3D Creation with Part-Labeled VR Sketches

Abstract: Text-to-3D generation lowers the barrier to 3D content creation, but text alone is a weak interface for specifying spatial intent: where parts should be placed, how they relate, and how an object should be organized in 3D. We present HandMade, a workflow that combines VR 3D sketching and language for open-domain 3D asset generation. HandMade treats coarse, part-labeled 3D sketches not as incomplete geometry to reconstruct directly, but as spatial prompts for existing generative models. It converts segmented VR strokes into multi-view part guidance and structured prompts, allowing users to specify object layout and part relationships through 3D sketching while using language for identity, material, style, and local details. A technical evaluation shows that HandMade better preserves user-authored spatial scaffolds than text-only and sketch-based baselines on 20 varied examples. A user study with eight participants characterizes how users make use of 3D sketching for spatial layout and language for identity, materials, and details across initial authoring and subsequent revision. HandMade contributes an interaction paradigm and interface-to-generation pipeline for spatially guided 3D creation.

• The paper introduces a novel framework that uses part-labeled VR sketches as spatial prompts to guide generative 3D asset creation.
• It combines multi-view rendering and structured text prompts to preserve both spatial arrangement and semantic details.
• Empirical results show that the method outperforms text-only and single-view baselines in geometric fidelity and part adherence.

HandMade: Spatial Prompting for Generative 3D Creation Using Part-Labeled VR Sketches

Motivation and Problem Formulation

HandMade addresses a critical limitation in current open-domain 3D generation pipelines: textual interfaces alone are insufficient for expressing precise spatial intent, such as relative part placement, proportions, and object organization. While generative models enable plausible 3D asset synthesis from text prompts, they cannot faithfully encode complex or imaginative object layouts formulated by designers and non-experts. Sketch-based or image-conditioned systems, on the other hand, typically require either complete, well-structured reference images or fail to interpret spatial cues from incomplete, abstract input. HandMade introduces a multimodal workflow—leveraging coarse, part-labeled VR sketches plus natural language—to bridge the communication gap between human intention and generative model conditioning.

The innovation lies in reinterpreting sparse 3D sketches not as noisy geometry to reconstruct directly, but as spatial prompts, yielding a flexible and domain-agnostic interface for open-vocabulary 3D asset generation. This modular pipeline circumvents the need for training custom sketch-to-3D models, for which large-scale paired data is lacking, by translating user-authored spatial scaffolds into visual guidance that modern image-to-3D models can exploit.

System Pipeline and Algorithmic Framework

The core pipeline stages of HandMade are as follows:

1. Part-Structured VR Sketching: Users author a rough spatial layout by drawing freehand 3D strokes in VR, segmented into parts with assigned colors, and providing a text description for both object-level semantics and part-specific attributes.
2. Multi-View Part Guidance: The segmented strokes are rendered into a set of four fixed camera views on a unit cube, forming a $2\times2$ grid of color-coded guidance images per example. These are explicitly used for part identity labeling rather than appearance, dissociating the color encoding from material semantics.
3. Structured Prompt Construction: The system concatenates an object-level specification, a color-to-part mapping, and user-written part descriptions into a structured prompt. This prompt, alongside the guidance image grid, conditions a multimodal image synthesis stage.
4. Multi-View Image Synthesis: Provisioned with both visual and textual inputs, a generative model (e.g., ChatGPT Images 2.0) produces a semantically consistent, multi-view object depiction conforming to the user's intended spatial arrangement and described appearance.
5. 3D Mesh Reconstruction: The resulting images are fused in a feed-forward image-to-3D pipeline (e.g., Hunyuan3D-2mv), yielding a textured mesh that encapsulates both the spatial constraints (from the sketch) and the semantic/compositional detail (from text).
6. Revision Mechanisms: The workflow is interactive and revision-centric, enabling users to iteratively modify spatial scaffolds, descriptions, or regenerate outputs at any stage, with explicit checkpoints for visual and mesh feedback.

   Figure 1: The HandMade pipeline: VR sketching and text input are converted into multi-view part guidance images, which condition image synthesis and subsequent mesh reconstruction for a textured 3D asset.

Empirical Analysis and Evaluation

Technical Metrics

HandMade was evaluated against six relevant baselines—including text-only, single-view ablations, and modern sketch- or image-conditioned 3D systems (e.g., Instant3dit [barda2025instant3dit], SketchDream [liu2024sketchdream], LAS-Diffusion [zheng2023lasdiffusion], MeshPad [li2025meshpad])—across 20 diverse, open-domain examples.

• Geometry Alignment: Using mean sketch$\to$mesh and mesh$\to$sketch distances, Chamfer distance, and [email protected] as primary metrics, HandMade achieved the lowest geometric discrepancy with the input spatial scaffold (sketch$\to$mesh distance $0.035$; [email protected] $0.727$), outperforming all ablations and baselines. Notably, Instant3dit was the closest in performance, but failed to preserve part semantics and cross-view relationships to the same degree.
• Ablation Insights: Removing the sketch guidance (text-only) resulted in high-fidelity objects that lacked correspondence to the user's part layout. Similarly, reducing guidance to a single view substantially degraded adherence to intended 3D structure, underscoring the necessity of sparse 3D part-guided conditioning.
• Qualitative Findings: The strongest advantage of HandMade was observed in cases where text prompts could not plausibly capture spatial complexity (e.g., eccentric layouts, custom part arrangements), with the system consistently honoring the user's scaffolded intent while leveraging generative priors for detail and style.

User Study

Eight participants engaged in the full authoring-refinement loop. The analysis revealed five central usage patterns:

• Evolving Intent: Users started with imprecise or partial intent, solidifying design ideas through early sketching and iterative generation.
• Complementary Modalities: VR sketches efficiently captured spatial layout and part relationships, while text handled identity, material, and fine-grained detail.
• Expressive Limitations: Sketching sufficed for coarse structure but was inherently noisy for local details or thin features, exacerbated by VR input constraints.
• Stage-Aware Revision: Visual checkpoints at multi-view and mesh stages allowed targeted correction—users typically refined the latest mismatched part rather than redoing the entire pipeline.
• Revision Efficacy: Lightweight edits (textual or re-generation) following initial mesh inspection quickly improved intent alignment without requiring full redrawing.

  Figure 2: User study Likert ratings: intent clarity, ease of VR drawing, and revision outcomes demonstrate moderate interface burden but effective spatial-semantic division of labor.

Analysis of Guidance Adherence and Interaction Mechanisms

Case studies highlight the nuanced interplay between spatial guidance and model priors. The system's adherence to the input sketch was tunable via prompt engineering—explicitly instructing the generative model to prioritize spatial guide yields higher fidelity to the user's arrangement, while relaxing guide adherence invites more semantic or stylistic diversity.

Figure 3: Modulation of guide adherence: initial generation (middle) accommodates plausible semantics but deviates from input geometry; an explicit instruction (right) causes the output to better follow the spatial guide (left).

This motivates future directions, such as exposing guide adherence as a user-adjustable parameter or augmenting spatial prompts with higher-level relational annotations (e.g., "attached to", "remains flat") to encode non-local geometric constraints.

Theoretical and Practical Implications

The primary theoretical contribution is the demonstration that rough, part-labeled VR sketches serve as effective high-level spatial prompts for general-purpose, open-domain 3D generative models. The pipeline demonstrates how imperfect human input, when mapped into legible, structured visual guidance, can be integrated with modern generative backbones without retraining or domain-specific supervision.

Practically, HandMade lowers the barrier for early-stage 3D ideation, supporting both novices and experts in externalizing spatial intent quickly, with an interactive loop that enables nuanced revision and alignment to evolving concepts. This division of labor—layout via sketch, appearance via language—balances expressive power and tolerance for imprecision, affording efficient creative workflows in design, scientific illustration, product modeling, and entertainment.

The intermediate representations—the part-guided multi-view images—function as both algorithmic constructs and direct interaction objects, facilitating explainable and editable generative systems.

Limitations and Future Directions

Despite strong geometric preservation, limitations remain in accurately expressing and constraining fine details, relational geometry, and topological structure via rough sketches. VR sketching is inherently constrained by input device ergonomics and human motor skills, and generative models may impose their own priors on part scales or contact.

Future extensions should:

• Automate and integrate the pipeline, supporting native, in-headset mesh inspection and part-wise local edits.
• Permit real-time, part-level locking, regeneration, or selective adherence controls within the VR authoring environment.
• Explore multimodal input modalities (e.g., speech, continuous gestures).
• Integrate explicit relational/spatial constraints beyond visual guides, increasing controllability and design specificity.
• Develop richer evaluation frameworks, encompassing workflow efficiency, downstream asset usability, and subjective design satisfaction rather than solely geometric alignment.

Conclusion

HandMade demonstrates that part-labeled VR sketches, interpreted as spatial prompts and integrated with structured textual descriptions, substantially improve the preservation of user-authored spatial intent in open-domain 3D generative workflows compared to text- or 2D-sketch-only baselines. The system leverages the respective strengths of sketching for spatial organization and language for semantic specification, yielding a pipeline that is robust to human input variability and conducive to iterative, user-driven refinement. This design paradigm advances human-AI collaboration in 3D asset creation, substantiating spatial prompting as a central organizing principle for future generative 3D interfaces.

Citations above refer to arXiv IDs and references provided inline in the main text. For further details, see (2606.27738).