CoGen3D: An Agentic Human-AI Co-Design Pipeline for 3D Asset Generation for Virtual Reality

Abstract: Creating 3D assets for virtual reality requires modeling expertise, which restricts the authorship of immersive experiences. Existing generative AI tools rely on unconstrained, command-driven prompting, lacking the conversational scaffolding needed for users to articulate their intent and validate designs prior to rendering. To address this, we introduce CoGen3D, an agentic human-AI co-design pipeline that proactively guides users through conversational intent elicitation, a concept image confirmation, and image-to-3D generation that directly deploys to immersive scenes. We evaluated this system through a user study (N=120) across six affectively diverse immersive scenes, observing 60 Design group participants who co-created 3D assets for the scenes, and 60 Validation group participants who experienced the scenes with generated assets. Our findings show that co-designed assets are associated with higher scene engagement and shifted affective responses, while participants generally preferred concept images over the final 3D assets, with no increased leniency toward degradation in their own creations. Analysis of the human-AI conversations further shows that target environments shape users' conversational patterns. Our results suggest that our staged, intent-based co-design can democratize virtual reality authoring and shift immersive content creation from technical execution toward collaborative spatial design.

• The paper introduces a staged human-AI co-design pipeline that uses conversational dialogue to lower the barrier for VR 3D asset creation.
• It details a tripartite system integrating LLM intent elicitation, text-to-image confirmation, and image-to-3D synthesis, validated through robust user studies.
• Despite strong semantic alignment, the pipeline highlights a fidelity bottleneck in converting 2D concepts to 3D meshes, suggesting areas for future improvement.

CoGen3D: An Agentic Pipeline for Human-AI Co-Design of 3D Assets for Virtual Reality

Introduction and Motivation

3D asset creation for virtual reality (VR) remains a significant bottleneck impeding democratized VR authoring. Traditional approaches typically demand advanced modeling expertise and time-intensive manual processes, sharply limiting access for creators with non-technical backgrounds. While a range of generative AI tools have emerged, most are limited to single-shot, text-prompt-based workflows, unattended to user intent clarification, pre-generation validation, or contextual adaptation to immersive scenes.

CoGen3D introduces a structured, agentic pipeline that re-centers conversational interaction as the locus for 3D asset generation and deployment, systematically scaffolding users through three staged phases: (1) intent elicitation via LLM-driven dialogue, (2) concept image confirmation (text-to-image generation), and (3) deployment of generated 3D assets into VR environments (image-to-3D synthesis).

Figure 1: CoGen3D's system design, integrating the agentic co-design pipeline with a web-based interface, generative model endpoints, and a Unity-based VR deployment client.

System Architecture and Pipeline Design

CoGen3D orchestrates three tightly integrated modules:

• Web-based co-design client: Implements incremental intent elicitation via an LLM (DeepSeek V3), supporting iterative dialogue and discrete prompt confirmation.
• Backend orchestration: Manages model interactions with FLUX.1 (text-to-image) and Hunyuan3D-2 (image-to-3D), leveraging local GPU servers for resource efficiency, with asynchronous job management to mitigate high-latency 3D synthesis.
• Immersive deployment: Unity-based VR runtime ingests GLB assets, enabling real-time manipulation (grabbing, scaling, rotation) and detailed telemetry capture for downstream behavioral analytics.

This staged arrangement explicitly fronts the low-latency, user-validated 2D conceptualization stage, deferring the computationally expensive, higher-latency 3D generation until artistic intent is ratified.

Figure 2: Latency benchmarks for FLUX.1 and Hunyuan3D-2 endpoints, justifying parameter selections that optimize responsiveness for interactive co-design while retaining generative fidelity.

Experimental Protocol and Evaluation

A comprehensive user study ($N=120$) was conducted with two separate groups:

• Design group ($n=60$): Participants engaged in full pipeline co-design, from initial VR scene exploration (asset-free baseline), conversational asset creation per scene, and in-VR validation of their generated assets.
• Validation group ($n=60$): Participants experienced populated VR scenes using assets designed by unrelated participants, providing affective and satisfaction ratings.

Affective variety was ensured by deploying the pipeline across six distinct VR environments spanning multiple quadrants of Russell's circumplex (valence–arousal) space.

Figure 3: Participant workflow for both Design and Validation groups, illustrating distinct stages in the experimental protocol.

Generative Pipeline Performance and Asset Quality

CoGen3D robustly supports scene-congruent, semantically adapted 3D asset creation by non-experts. LLM-elicited prompts and 2D concept images captured a wide spread of contextually appropriate artifacts (from playful to utilitarian), driven by the affective tone and affordances of target scenes.

Figure 4: End-to-end sample progression for all six VR scenes—from LLM prompts to 2D images, through 3D asset generation, to in-scene deployment.

Despite strong semantic correspondence and successful creative scaffolding, a marked degradation in fidelity was consistently observed between image and 3D mesh stages:

• 2D concept images were rated significantly higher than downstream 3D assets, with satisfaction means dropping ~0.5–1 point (on a 5-point scale) after conversion.
• There was no evidence of author leniency: participants who co-designed the assets showed no higher tolerance for mesh artifacts than independent validators.

  Figure 5: Joint satisfaction ratings for 2D images vs. 3D assets, with kernel density plots confirming the consistent drop in quality perception and exemplar pairs for high/low satisfaction cases.

Affective Impact and Behavioral Outcomes

Populating scenes with co-designed 3D assets yielded robust, scene-selective shifts in user affect and engagement:

• Affective response (SAM): Significant increases in valence, arousal, and dominance occurred in negatively valenced or neutral scenes (Tunnel, Solitary Confinement), without effect in scenes already at affective ceiling (Puppies, Lake, Elephants). Main scene effects dominated.
• Engagement: Participants in populated-asset conditions exhibited up to 2.3× longer dwell time across scenes, with the largest boosts in positive, interactive environments.

Figure 6: Differences in SAM (valence–arousal–dominance) ratings per scene/group, with enhanced affective engagement contingent on asset presence and scene context.

Asset Manipulation: Spatial interaction patterns closely tracked scene affordances, with open/natural scenes eliciting higher translation and up-scaling, while indoor/constrained scenes produced smaller, localized adjustments.

Figure 7: Scene engagement time and analytics of asset translation, rotation, and scaling across environments confirm context-sensitive user behaviors.

Human-AI Interaction Analysis

Conversational analysis revealed:

• Conversational tempo and cognitive burden varied systematically with scene affect: most time and deliberation were required for negative/high-arousal scenes (Shouting Man with Gun, Solitary Confinement).
• AI agent response was consistently swift (5–6s per turn), while human response times varied widely by emotional context (22–31s), revealing affect-sensitive design cognition.

Figure 8: Temporal metrics for the co-design pipeline, illustrating per-scene differences in message counts, human/AI response times, and generative endpoint durations.

Prompt Semantics and Design Strategies

Analysis of accepted text-to-image prompts demonstrated:

• Dominance of 'realistic' and 'minimalist' design styles, with contextually driven functional or playful object choices.
• Semantic congruence: Asset categories and styles were systematically matched to the emotional tone and spatial logic of each environment.

  Figure 9: Distribution of most frequent objects, categories, and styles in submitted text-to-image prompts, with strong scene-dependency.

User Perception and Design Experience

Post-study feedback indicated high perceived usability, helpfulness, and creativity support (~4.2/5), primarily praising the agent’s ability to structure and clarify intent. Criticisms focused on limited downstream 3D fidelity, constrained interaction, and a desire for richer scene-aware control over scaling and positioning.

Figure 10: Quantitative feedback on system experience and most frequent themes in participant comments regarding co-design workflow.

Theoretical and Practical Implications

1. Pipeline Efficacy and Generalizability: The agentic conversation-first approach facilitates intent articulation and democratizes 3D asset authoring, allowing rapid, iterative design even by novices.
2. Bottleneck in Image-to-3D Generation: Persistent quality degradation in current diffusion-based mesh synthesis disrupts potential 'IKEA effect' (ownership-induced leniency), necessitating advances in geometry/texture learning and/or hybrid manual refinement.
3. Scene-driven Reasoning and World-Model Need: User interaction patterns demonstrate strong world-model priors. Generative pipelines that integrate spatial context—bounding boxes, semantic surfaces—could significantly reduce post-hoc re-scaling and placement corrections [safaribazargani20262d].
4. Beyond Static Geometry: True VR immersion demands co-generation of physical affordances (kinematics, colliders, materiality), not just static meshes; future pipelines should integrate animation and interaction capabilities [yang2025eyes, cai2025hpipainting].
5. Limitations and Scalability: Current architecture is validated for single objects per scene and a relatively homogeneous participant sample; extension to multi-object, multi-user, and real-world collaborative deployments remains a key next step.

Future Directions

Research should focus on:

• Generative world models: Automated placement, scaling, and semantic alignment to mitigate manual adjustment after asset import.
• Higher-fidelity diffusion models: Progress in end-to-end 3D texture synthesis with improved physical realism, reduced geometric artifacts, and optional fine-tuning.
• Unification of geometry and kinematics: Direct generation of jointed, interactive assets supporting physically plausible user/object interactions.
• Expanded multi-asset authoring: Planning and populating complex scenes, integrating conversational scene graph specification, and multi-stage generative workflows.

Conclusion

CoGen3D demonstrates that an agentic, staged, human-AI co-design pipeline can lower the skill floor for immersive VR content creation, foster scene-driven creative reasoning, and measurably improve affective and engagement outcomes. However, the fidelity bottleneck at the image-to-3D transition persists as the most acute technical impediment. Addressing world-modeling, fidelity, and interactivity within generative workflows is essential for the full democratization and enrichment of VR authoring pipelines.

References: (2607.03731), [zhang2024vrcopilot], [lee2025imaginatear], [zhao2025hunyuan3d], [labs2025flux1kontextflowmatching], [kuang2026understanding].