Papers
Topics
Authors
Recent
Search
2000 character limit reached

SHAP-EDITOR: Instruction-guided Latent 3D Editing in Seconds

Published 14 Dec 2023 in cs.CV | (2312.09246v1)

Abstract: We propose a novel feed-forward 3D editing framework called Shap-Editor. Prior research on editing 3D objects primarily concentrated on editing individual objects by leveraging off-the-shelf 2D image editing networks. This is achieved via a process called distillation, which transfers knowledge from the 2D network to 3D assets. Distillation necessitates at least tens of minutes per asset to attain satisfactory editing results, and is thus not very practical. In contrast, we ask whether 3D editing can be carried out directly by a feed-forward network, eschewing test-time optimisation. In particular, we hypothesise that editing can be greatly simplified by first encoding 3D objects in a suitable latent space. We validate this hypothesis by building upon the latent space of Shap-E. We demonstrate that direct 3D editing in this space is possible and efficient by building a feed-forward editor network that only requires approximately one second per edit. Our experiments show that Shap-Editor generalises well to both in-distribution and out-of-distribution 3D assets with different prompts, exhibiting comparable performance with methods that carry out test-time optimisation for each edited instance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (89)
  1. Sine: Semantic-driven image-based nerf editing with prior-guided editing field. In CVPR, 2023.
  2. Text2live: Text-driven layered image and video editing. In ECCV, 2022.
  3. Instructpix2pix: Learning to follow image editing instructions. In CVPR, pages 18392–18402, 2023.
  4. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. In SIGGRAPH, 2023.
  5. Neuraleditor: Editing neural radiance fields via manipulating point clouds. In CVPR, 2023a.
  6. Training-free layout control with cross-attention guidance. In WACV, 2023b.
  7. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.13873, 2023c.
  8. Transformers as meta-learners for implicit neural representations. In ECCV, 2022.
  9. Progressive3d: Progressively local editing for text-to-3d content creation with complex semantic prompts. arXiv preprint arXiv:2310.11784, 2023.
  10. Diffusion self-guidance for controllable image generation. In NeurIPS, 2023.
  11. Shapecrafter: A recursive text-conditioned 3d shape generation model. NeurIPS, 2022.
  12. Stylegan-nada: Clip-guided domain adaptation of image generators. In SIGGRAPH, 2021.
  13. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
  14. Get3d: A generative model of high quality 3d textured shapes learned from images. NeurIPS, 2022.
  15. Recolornerf: Layer decomposed radiance field for efficient color editing of 3d scenes. arXiv preprint arXiv:2301.07958, 2023.
  16. Blended-nerf: Zero-shot object generation and blending in existing neural radiance fields. ICCV, 2023.
  17. Instruct-nerf2nerf: Editing 3d scenes with instructions. In ICCV, 2023.
  18. Prompt-to-prompt image editing with cross-attention control. In ICLR, 2023.
  19. Classifier-free diffusion guidance. abs/2207.12598, 2022.
  20. Denoising diffusion probabilistic models. NeurIPS, 33:6840–6851, 2020.
  21. Zero-shot text-guided object generation with dream fields. In CVPR, 2022.
  22. Nerfshop: Interactive editing of neural radiance fields. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 6(1), 2023.
  23. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463, 2023.
  24. Instruct 3d-to-3d: Text instruction guided 3d-to-3d conversion. arXiv preprint arXiv:2303.15780, 2023.
  25. Conerf: Controllable neural radiance fields. In CVPR, 2022.
  26. Relu fields: The little non-linearity that could. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–9, 2022.
  27. Imagic: Text-based real image editing with diffusion models. In CVPR, 2023.
  28. Decomposing nerf for editing via feature field distillation. NeurIPS, 35:23311–23330, 2022.
  29. Nerf-vae: A geometry aware 3d scene generative model. In ICML. PMLR, 2021.
  30. Palettenerf: Palette-based appearance editing of neural radiance fields. In CVPR, 2023.
  31. Multi-concept customization of text-to-image diffusion. In CVPR, 2023.
  32. Ice-nerf: Interactive color editing of nerfs via decomposition-aware weight optimization. In ICCV, 2023.
  33. Tango: Text-driven photorealistic and robust 3d stylization via lighting decomposition. NeurIPS, 2022.
  34. Language-driven semantic segmentation. In ICLR, 2022a.
  35. Grounded language-image pre-training. In CVPR, 2022b.
  36. Focaldreamer: Text-driven 3d editing via focal-fusion assembly. arXiv preprint arXiv:2308.10608, 2023a.
  37. Gligen: Open-set grounded text-to-image generation. In CVPR, 2023b.
  38. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023.
  39. Editing conditional radiance fields. In ICCV, 2021.
  40. Image segmentation using text and image prompts. In CVPR, 2022.
  41. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
  42. Latent-nerf for shape-guided generation of 3d shapes and textures. In CVPR, 2023.
  43. Text2mesh: Text-driven neural stylization for meshes. In CVPR, 2022.
  44. Sked: Sketch-guided text-based 3d editing. In ICCV, 2023.
  45. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
  46. Spin-nerf: Multiview segmentation and perceptual inpainting with neural radiance fields. In CVPR, 2023.
  47. Clip-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 conference papers, pages 1–8, 2022.
  48. Null-text inversion for editing real images using guided diffusion models. In CVPR, 2023.
  49. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022.
  50. Ed-nerf: Efficient text-guided editing of 3d scene using latent space nerf. arXiv preprint arXiv:2310.02712, 2023.
  51. Zero-shot image-to-image translation. In SIGGRAPH, 2023.
  52. Cagenerf: Cage-based neural radiance field for generalized 3d deformation and animation. NeurIPS, 2022.
  53. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.
  54. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763. PMLR, 2021.
  55. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684–10695, 2022.
  56. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
  57. Clip-sculptor: Zero-shot generation of high-fidelity and diverse shapes from natural language. In CVPR, 2023.
  58. Vox-e: Text-guided voxel editing of 3d objects. In ICCV, 2023a.
  59. Vox-e: Text-guided voxel editing of 3d objects. In ICCV, 2023b.
  60. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. NeurIPS, 2021.
  61. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, pages 2256–2265. PMLR, 2015.
  62. Blending-nerf: Text-driven localized editing in neural radiance fields. In ICCV, 2023.
  63. Denoising diffusion implicit models. In ICLR, 2021.
  64. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, 2022.
  65. Neural feature fusion fields: 3d distillation of self-supervised 2d image representations. In 2022 International Conference on 3D Vision (3DV), pages 443–453. IEEE, 2022.
  66. Splicing vit features for semantic appearance transfer. In CVPR, 2022.
  67. Plug-and-play diffusion features for text-driven image-to-image translation. In CVPR, 2023.
  68. Proteusnerf: Fast lightweight nerf editing using 3d-aware image context. arXiv preprint arXiv:2310.09965, 2023a.
  69. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. In CVPR, 2022.
  70. Nerf-art: Text-driven neural radiance fields stylization. IEEE TVCG, 2023b.
  71. Inpaintnerf360: Text-guided 3d inpainting on unbounded neural radiance fields. arXiv preprint arXiv:2305.15094, 2023c.
  72. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305.16213, 2023d.
  73. Removing objects from neural radiance fields. In CVPR, 2023.
  74. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In CVPR, 2023.
  75. Desrf: Deformable stylized radiance field. In CVPR, 2023.
  76. Deforming radiance fields with cages. In ECCV, 2022.
  77. Learning object-compositional neural radiance field for editable scene rendering. In ICCV, 2021.
  78. Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. In ECCV, 2022.
  79. Edit-diffnerf: Editing 3d neural radiance fields using 2d diffusion model. arXiv preprint arXiv:2306.09551, 2023.
  80. Nerf-editing: geometry editing of neural radiance fields. In CVPR, 2022.
  81. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint arXiv:2210.06978, 2022.
  82. Text-guided generation and editing of compositional 3d avatars. arXiv preprint arXiv:2309.07125, 2023a.
  83. Arf: Artistic radiance fields. In ECCV, 2022.
  84. Magicbrush: A manually annotated dataset for instruction-guided image editing. In NeurIPS, 2023b.
  85. Adding conditional control to text-to-image diffusion models. In ICCV, 2023c.
  86. Hive: Harnessing human feedback for instructional visual editing. arXiv preprint arXiv:2303.09618, 2023d.
  87. Editablenerf: Editing topologically varying neural radiance fields by key points. In CVPR.
  88. Repaint-nerf: Nerf editting via semantic masks and diffusion models. arXiv preprint arXiv:2306.05668, 2023.
  89. Dreameditor: Text-driven 3d scene editing with neural fields. arXiv preprint arXiv:2306.13455, 2023.
Citations (7)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 2 tweets with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free