Papers
Topics
Authors
Recent
Search
2000 character limit reached

Image Sculpting: Precise Object Editing with 3D Geometry Control

Published 2 Jan 2024 in cs.GR and cs.CV | (2401.01702v1)

Abstract: We present Image Sculpting, a new framework for editing 2D images by incorporating tools from 3D geometry and graphics. This approach differs markedly from existing methods, which are confined to 2D spaces and typically rely on textual instructions, leading to ambiguity and limited control. Image Sculpting converts 2D objects into 3D, enabling direct interaction with their 3D geometry. Post-editing, these objects are re-rendered into 2D, merging into the original image to produce high-fidelity results through a coarse-to-fine enhancement process. The framework supports precise, quantifiable, and physically-plausible editing options such as pose editing, rotation, translation, 3D composition, carving, and serial addition. It marks an initial step towards combining the creative freedom of generative models with the precision of graphics pipelines.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (88)
  1. CLIP2StyleGAN: Unsupervised extraction of StyleGAN edit directions. In SIGGRAPH, 2022.
  2. Adobe. Adobe After Effects. https://www.adobe.com/products/aftereffects.html, 2023a.
  3. Adobe. Adobe Firefly. https://www.adobe.com/sensei/generative-ai/firefly.html, 2023b.
  4. Adobe. Adobe Substance 3D. https://www.adobe.com/creativecloud/3d-ar.html, 2023c.
  5. Stability AI. Stable Diffusion XL Refiner 1.0. https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0, 2023.
  6. Automatic rigging and animation of 3D characters. TOG, 2007.
  7. State of the art in quad meshing. In STARs, 2012.
  8. On linear variational surface deformation methods. TVCG, 2008.
  9. Polygon Mesh Processing. AK Peters, 2010.
  10. InstructPix2Pix: Learning to follow image editing instructions. In CVPR, 2023.
  11. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. 2019.
  12. Efficient geometry-aware 3D generative adversarial networks. In CVPR, 2022.
  13. ShapeNet: An information-rich 3D model repository. arXiv:1512.03012, 2015.
  14. Objaverse-XL: A universe of 10M+ 3D objects. In NeurIPS, 2023a.
  15. Objaverse: A universe of annotated 3D objects. In CVPR, 2023b.
  16. Diffusion models beat gans on image synthesis. In NeurIPS, 2021.
  17. Variational barycentric coordinates. TOG, 2023.
  18. Gerald Farin. Curves and Surfaces for CAGD: A Practical Guide. Morgan Kaufmann Publishers Inc., 5th edition, 2001.
  19. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576, 2015.
  20. Generative adversarial networks. In NeurIPS, 2014.
  21. StyleNeRF: A style-based 3D-aware generator for high-resolution image synthesis. In ICLR, 2022.
  22. threestudio: A unified framework for 3D content generation. https://github.com/threestudio-project/threestudio, 2023.
  23. Multiple view geometry in computer vision. Cambridge university press, 2003.
  24. Prompt-to-prompt image editing with cross attention control. In ICLR, 2023.
  25. LRM: Large reconstruction model for single image to 3D. arXiv:2311.04400, 2023.
  26. LoRA: Low-rank adaptation of large language models. In ICLR, 2021.
  27. Autodesk Inc. AutoDesk 3ds Max 2023. https://www.autodesk.com/products/3ds-max/overview, 2023.
  28. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
  29. Bounded biharmonic weights for real-time deformation. TOG, 2011.
  30. Skinning: Real-time shape deformation. TOG, 2014.
  31. Mean value coordinates for closed triangular meshes. TOG, 2005.
  32. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
  33. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
  34. Imagic: Text-based real image editing with diffusion models. In CVPR, 2023.
  35. Segment anything. arXiv:2304.02643, 2023.
  36. Modular primitives for high-performance differentiable rendering. TOG, 2020.
  37. Learning skeletal articulations with neural blend shapes. TOG, 2021.
  38. Consistent123: One image to highly consistent 3D asset using case-aware diffusion priors. arXiv:2309.17261, 2023.
  39. FreeDrag: Point tracking is not you need for interactive point-based image editing. arXiv:2307.04684, 2023.
  40. One-2-3-45: Any single image to 3D mesh in 45 seconds without per-shape optimization. arXiv:2306.16928, 2023a.
  41. Zero-1-to-3: Zero-shot one image to 3D object. In ICCV, 2023b.
  42. SyncDreamer: Generating multiview-consistent images from a single-view image. arXiv:2309.03453, 2023c.
  43. Wonder3D: Single image to 3D using cross-domain diffusion. arXiv:2310.15008, 2023.
  44. Marching cubes: A high resolution 3D surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field. ACM, 1998.
  45. Joint-dependent local deformations for hand animation and object grasping. In GI. CIPS, 1989.
  46. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
  47. OBJECT 3DIT: Language-guided 3D-aware image editing. arXiv:2307.11073, 2023.
  48. MidJourney. MidJourney. www.midjourney.com.
  49. Null-text inversion for editing real images using guided diffusion models. In CVPR, 2023.
  50. Instant neural graphics primitives with a multiresolution hash encoding. TOG, 2022.
  51. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. 2022.
  52. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR, 2020.
  53. OpenAI. DALL·E 3 System Card. https://openai.com/research/dall-e-3-system-card, 2023.
  54. Drag Your GAN: Interactive point-based manipulation on the generative image manifold. In TOG, 2023.
  55. StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery. In ICCV, 2021.
  56. Scalable diffusion models with transformers. In CVPR, 2023.
  57. SDXL: Improving latent diffusion models for high-resolution image synthesis. arXiv:2307.01952, 2023.
  58. DreamFusion: Text-to-3D using 2D diffusion. ICLR, 2023.
  59. Magic123: One image to high-quality 3D object generation using both 2D and 3D diffusion priors. arXiv:2306.17843, 2023.
  60. Hierarchical text-conditional image generation with clip latents. arXiv:2204.06125, 2022.
  61. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. TPAMI, 2020.
  62. Common objects in 3D: Large-scale learning and evaluation of real-life 3D category reconstruction. In ICCV, 2021.
  63. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  64. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
  65. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
  66. Free-form deformation of solid geometric models. In Computer Graphics. ACM, 1986.
  67. Interpreting the latent space of gans for semantic face editing. In CVPR, 2020.
  68. Zero123++: a single image to consistent multi-view diffusion base model. arXiv:2310.15110, 2023a.
  69. Mvdream: Multi-view diffusion for 3d generation. arXiv:2308.16512, 2023b.
  70. DragDiffusion: Harnessing diffusion models for interactive point-based image editing. arXiv:2306.14435, 2023c.
  71. Fundamentals of Computer Graphics. AK Peters, 2009.
  72. Fem simulation of 3d deformable solids: A practitioner’s guide to theory, discretization and model reduction. TOG, 2012.
  73. Scene representation networks: Continuous 3D-structure-aware neural scene representations. In NeurIPS, 2019.
  74. Denoising diffusion implicit models. In ICLR, 2021.
  75. As-rigid-as-possible surface modeling. In SGP, 2007.
  76. Plug-and-play diffusion features for text-driven image-to-image translation. In CVPR, 2023.
  77. Score jacobian chaining: Lifting pretrained 2D diffusion models for 3D generation. In CVPR, 2023a.
  78. Deep learning for image super-resolution: A survey. TPAMI, 43(10):3365–3387, 2020.
  79. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurIPS, 2023b.
  80. Multiview compressive coding for 3d reconstruction. In CVPR, 2023.
  81. Stylespace analysis: Disentangled controls for stylegan image generation. In CVPR, 2021.
  82. Tedigan: Text-guided diverse face image generation and manipulation. In CVPR, 2021.
  83. Holistically-nested edge detection. In ICCV, 2015.
  84. Jonathan Young. xatlas. https://github.com/jpcy/xatlas, 2023.
  85. PixelNeRF: Neural radiance fields from one or few images. In CVPR, 2021.
  86. Scaling autoregressive models for content-rich text-to-image generation. TMLR, 2023.
  87. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
  88. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
Citations (9)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 2 tweets with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free