Papers
Topics
Authors
Recent
Search
2000 character limit reached

Generating Non-Stationary Textures using Self-Rectification

Published 5 Jan 2024 in cs.CV, cs.GR, and cs.LG | (2401.02847v2)

Abstract: This paper addresses the challenge of example-based non-stationary texture synthesis. We introduce a novel twostep approach wherein users first modify a reference texture using standard image editing tools, yielding an initial rough target for the synthesis. Subsequently, our proposed method, termed "self-rectification", automatically refines this target into a coherent, seamless texture, while faithfully preserving the distinct visual characteristics of the reference exemplar. Our method leverages a pre-trained diffusion network, and uses self-attention mechanisms, to gradually align the synthesized texture with the reference, ensuring the retention of the structures in the provided target. Through experimental validation, our approach exhibits exceptional proficiency in handling non-stationary textures, demonstrating significant advancements in texture synthesis when compared to existing state-of-the-art techniques. Code is available at https://github.com/xiaorongjun000/Self-Rectification

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Cross-image attention for zero-shot appearance transfer, 2023.
  2. A neural space-time representation for text-to-image personalization, 2023.
  3. Time-varying weathering in texture space. ACM Trans. on Graphics (Proc. of SIGGRAPH), 35(4):1–11, 2016.
  4. Structural analogy from a single image pair. Computer Graphics Forum, 40(1):249–265, 2021.
  5. Learning texture manifolds with the periodic spatial gan. In Proc. of IEEE Int. Conf. on Machine Learning, page 469–477, 2017.
  6. MasaCtrl: tuning-free mutual self-attention control for consistent image synthesis and editing. In Proc. of Int. Conf. on Computer Vision, pages 22560–22570, October 2023.
  7. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models, 2023.
  8. Image quilting for texture synthesis and transfer. In Proc. of SIGGRAPH, pages 341–346, 2001.
  9. Texture synthesis by non-parametric sampling. In Proc. of Int. Conf. on Computer Vision, 1999.
  10. Tilegan: synthesis of large-scale non-homogeneous textures. ACM Trans. on Graphics (Proc. of SIGGRAPH), 38(4):1–11, 2019.
  11. An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022.
  12. Texture synthesis using convolutional neural networks. In Advances in Neural Information Processing Systems, 2015.
  13. A sliced wasserstein loss for neural texture synthesis. In Proc. of IEEE Conf. on Computer Vision & Pattern Recognition, June 2021.
  14. Prompt-to-prompt image editing with cross attention control. In Proc. of International Conference on Learning Representations, 2023.
  15. Image analogies. In Proc. of SIGGRAPH, page 327–340, 2001.
  16. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  17. Lora: Low-rank adaptation of large language models, 2021.
  18. Texture synthesis with spatial generative adversarial networks. arXiv preprint arXiv:1611.08207, 2016.
  19. Self tuning texture optimization. Computer Graphics Forum (Proc. of Eurographics), 34(2):349–359, may 2015.
  20. Anant Khandelwal. Infusion: Inject and attention fusion for multi concept zero shot text based video editing. arXiv preprint arXiv:2308.00135, 2023.
  21. Texture optimization for example-based synthesis. ACM Trans. on Graphics (Proc. of SIGGRAPH), August 2005.
  22. Graphcut textures: Image and video synthesis using graph cuts. ACM Trans. on Graphics (Proc. of SIGGRAPH), 22(3):277–286, jul 2003.
  23. Precomputed real-time texture synthesis with markovian generative adversarial networks. In Proc. of Euro. Conf. on Computer Vision, pages 702–716, 2016.
  24. Multi-scale label-map extraction for texture synthesis. ACM Trans. on Graphics (Proc. of SIGGRAPH), 35(4):140:1–140:12, July 2016.
  25. Brushables: Example-based edge-aware directional texture painting. Computer Graphics Forum (Proc. of Pacific Conf. on Computer Graphics & Applications), 34(7):257–267, oct 2015.
  26. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–11, 2023.
  27. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  28. Seamlessgan: Self-supervised synthesis of tileable texture maps. IEEE Trans. Visualization & Computer Graphics, 2022.
  29. High-resolution image synthesis with latent diffusion models. In Proc. of IEEE Conf. on Computer Vision & Pattern Recognition, pages 10684–10695, 2022.
  30. Layered shape synthesis: Automatic generation of control maps for non-stationary textures. ACM Trans. on Graphics (Proc. of SIGGRAPH), 28(5):107:1–9, Dec. 2009.
  31. Singan: Learning a generative model from a single natural image. In Proc. of Int. Conf. on Computer Vision, 2019.
  32. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proc. of IEEE Conf. on Computer Vision & Pattern Recognition, 2023.
  33. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models, 2023.
  34. Deep correlations for texture synthesis. ACM Trans. on Graphics, 36(5):161:1–161:15, July 2017.
  35. Fast texture synthesis via pseudo optimizer. In Proc. of IEEE Conf. on Computer Vision & Pattern Recognition, pages 5498–5507, 2020.
  36. Ingan: Capturing and retargeting the ”dna” of a natural image. In Proc. of Int. Conf. on Computer Vision, 2019.
  37. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
  38. Plug-and-play diffusion features for text-driven image-to-image translation. In Proc. of IEEE Conf. on Computer Vision & Pattern Recognition, pages 1921–1930, June 2023.
  39. Texture networks: Feed-forward synthesis of textures and stylized images. In Proc. of IEEE Int. Conf. on Machine Learning, page 1349–1357, 2016.
  40. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
  41. Appearance manifolds for modeling time-variant appearance of materials. ACM Trans. on Graphics (Proc. of SIGGRAPH), 25(3):754–761, 2006.
  42. Fast texture synthesis using tree-structured vector quantization. In Proc. of SIGGRAPH, pages 479–488, 2000.
  43. Space-time completion of video. IEEE Trans. Pattern Analysis & Machine Intelligence, 29(3):463–476, 2007.
  44. State of the Art in Example-based Texture Synthesis. In M. Pauly and G. Greiner, editors, Eurographics 2009 - State of the Art Reports. The Eurographics Association, 2009.
  45. Paint by example: Exemplar-based image editing with diffusion models. arXiv preprint arXiv:2211.13227, 2022.
  46. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. In arXiv preprint arxiv:2308.06721, 2023.
  47. Adding conditional control to text-to-image diffusion models, 2023.
  48. Neural texture synthesis with guided correspondence. In Proc. of IEEE Conf. on Computer Vision & Pattern Recognition, pages 18095–18104, 2023.
  49. Analysis and controlled synthesis of inhomogeneous textures. Computer Graphics Forum (Proc. of Eurographics), 36(2):199–212, 2017.
  50. Non-stationary texture synthesis by adversarial expansion. ACM Trans. on Graphics (Proc. of SIGGRAPH), 37(4):49:1–49:13, 2018.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free