Papers
Topics
Authors
Recent
Search
2000 character limit reached

Design2Cloth: 3D Cloth Generation from 2D Masks

Published 3 Apr 2024 in cs.CV | (2404.02686v1)

Abstract: In recent years, there has been a significant shift in the field of digital avatar research, towards modeling, animating and reconstructing clothed human representations, as a key step towards creating realistic avatars. However, current 3D cloth generation methods are garment specific or trained completely on synthetic data, hence lacking fine details and realism. In this work, we make a step towards automatic realistic garment design and propose Design2Cloth, a high fidelity 3D generative model trained on a real world dataset from more than 2000 subject scans. To provide vital contribution to the fashion industry, we developed a user-friendly adversarial model capable of generating diverse and detailed clothes simply by drawing a 2D cloth mask. Under a series of both qualitative and quantitative experiments, we showcase that Design2Cloth outperforms current state-of-the-art cloth generative models by a large margin. In addition to the generative properties of our network, we showcase that the proposed method can be used to achieve high quality reconstructions from single in-the-wild images and 3D scans. Dataset, code and pre-trained model will become publicly available.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Layered-garment net: Generating multiple implicit garment layers from a single image. In Proceedings of the Asian Conference on Computer Vision (ACCV), pages 3000–3017, 2022.
  2. Blazepose: On-device real-time body pose tracking, 2020.
  3. Cloth3d: clothed 3d humans. In European Conference on Computer Vision, pages 344–359. Springer, 2020.
  4. Multi-garment net: Learning to dress 3d people from images. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5420–5430, 2019.
  5. Dynamic surface function networks for clothed human bodies. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10754–10764, 2021.
  6. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123–16133, 2022.
  7. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11594–11604, 2021.
  8. Neural dual contouring. ACM Transactions on Graphics (TOG), 41(4):1–13, 2022.
  9. Smplicit: Topology-aware generative model for clothed people. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11875–11885, 2021.
  10. Dressing in order: Recurrent person image generation for pose transfer, virtual try-on and outfit editing. In Proceedings of the IEEE/CVF international conference on computer vision, pages 14638–14647, 2021.
  11. Fashiongan: display your fashion design using conditional generative adversarial nets. In Computer Graphics Forum, pages 109–119. Wiley Online Library, 2018.
  12. Deepgarment: 3d garment shape estimation from a single image. In Computer Graphics Forum, pages 269–280. Wiley Online Library, 2017.
  13. Drapenet: Garment generation and self-supervised draping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1451–1460, 2023.
  14. Moulding humans: Non-parametric 3d human shape estimation from single images. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2232–2241, 2019.
  15. Hood: Hierarchical graphs for generalized modelling of clothing dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16965–16974, 2023.
  16. Drape: Dressing any person. ACM Transactions on Graphics (ToG), 31(4):1–10, 2012.
  17. Meshudf: Fast and differentiable meshing of unsigned distance field networks. In European Conference on Computer Vision, pages 576–592. Springer, 2022.
  18. Garnet: A two-stream network for fast and accurate 3d cloth draping. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
  19. Viton: An image-based virtual try-on network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7543–7552, 2018.
  20. Learning locally editable virtual humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21024–21035, 2023.
  21. Shion Honda. Viton-gan: Virtual try-on image generator trained with adversarial loss. arXiv preprint arXiv:1911.07926, 2019.
  22. Invertible neural skinning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8715–8725, 2023.
  23. Segment anything. arXiv:2304.02643, 2023.
  24. Neuraltailor: Reconstructing sewing pattern structures from 3d point clouds of garments. ACM Transactions on Graphics (TOG), 41(4):1–16, 2022.
  25. Deepwrinkles: Accurate and realistic clothing modeling. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  26. Style2vec: Representation learning for fashion items from style sets. arXiv preprint arXiv:1708.04014, 2017.
  27. Dig: Draping implicit garment over the human body. In Proceedings of the Asian Conference on Computer Vision (ACCV), pages 2780–2795, 2022.
  28. Leveraging intrinsic properties for non-rigid garment alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14485–14496, 2023.
  29. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1–248:16, 2015.
  30. Learning to dress 3d people in generative clothing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  31. Controllable person image synthesis with attribute-decomposed gan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5084–5093, 2020.
  32. 3d clothed human reconstruction in the wild. In European Conference on Computer Vision (ECCV), 2022.
  33. Tailornet: Predicting clothing in 3d as a function of human pose, shape and garment style. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  34. Clothcap: Seamless 4d clothing capture and retargeting. ACM Transactions on Graphics (ToG), 36(4):1–15, 2017.
  35. Revisiting point cloud simplification: A learnable feature preserving approach. In European Conference on Computer Vision, pages 586–603. Springer, 2022.
  36. Handy: Towards a high fidelity 3d hand shape and appearance model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4670–4680, 2023.
  37. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
  38. Garmentgan: Photo-realistic adversarial fashion transfer. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 3923–3930. IEEE, 2021.
  39. Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration. In IEEE International Conference on Computer Vision Workshops, 2021.
  40. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2304–2314, 2019.
  41. Scanimate: Weakly supervised learning of skinned clothed avatar networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2886–2897, 2021.
  42. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  43. Learning-based animation of clothing for virtual try-on. In Computer Graphics Forum, pages 355–366. Wiley Online Library, 2019.
  44. Gan-based garment generation using sewing pattern images. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16, pages 225–247. Springer, 2020.
  45. Metaavatar: Learning animatable clothed human models from few depth images. Advances in Neural Information Processing Systems, 34:2810–2822, 2021.
  46. Learning a shared shape space for multimodal garment design. arXiv preprint arXiv:1806.11335, 2018.
  47. Clothgan: generation of fashionable dunhuang clothes using generative adversarial networks. Connection Science, 33(2):341–358, 2021.
  48. Banmo: Building animatable 3d neural models from many casual videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2863–2873, 2022.
  49. Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5746–5756, 2021.
  50. Detailed, accurate, human shape estimation from clothed 3d scan sequences. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  51. Surface extraction from neural unsigned distance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 0000–0000, 2023.
  52. Learning anchor transformations for 3d garment animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 491–500, 2023.
  53. Ilsh: The imperial light-stage head dataset for human head view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1112–1120, 2023.
  54. Pamir: Parametric model-conditioned implicit representation for image-based human reconstruction. IEEE transactions on pattern analysis and machine intelligence, 44(6):3170–3184, 2021.
  55. Clothesnet: An information-rich 3d garment model repository with simulated clothes environment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20428–20438, 2023.
  56. Deep fashion3d: A dataset and benchmark for 3d garment reconstruction from single images. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 512–530. Springer, 2020.
  57. Cloth4d: A dataset for clothed human reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12847–12857, 2023.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up