Papers
Topics
Authors
Recent
Search
2000 character limit reached

Deep Spectral Meshes: Multi-Frequency Facial Mesh Processing with Graph Neural Networks

Published 15 Feb 2024 in cs.CV, cs.CG, and cs.GR | (2402.10365v1)

Abstract: With the rising popularity of virtual worlds, the importance of data-driven parametric models of 3D meshes has grown rapidly. Numerous applications, such as computer vision, procedural generation, and mesh editing, vastly rely on these models. However, current approaches do not allow for independent editing of deformations at different frequency levels. They also do not benefit from representing deformations at different frequencies with dedicated representations, which would better expose their properties and improve the generated meshes' geometric and perceptual quality. In this work, spectral meshes are introduced as a method to decompose mesh deformations into low-frequency and high-frequency deformations. These features of low- and high-frequency deformations are used for representation learning with graph convolutional networks. A parametric model for 3D facial mesh synthesis is built upon the proposed framework, exposing user parameters that control disentangled high- and low-frequency deformations. Independent control of deformations at different frequencies and generation of plausible synthetic examples are mutually exclusive objectives. A Conditioning Factor is introduced to leverage these objectives. Our model takes further advantage of spectral partitioning by representing different frequency levels with disparate, more suitable representations. Low frequencies are represented with standardised Euclidean coordinates, and high frequencies with a normalised deformation representation (DR). This paper investigates applications of our proposed approach in mesh reconstruction, mesh interpolation, and multi-frequency editing. It is demonstrated that our method improves the overall quality of generated meshes on most datasets when considering both the $L_1$ norm and perceptual Dihedral Angle Mesh Error (DAME) metrics.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Russo, M. Polygonal Modeling: Basic and Advanced Techniques; Jones & Bartlett Learning: Burlington, MA, USA, 2010.
  2. Surface representation and processing. In Proceedings of the 2009 8th IEEE International Conference on Cognitive Informatics, Hong Kong, China, 15–17 June 2009; IEEE: Piscatway, NJ, USA, 2009 pp. 542–545. https://doi.org/10.1109/COGINF.2009.5250681.
  3. Laplacian surface editing. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing, Nice, France, 8–10 July 2004, pp. 175–184.
  4. Spectral mesh processing. Comput. Graph. Forum 2010, 29, pp. 1865–1894. https://doi.org/10.1145/1837101.1837109.
  5. Sorkine, O. Laplacian Mesh Processing. In Eurographics (STARs); The Eurographics Association: Eindhoven The Netherlands, 2005, 29.
  6. Geometric Deep Learning: Going beyond Euclidean data. IEEE Signal Process. Mag. 2017, 34, pp. 18–42. https://doi.org/10.1109/MSP.2017.2693418.
  7. 3D Morphable Face Models—Past, Present, and Future. ACM Trans. Graph. 2020, 38, 157. https://doi.org/10.1145/3395208.
  8. A survey on deep geometry learning: From a representation perspective. Comput. Vis. Media 2020, 6, pp. 113–133. https://doi.org/10.1007/s41095-020-0174-8.
  9. Efficient and flexible deformation representation for data-driven surface modeling. ACM Trans. Graph. 2016, 35, 158. https://doi.org/10.1145/2908736.
  10. Sparse Data Driven Mesh Deformation. IEEE Trans. Vis. Comput. Graph. 2021, 27, pp. 2085–2100. https://doi.org/10.1109/tvcg.2019.2941200.
  11. Variational Autoencoders for Localized Mesh Deformation Component Analysis. IEEE Trans. Pattern Anal. Mach. Intell. 2022, 44, pp. 6297–6310. https://doi.org/10.1109/TPAMI.2021.3085887.
  12. Alive Caricature from 2D to 3D. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; IEEE: Piscataway, NJ, USA, 2018, pp. 7336–7345. https://doi.org/10.1109/CVPR.2018.00766.
  13. Localized Manifold Harmonics for Spectral Shape Analysis. Comput. Graph. Forum 2018, 37, pp. 20–34. https://doi.org/10.1111/cgf.13309.
  14. Fast calculation of Laplace-Beltrami eigenproblems via subdivision linear subspace. Comput. Graph. 2021, 97, pp. 236–247. https://doi.org/10.1016/j.cag.2021.04.019.
  15. Spectral Mesh Simplification. Comput. Graph. Forum 2020, 39, pp. 315–324. https://doi.org/10.1111/cgf.13932.
  16. Spectral 3D mesh segmentation with a novel single segmentation field. Graph. Model. 2014, 76, pp. 440–456. https://doi.org/10.1016/j.gmod.2014.04.009.
  17. Spectral mesh segmentation via ℓ0subscriptℓ0\ell_{0}roman_ℓ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT gradient minimization. IEEE Trans. Vis. Comput. Graph. 2020, 26, pp. 440–456. https://doi.org/10.1109/TVCG.2018.2882212.
  18. A Spectral Segmentation Method for Large Meshes. Commun. Math. Stat. 2023, 11, pp. 583–607. https://doi.org/10.1007/s40304-021-00265-4.
  19. Robust 3D Shape Correspondence in the Spectral Domain. In Proceedings of the IEEE International Conference on Shape Modeling and Applications 2006 (SMI’06), Matsushima, Japan, 14–16 June 2006, pp. 1–12. https://doi.org/10.1109/SMI.2006.31.
  20. Matching shapes by eigendecomposition of the Laplace-Beltrami operator. In Proceedings of the 5th International Symposium 3D Data Processing, Visualization and Transmission, Paris, France, 17–20 May 2010, pp. 1–8.
  21. ZoomOut: Spectral upsampling for efficient shape correspondence. ACM Trans. Graph, 2019, 38, 155: pp. 1–14. https://doi.org/10.1145/3355089.3356524.
  22. A Comprehensive Survey on Graph Neural Networks. IEEE Trans. Neural Netw. Learn. Syst. 2021, 32, pp. 4–24. https://doi.org/10.1109/tnnls.2020.2978386.
  23. Laplacian2Mesh: Laplacian-Based Mesh Understanding. IEEE Transactions on Visualization & Computer Graphics 2023, pp. 1–13 . https://doi.org/10.1109/TVCG.2023.3259044.
  24. SpecTrHuMS: Spectral transformer for human mesh sequence learning. Comput. Graph. 2023, 115, pp. 191–203. https://doi.org/10.1016/j.cag.2023.07.001.
  25. Learning on 3D Meshes With Laplacian Encoding and Pooling. IEEE Trans. Vis. Comput. Graph. 2022, 28, pp. 1317–1327. https://doi.org/10.1109/TVCG.2020.3014449.
  26. The Hierarchical Subspace Iteration Method for Laplace–Beltrami Eigenproblems. ACM Trans. Graph. 2022, 41(2), 17. https://doi.org/10.1145/3495208.
  27. Generating 3D Faces Using Convolutional Mesh Autoencoders. In Proceedings of the European Conference on Computer Vision (ECCV), ECCV, Munich, Germany, 8–14 September 2018, pp. 725–741.
  28. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain, 5–10 December 2016; pp. 3844–3852.
  29. Neural 3D morphable models: Spiral convolutional networks for 3D shape representation learning and generation. In Proceedings of the IEEE International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019, pp. 7212–7221. https://doi.org/10.1109/ICCV.2019.00731.
  30. Learning feature aggregation for deep 3D morphable models. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021, pp. 13159–13168. https://doi.org/10.1109/CVPR46437.2021.01296.
  31. Learning Local Neighboring Structure for Robust 3D Shape Representation. In Proceedings of The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21) Learning, Virtual, 2–9 February 2021, 35(2), pp. 1397–1405.
  32. Fully Convolutional Mesh Autoencoder using Efficient Spatially Varying Kernels. Adv. Neural Inf. Process. Syst. 2020, 33, pp. 9251–9262.
  33. FeaStNet: Feature-Steered Graph Convolutions for 3D Shape Analysis. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 2598–2606, https://doi.org/10.1109/CVPR.2018.00275.
  34. MeshGAN: Non-linear 3D Morphable Models of Faces. arXiv 2019. https://doi.org/10.48550/arXiv.1903.10384. https://doi.org/10.48550/arXiv.1903.10384
  35. Dense 3D Face Decoding over 2500FPS: Joint Texture & Shape Convolutional Mesh Decoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019, pp. 1097–1106.
  36. Mesh Variational Autoencoders with Edge Contraction Pooling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Long Beach, CA, USA, 15–20 June, 2019; IEEE: Piscataway, NJ, USA, 2019, pp. 274–275.
  37. Disentangled representation learning for 3D face shape. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; IEEE: Piscataway, NJ, USA, 2019, pp. 11949–11958. https://doi.org/10.1109/CVPR.2019.01223.
  38. Deformation representation based convolutional mesh autoencoder for 3D hand generation. Neurocomputing 2021, 444, pp. 356–365. https://doi.org/10.1016/j.neucom.2020.01.122.
  39. Semantic Deformation Transfer. In ACM SIGGRAPH 2009 papers, Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference, New Orleans, LA, USA, 3–7 August, 2009; ACM: New York, NY, USA, 2009; pp. 1–6. https://doi.org/10.1145/1576246.1531342.
  40. As-Rigid-As-Possible Surface Modeling. In Proceedings of the Symposium on Geometry Processing; Belyaev, A., Ed.; The Eurographics Association: Eindhoven The Netherlands, 2007, 4, pp. 109–116.
  41. Improving neural networks by preventing co-adaptation of feature detectors. arXiv 2012. https://doi.org/10.48550/arXiv.1207.0580.
  42. Humain Limited. Humain Limited—Research & Development, accessed on 12 May 2022, 2022. https://www.humain-studios.com/.
  43. FaceScape: A Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seatle, WA, USA, 13–19 June 2020, pp. 601–610.
  44. FaceWarehouse: A 3D facial expression database for visual computing. IEEE Trans. Vis. Comput. Graph. 2014, 20, pp. 413–425. https://doi.org/10.1109/TVCG.2013.249.
  45. ARPACK Users’ Guide: Solution of Large-Scale Eigenvalue Problems with Implicitly Restarted Arnoldi Methods; Society for Industrial and Applied Mathematics: Philadelphia, PA, USA, 1998.
  46. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 2014, 32, pp. 8026–8037.
  47. Dihedral Angle Mesh Error: A fast perception correlated distortion measure for fixed connectivity triangle meshes. Eurographics Symp. Geom. Process. 2012, 31, pp. 1715–1724. https://doi.org/10.1111/j.1467-8659.2012.03176.x.
  48. Perceptual metrics for static and dynamic triangle meshes. Comput. Graph. Forum 2013, 32, pp. 101–125. https://doi.org/10.1111/cgf.12001.
  49. SpiralNet++: A fast and highly efficient mesh convolution operator. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop, ICCVW 2019, Seoul, Republic of Korea, 27–28 October 2019, pp. 4141–4148. https://doi.org/10.1109/ICCVW.2019.00509.
  50. MeshCNN: A Network with an Edge. ACM Trans. Graph. (TOG) 2019, 38(4), pp. 1–12. https://doi.org/10.1145/3306346.3322959.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up