Papers
Topics
Authors
Recent
Search
2000 character limit reached

Incorporating dense metric depth into neural 3D representations for view synthesis and relighting

Published 4 Sep 2024 in cs.CV, cs.GR, and cs.RO | (2409.03061v1)

Abstract: Synthesizing accurate geometry and photo-realistic appearance of small scenes is an active area of research with compelling use cases in gaming, virtual reality, robotic-manipulation, autonomous driving, convenient product capture, and consumer-level photography. When applying scene geometry and appearance estimation techniques to robotics, we found that the narrow cone of possible viewpoints due to the limited range of robot motion and scene clutter caused current estimation techniques to produce poor quality estimates or even fail. On the other hand, in robotic applications, dense metric depth can often be measured directly using stereo and illumination can be controlled. Depth can provide a good initial estimate of the object geometry to improve reconstruction, while multi-illumination images can facilitate relighting. In this work we demonstrate a method to incorporate dense metric depth into the training of neural 3D representations and address an artifact observed while jointly refining geometry and appearance by disambiguating between texture and geometry edges. We also discuss a multi-flash stereo camera system developed to capture the necessary data for our pipeline and show results on relighting and view synthesis with a few training views.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (112)
  1. Artec Leo, 2023.
  2. Ensenso XR series scanners, 2023.
  3. Photoneo PhoXi3D scanners, 2023.
  4. Azure Kinect 3D sensors, 2023.
  5. 3DZephyr. 3df zephyr - photogrammetry software - 3d models from photos, 2022.
  6. AliceVision. Alicevision meshroom, 2022.
  7. Törf: Time-of-flight radiance fields for dynamic scene view synthesis. Advances in neural information processing systems, 34:26289–26301, 2021.
  8. Neural rgb-d surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6290–6301, 2022.
  9. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008.03824, 2020.
  10. Blender. Blender - a 3D modelling and rendering package. Blender Foundation, Blender Institute, Amsterdam, 2024.
  11. Pierre Boudoin. Hyper capture: 3d object scan, 2023.
  12. Supervol: Super-resolution shape and reflectance estimation in inverse volume rendering. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3139–3149, 2024.
  13. Brent Burley. Physically-based shading at disney. In Acm Siggraph, pages 1–7. vol. 2012, 2012.
  14. On differential photometric reconstruction for unknown, isotropic brdfs. IEEE transactions on pattern analysis and machine intelligence, 35(12):2941–2955, 2012.
  15. Shape from shading for robotic manipulation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8389–8398, 2024.
  16. Self-calibrating deep photometric stereo networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8739–8747, 2019.
  17. Auto-tuning structured light by optical stochastic gradient descent. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  18. Learning depth with convolutional spatial propagation network. IEEE transactions on pattern analysis and machine intelligence, 42(10):2361–2379, 2019.
  19. Wildlight: In-the-wild inverse rendering with a flashlight. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4305–4314, 2023.
  20. Robust reconstruction of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5556–5565, 2015.
  21. MeshLab: an Open-Source Mesh Processing Tool. In Eurographics Italian Chapter Conference. The Eurographics Association, 2008.
  22. Viscosity solutions of hamilton-jacobi equations. Transactions of the American mathematical society, 277(1):1–42, 1983.
  23. CREE. Cree xlamp cxa2540, 2024.
  24. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 303–312, 1996.
  25. Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics (ToG), 36(4):1, 2017.
  26. Pie-net: Photometric invariant edge guided network for intrinsic image decomposition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19790–19799, 2022.
  27. Pandora: Polarization-aided neural decomposition of radiance. In European Conference on Computer Vision, pages 538–556. Springer, 2022.
  28. Depth-supervised NeRF: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  29. Smerf: Streamable memory efficient radiance fields for real-time large-scene exploration, 2023.
  30. EdmundOptics. C series fixed focal length lenses: Edmund Optics, 2024.
  31. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10786–10796, 2021.
  32. Distance transforms of sampled functions. Theory of computing, 8(1):415–428, 2012.
  33. Specular reflection reduction with multi-flash imaging. In Proceedings. 17th Brazilian symposium on computer graphics and image processing, pages 316–321. IEEE, 2004.
  34. Discontinuity preserving stereo with small baseline multi-flash illumination. In Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, pages 412–419. IEEE, 2005.
  35. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381–395, 1981.
  36. FLIR. Grasshopper3 USB3 model: GS3-U3-41C6C, 2024.
  37. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501–5510, 2022.
  38. Geo-NeUS: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction. Advances in Neural Information Processing Systems, 35:3403–3416, 2022.
  39. Massively parallel multiview stereopsis by surface normal diffusion. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
  40. Ginibird. Garden tea lights [license: Cc-0], 2021.
  41. Bayes’ Rays: Uncertainty quantification in neural radiance fields. CVPR, 2024.
  42. Photogeometric scene flow for high-detail dynamic 3d reconstruction. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
  43. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002.10099, 2020.
  44. Delira: Self-supervised depth, light, and radiance fields. arXiv preprint arXiv:2304.02797, 2023.
  45. Heiko Hirschmuller. Accurate and efficient stereo processing by semi-global matching and mutual information. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), pages 807–814. IEEE, 2005.
  46. Large scale multi-view stereopsis evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 406–413, 2014.
  47. Learning efficient illumination multiplexing for joint capture of reflectance and shape. ACM Trans. Graph., 38(6):165–1, 2019.
  48. Neural reflectance capture in the view-illumination domain. IEEE Transactions on Visualization and Computer Graphics, 29(2):1450–1462, 2021.
  49. Screened poisson surface reconstruction. ACM Transactions on Graphics (ToG), 32(3):1–13, 2013.
  50. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
  51. Intel RealSense stereoscopic depth cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1–10, 2017.
  52. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  53. E-nerf: Neural radiance fields from a moving event camera. IEEE Robotics and Automation Letters, 8(3):1587–1594, 2023.
  54. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (ToG), 36(4):1–13, 2017.
  55. A theory of shape by space carving. In Proceedings of the Seventh IEEE International Conference on Computer Vision, pages 307–314. IEEE, 1999.
  56. Self-calibrating photometric stereo by neural inverse rendering. In European Conference on Computer Vision, pages 166–183. Springer, 2022.
  57. Multi-view photometric stereo: A robust solution and benchmark dataset for spatially varying isotropic materials. IEEE Transactions on Image Processing, 29:4159–4173, 2020.
  58. Nerfacc: A general nerf acceleration toolbox. arXiv preprint arXiv:2210.04847, 2022.
  59. Neuralangelo: High-fidelity neural surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  60. Near-light photometric stereo using circularly placed point light sources. In 2018 IEEE International Conference on Computational Photography (ICCP), pages 1–10. IEEE, 2018.
  61. Openillumination: A multi-illumination dataset for inverse rendering evaluation on real objects. arXiv preprint arXiv:2309.07921, 2023a.
  62. Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images. arXiv preprint arXiv:2305.17398, 2023b.
  63. Robust e-nerf: Nerf from sparse & noisy events under non-uniform motion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 18335–18346, 2023.
  64. LumaLabs. Luma ai: Ai for gorgeous 3d capture, 2023.
  65. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
  66. Matterport. Matterport: Drive results with digital twins., 2023.
  67. Nelson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, 1(2):99–108, 1995.
  68. Exposure fusion. In 15th Pacific Conference on Computer Graphics and Applications (PG’07), pages 382–390. IEEE, 2007.
  69. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  70. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
  71. NeRF in the dark: High dynamic range view synthesis from noisy raw images. CVPR, 2022.
  72. Turbosl: Dense accurate and fast 3d by neural inverse structured light. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 25067–25076, 2024.
  73. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1–15, 2022.
  74. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5589–5599, 2021.
  75. Colored point cloud registration revisited. In Proceedings of the IEEE international conference on computer vision, pages 143–152, 2017.
  76. Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging. ACM transactions on graphics (TOG), 23(3):679–688, 2004.
  77. RealityCapture. Realitycapture: 3d models from photos and/or laser scans, 2022.
  78. Photographic tone reproduction for digital images. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 661–670. 2023.
  79. Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes. ACM Transactions on Graphics (TOG), 42(4):1–12, 2023.
  80. Dense depth priors for neural radiance fields from sparse input views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  81. Point-slam: Dense neural point cloud-based slam. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
  82. From coarse to fine: Robust hierarchical localization at large scale. In CVPR, 2019.
  83. On joint estimation of pose, geometry and svbrdf from a handheld scanner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  84. Towards scalable multi-view reconstruction of geometry and materials. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  85. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016.
  86. Neural fields for structured lighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3512–3522, 2023.
  87. A benchmark dataset and evaluation for non-lambertian and uncalibrated photometric stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3707–3716, 2016.
  88. Shining3D. Einscansp, 2023.
  89. Sketchfab. Sketchfab: Online 3d geometry viewer, 2024.
  90. Nuvo: Neural uv mapping for unruly 3d representations. arXiv, 2023.
  91. The Replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797, 2019.
  92. Neural 3d reconstruction in the wild. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–9, 2022.
  93. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537–7547, 2020.
  94. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, 2023.
  95. Relight my nerf: A dataset for novel view synthesis and relighting of real world objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20762–20772, 2023.
  96. Shinji Umeyama. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis & Machine Intelligence, 13(04):376–380, 1991.
  97. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689, 2021.
  98. Adaptive shells for efficient neural radiance field rendering. ACM Trans. Graph., 42(6), 2023.
  99. Embree ray tracing kernels for cpus and the xeon phi architecture. In ACM SIGGRAPH 2013 Talks, pages 1–1. 2013.
  100. Gmflow: Learning optical flow via global matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8121–8130, 2022.
  101. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. Computer Vision and Pattern Recognition (CVPR), 2020.
  102. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805–4815, 2021.
  103. Bakedsdf: Meshing neural sdfs for real-time view synthesis. arXiv preprint arXiv:2302.14859, 2023.
  104. Jonathan Young. Sketchfab: Online 3d geometry viewer, 2024.
  105. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in Neural Information Processing Systems (NeurIPS), 2022.
  106. Non-parametric local transforms for computing visual correspondence. In Computer Vision—ECCV’94: Third European Conference on Computer Vision Stockholm, Sweden, May 2–6 1994 Proceedings, Volume II 3, pages 151–158. Springer, 1994.
  107. ZDNet, 2023.
  108. Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5565–5574, 2022.
  109. Du 2 net: Learning depth estimation from dual-cameras and dual-pixels. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 582–598. Springer, 2020.
  110. Fast global registration. In European Conference on Computer Vision, pages 766–782. Springer, 2016.
  111. Multi-view photometric stereo with spatially varying isotropic materials. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1482–1489, 2013.
  112. Shading-based refinement on volumetric signed distance functions. ACM Transactions on Graphics (ToG), 34(4):1–14, 2015.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up