A Multiple-View Geometric Model for Specularity Prediction on General Curved Surfaces

Abstract: Specularity prediction is essential to many computer vision applications, giving important visual cues usable in Augmented Reality (AR), Simultaneous Localisation and Mapping (SLAM), 3D reconstruction and material modeling. However, it is a challenging task requiring numerous information from the scene including the camera pose, the geometry of the scene, the light sources and the material properties. Our previous work addressed this task by creating an explicit model using an ellipsoid whose projection fits the specularity image contours for a given camera pose. These ellipsoid-based approaches belong to a family of models called JOint-LIght MAterial Specularity (JOLIMAS), which we have gradually improved by removing assumptions on the scene geometry. However, our most recent approach is still limited to uniformly curved surfaces. This paper generalises JOLIMAS to any surface geometry while improving the quality of specularity prediction, without sacrificing computation performances. The proposed method establishes a link between surface curvature and specularity shape in order to lift the geometric assumptions made in previous work. Contrary to previous work, our new model is built from a physics-based local illumination model namely Torrance-Sparrow, providing an improved reconstruction. Specularity prediction using our new model is tested against the most recent JOLIMAS version on both synthetic and real sequences with objects of various general shapes. Our method outperforms previous approaches in specularity prediction, including the real-time setup, as shown in the supplementary videos.

• The paper introduces Canonical Dual JOLIMAS, a novel geometric model that overcomes uniform curvature assumptions to accurately predict specularity on arbitrary curved surfaces.
• It employs a canonical mapping technique using invariant limit angles to transform specularity contours, ensuring robustness across dynamic deformations in both synthetic and real environments.
• Empirical results show significant error reduction (as low as 2.4%) with real-time performance, offering impactful benefits for AR, SLAM, and photorealistic rendering applications.

Multiple-View Geometric Modelling for Generalized Specularity Prediction

Introduction

The paper "A Multiple-View Geometric Model for Specularity Prediction on General Curved Surfaces" (2108.09378) introduces a robust, physically-motivated framework for predicting the behavior of specular highlights (henceforth specularities) on curved surfaces using only geometry and multiple views. By generalizing the previous JOLIMAS family of geometric models, the authors provide a formulation capable of accurate prediction of specularity shape and motion for arbitrarily curved surfaces, overcoming the limitations of uniform curvature assumptions.

Background and Problem Analysis

Specularity prediction is essential for high-fidelity rendering in AR, SLAM, and material acquisition. Specularities are strongly dependent not only on intrinsic material properties but also on local geometry, camera pose, and lighting. Previous approaches fall into three main categories:

• Global/local illumination models, which tend to be computationally expensive and data hungry, with poor generalization beyond explicit training environments.
• Deep learning models, which offer some flexibility but require large synthetic or annotated datasets and often lack physical interpretability.
• Geometric models, notably the JOLIMAS family, which utilize parameterized 3D ellipsoids to abstract and predict specularities based on multiple views.

However, as shown in the JOLIMAS model family, these geometric methods previously suffered from a reliance on planar or uniformly curved surfaces, drastically limiting real-world applicability. This is illustrated in the context of mirrored reflections and specular deformations with changing curvature.

Figure 2: Mirrored scenes reflected by surfaces of differing curvature, highlighting the complex transformation of specularities due to local geometry.

Dual and Canonical Dual JOLIMAS Models

Dual JOLIMAS

The predecessor, Dual JOLIMAS, employs virtual camera placements via reflection about the surface tangent at the brightest point, allowing ellipsoid reconstruction in non-planar but uniformly curved (e.g., spherical, cylindrical) settings. However, it fails to handle variation in curvature over the specularity region or during body deformations.

Figure 3: Dual JOLIMAS constructs virtual cameras for ellipsoid fitting but is limited to uniform surface geometry.

Canonical Dual JOLIMAS

This work introduces the Canonical Dual JOLIMAS model, leveraging the physics-based Torrance-Sparrow microfacet BRDF to bridge local curvature and specularity contour shape, thereby generalizing prediction capabilities to arbitrary surface geometry. The pivotal contribution is a mechanism by which specularity contours are mapped between deformed surfaces and a canonical tangent-plane representation via the notion of "limit angles," which are invariant with respect to local curvature. The model reconstructs a 3D ellipsoid in this canonical space and then warps predictions back to the current view and local geometry.

Figure 1: The Canonical Dual JOLIMAS pipeline, mapping observed ellipses from the curved surface to the tangent plane for robust ellipsoid reconstruction across arbitrary geometries.

Figure 6: Illustration of ellipse transformation from observed curved surface to the tangent plane at the brightest point, preserving limit angles.

Local Curvature and Field of View Effects

A distinguishing analysis in the paper is the rigorous examination of how local curvature affects the specularity's field of view and image deformation—a phenomenon not accounted for in other models.

Figure 7: Relationship between curvature and field of view, with convex surfaces expanding and concave reducing the visible area via reflected rays.

Figure 4: Synthetic illustration linking curvature, field of view, and specularity deformation, quantifying the mapping through canonicalization.

The authors establish that as curvature increases (becomes more convex), the reflected field of view widens, and specularity contours expand accordingly. This geometric mapping underpins the superior predictive performance for non-uniformly curved surfaces.

Empirical Validation and Numerical Results

Comprehensive experiments test the method on both synthetic and real sequences, with varying curvature and dynamic deformations.

• Synthetic experiment (plane to cylinder): Canonical Dual JOLIMAS maintains low prediction error (<2%) even when using a fixed initial ellipsoid as curvature changes. In contrast, Dual JOLIMAS error increases up to 12% as surface curvature diverges from the initialization scenario.

Figure 5: Quantitative results—Canonical Dual JOLIMAS (Exp 1 and Exp 2 in blue/red) exhibits robust accuracy across curvature changes, unlike Dual JOLIMAS (green).

• Non-uniform curvature (ellipsoid object): Canonical Dual JOLIMAS achieves average errors of 2.4%, while Dual JOLIMAS degrades to 20.2%.
• Real data (rocket, book with shiny cover, Kinect sequence): The model robustly predicts specularity deformations induced by real-world geometry and sensor noise, outperforming geometric-only baselines.

Bold claim: The canonicalization method achieves real-time prediction with substantial accuracy improvements, generalized for arbitrary surface variations and practical for live AR/SLAM pipelines.

Limitations

• Reliance on accurate geometry and normals: In practical AR/SLAM contexts, the method's performance is bounded by noise characteristics and resolution of the source geometry and normal maps (e.g., Kinect v2-induced jitter).
• Perfect mirrors and non-elliptical highlights: Assumption of elliptical specularity may break for highly anisotropic materials, surfaces with defects, or very high roughness. Some cases may require full 3D light source inverse reconstruction.
• Brightness estimation and image resolution: Localization of the brightest point and accurate contour mapping are both sensitive to sampling and field of view effects, though subpixel refinement mitigates some practical issues.

Implications and Future Work

This canonicalization strategy—projecting geometric cues and warping them back using physically-motivated models—serves as a template for handling other view-dependent appearance phenomena. The approach directly benefits AR rendering, relighting, inverse rendering, and dynamic retexturing tasks.

The authors suggest several future developments:

• Integration with more advanced BRDF models (e.g., GGX), expanding material generality
• Joint optimization of both specular and diffuse reflectance for complete photometric scene understanding
• Potential use as a data labeling tool to accelerate supervised learning for specularity detection and segmentation
• Robust temporal and spatial filtering strategies to address RGB-D sensor noise and further generalize to non-elliptical shapes

Conclusion

By incorporating differential geometry, multi-view cues, and the Torrance-Sparrow reflectance model, the paper delivers a unified, robust, and computationally-efficient solution for specularity prediction across general curved surfaces. The Canonical Dual JOLIMAS model combines geometric consistency with physically-based rendering principles, setting a platform for future advances in photorealistic AR and dynamic scene understanding (2108.09378).