Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation

Abstract: It has been recently shown that neural networks can recover the geometric structure of a face from a single given image. A common denominator of most existing face geometry reconstruction methods is the restriction of the solution space to some low-dimensional subspace. While such a model significantly simplifies the reconstruction problem, it is inherently limited in its expressiveness. As an alternative, we propose an Image-to-Image translation network that jointly maps the input image to a depth image and a facial correspondence map. This explicit pixel-based mapping can then be utilized to provide high quality reconstructions of diverse faces under extreme expressions, using a purely geometric refinement process. In the spirit of recent approaches, the network is trained only with synthetic data, and is then evaluated on in-the-wild facial images. Both qualitative and quantitative analyses demonstrate the accuracy and the robustness of our approach.

• The paper introduces a novel framework that leverages image-to-image translation for reconstructing detailed 3D facial geometry.
• It employs synthetic facial data and a bidirectional mapping strategy to deliver high-fidelity depth and correspondence maps.
• The method demonstrates improved reconstruction accuracy and robustness in non-frontal views with applications in AR, animation, and medical simulations.

Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation: An Analysis

The paper "Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation" presents a novel approach to reconstructing 3D facial geometries from 2D images. The methodology addresses limitations inherent to low-dimensional subspace restrictions commonly used in existing facial geometry reconstruction models. The authors leverage an Image-to-Image translation neural network framework to extract depth images and densely correspond facial map data directly from input images. This work stands out by employing synthetic data both for training and to test "in-the-wild" images, reflecting robustness and versatility in real-world applications.

Methodological Advances

Key to this research is the avoidance of confinement to a predefined low-dimensional model space, which traditionally limits the expressiveness and accuracy of reconstructed geometries. The authors implement a fully convolutional network that learns to translate input images into two outputs: a depth image and a facial correspondence map. This representation method allows handling detailed reconstructions across diverse facial expressions and conditions. The model's strength is in its ability to extrapolate geometric structures beyond the scope of its training data, which is achieved through several innovative strategies:

1. Synthetic Data Utilization: The training is conducted using synthetic datasets derived from facial morphable models. These datasets feature a wide variety of facial identities, expressions, poses, lighting conditions, and textures.
2. Bidirectional Mapping: The neural network predicts depth and correspondence maps that align with a reference mesh template. This enables a direct comparison of each pixel's geometric features, which is pivotal in refining reconstructions from extreme expressions or poses.
3. Refinement and Deformation Process: The novel approach blends neural techniques with geometric refinement. Following the initial network output, the model undergoes a non-rigid deformation process and a level of fine detail recovery, allowing for high-quality mesoscopic detail reconstruction.
4. Application of Norm-Based Loss: To enhance depth map estimate accuracy, the paper proposes the application of a norm-based loss function that complements the primary L1 loss, proving greater adeptness in finer details generation.

Experimental Observations and Results

Qualitative and quantitative assessments in the research exhibit robust reconstruction capabilities. The network effectively discriminates facial structures even under noisy data conditions. The proposed method achieves lower absolute reconstruction errors on benchmark datasets compared to contemporary techniques, demonstrating superior capacity to maintain geometric fidelity across wide-ranging scenes. Heatmaps and visual comparisons indicate improved handling of non-frontal views and intricate detail such as facial contours, a limitation for prior state-of-the-art models.

Implications and Future Work

Practically, the methods presented hold significance in a variety of fields such as animated filmmaking, augmented reality, and facial recognition systems. Moreover, the paper identifies potential in fields requiring high-fidelity geometric reconstructions like medical simulations and custom prosthetic design. Additionally, employing a fully learned system bypasses the complexities associated with iterative feature detection methods, encouraging potential ease of scalability and application in real-time systems.

Looking forward, extending this research may involve enhancing the network's resilience to occlusions and further reducing processing times in the registration phase. Future developments may explore the integration of real-world training datasets to complement synthetic data robustness, providing a more comprehensive understanding of environmental variables in facial geometric reconstruction. The integration of this model with advanced GAN architectures might also serve to maintain more consistent geometric fidelity while translating between domains.

In summary, this paper significantly enriches the field of facial geometry reconstruction by advancing the premise that unrestricted, pixel-level map generation integrated with geometric refinement can overcome conventional subspace limitations, delivering accurate, high-fidelity, and application-ready 3D facial models.