DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models

Abstract: 2D diffusion model, which often contains unwanted baked-in shading effects and results in unrealistic rendering effects in the downstream applications. Generating Physically Based Rendering (PBR) materials instead of just RGB textures would be a promising solution. However, directly distilling the PBR material parameters from 2D diffusion models still suffers from incorrect material decomposition, such as baked-in shading effects in albedo. We introduce DreamMat, an innovative approach to resolve the aforementioned problem, to generate high-quality PBR materials from text descriptions. We find out that the main reason for the incorrect material distillation is that large-scale 2D diffusion models are only trained to generate final shading colors, resulting in insufficient constraints on material decomposition during distillation. To tackle this problem, we first finetune a new light-aware 2D diffusion model to condition on a given lighting environment and generate the shading results on this specific lighting condition. Then, by applying the same environment lights in the material distillation, DreamMat can generate high-quality PBR materials that are not only consistent with the given geometry but also free from any baked-in shading effects in albedo. Extensive experiments demonstrate that the materials produced through our methods exhibit greater visual appeal to users and achieve significantly superior rendering quality compared to baseline methods, which are preferable for downstream tasks such as game and film production.

• The paper introduces DreamMat, a novel approach that uses geometry- and light-aware diffusion models to accurately decompose PBR materials into albedo, roughness, and metalness.
• The method leverages a randomized HDR lighting context and classifier score distillation loss to minimize shading artifacts and ensure consistency with light conditions.
• Experimental results show higher CLIP scores and lower FID metrics than previous methods, demonstrating superior semantic alignment and visual realism in material generation.

DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models

Introduction

The paper "DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models" addresses the challenges of generating photo-realistic Physically Based Rendering (PBR) materials from textual descriptions. Conventional 2D diffusion models often integrate unwanted shading effects into RGB textures, resulting in unrealistic renderings. To overcome these challenges, the authors propose DreamMat, a novel approach that utilizes geometry- and light-aware diffusion models to ensure high-quality PBR materials generation.

Problem and Approach

The primary issue with existing methods is their focus on generating final shading colors rather than accurately decomposing materials into distinct PBR parameters like albedo, roughness, and metalness. The authors identify that mainstream 2D diffusion models lack sufficient constraints for material decomposition due to their training on final shading colors alone.

To mitigate this, DreamMat introduces the following key innovations:

1. Light-aware Diffusion Model: The diffusion model is finetuned to consider a specified lighting environment. This ensures that generated textures align with given lighting conditions, reducing baked-in shading effects.
2. Random Lighting Context: The distillation process incorporates random selection from a set of predefined High-Dynamic-Range (HDR) images, guiding material generation to focus on consistent geometry and light conditions.

Implementation Details

DreamMat leverages an inverse rendering-based approach using hash-grid-based representation to model SVBRDF, i.e., materials are computed and rendered using Monte Carlo sampling. The training involves optimizing material representation via a distillation loss paradigm. A key technical contribution is the finetuning of the Stable Diffusion model to be geometry- and light-aware using such conditions to accurately predict the appearance.

• Material Representation: The SVBRDF is represented and optimized in a hash-grid format, which encodes albedo, roughness, and metallic properties.
• Training Strategy: The pipeline utilizes Classifier Score Distillation (CSD) loss to iteratively improve the generated materials, focusing on alignment between rendered images and desired prompts under multiple lighting conditions.
• Computational Tools: High-performance computation is achieved using ThreeStudio and large-scale GPU resources for training the ControlNet, which integrates both geometric and light conditions.

Experimental Results

The paper provides extensive experimental validation, showcasing superior performance over techniques like TEXTure, Fantasia3D, and others. The results highlight DreamMat's ability to efficiently generate detailed, realistic PBR materials while maintaining consistency under varied lighting conditions.

• Qualitative Comparisons: DreamMat produces textures that are visually appealing, with enhanced fidelity to geometric structures and environmental lighting, as compared to prior methods.
• Quantitative Metrics: The method achieves higher CLIP scores and lower FID compared to competitors, indicating better semantic alignment and visual quality.
• User Studies: Feedback from studies demonstrates a preference for DreamMat-generated materials in terms of overall quality, text fidelity, and realistic rendering capabilities.

Limitations and Future Work

Despite its advancements, DreamMat has limitations in handling materials with complex physical interactions like transparency and subsurface scattering. Additionally, the computational cost of distillation presents challenges for real-time applications. Future work may explore optimizing indirect lighting effects and reducing computational overhead for broader applicability.

Conclusion

DreamMat represents a significant step forward in the automated generation of high-quality, realistic PBR materials using diffusion models. By integrating geometry and light awareness into the distillation process, DreamMat advances the state of the art in computer graphics, making it a valuable tool for applications in gaming, film, and virtual reality production.