Evaluation Metric for Quality Control and Generative Models in Histopathology Images

Abstract: Our study introduces ResNet-L2 (RL2), a novel metric for evaluating generative models and image quality in histopathology, addressing limitations of traditional metrics, such as Frechet inception distance (FID), when the data is scarce. RL2 leverages ResNet features with a normalizing flow to calculate RMSE distance in the latent space, providing reliable assessments across diverse histopathology datasets. We evaluated the performance of RL2 on degradation types, such as blur, Gaussian noise, salt-and-pepper noise, and rectangular patches, as well as diffusion processes. RL2's monotonic response to increasing degradation makes it well-suited for models that assess image quality, proving a valuable advancement for evaluating image generation techniques in histopathology. It can also be used to discard low-quality patches while sampling from a whole slide image. It is also significantly lighter and faster compared to traditional metrics and requires fewer images to give stable metric value.

• The paper presents RL2, a novel metric that computes the RMSE between latent vectors extracted from a ResNet-18 using normalizing flows.
• It leverages domain-specific feature extraction and computational efficiency to accurately assess image quality under various noise conditions.
• Empirical results demonstrate RL2’s robustness with fewer samples and superior performance compared to traditional metrics like FID and IS.

Evaluation Metric for Quality Control and Generative Models in Histopathology Images

Introduction

Generative models, including GANs, VAEs, and diffusion models, are pivotal in histopathology, addressing data scarcity challenges through data augmentation, anomaly detection, and synthetic image generation. These models enhance diagnostic accuracy and data efficiency, a core necessity in medical imaging. Yet evaluating their quality remains problematic, given the subjective nature of image quality metrics, demanding computational efficiency in resource-constrained environments. Traditional metrics such as FID, IS, KID, and CMMD often require large image datasets, scarce in the medical domain, and depend on features derived from networks pre-trained on natural images, leading to ineffective assessments in medical imaging domains like histopathology where images differ significantly from natural scenes.

ResNet-L2 Overview

ResNet-L2 (RL2) introduces a compelling approach by utilizing normalizing flows applied to features extracted from a pre-trained ResNet-18, focusing on resolving dimensionality challenges and computational inefficiencies associated with traditional methods. RL2 computes the RMSE distance between latent vectors of real and generated images, leveraging the ResNet-18's lighter architecture and domain specificity, unmatched by previous approaches relying on less adaptable ImageNet-trained networks.

Figure 1: The process of computing RL2, encompassing training on high-quality images and evaluation using L2 distance between latent vectors.

Methodology

Implementation

The RL2 metric is computed by initially processing high-quality real images through a ResNet-18 to extract image features, subsequently subjected to a normalizing flow. Training optimizes the likelihood of real image features, ensuring higher probabilities are ascribed to domain-specific histopathology images, enhancing adaptive accuracy beyond traditional metrics.

Post-training, the method involves computing a stable metric by accumulating latent vectors from real and generated images, applying Euclidean distance to quantify image quality disparities efficiently. RL2 optimizes computational efficacy, demanding fewer resources and achieving significant state-of-the-art metrics’ stability using fewer samples, a breakthrough for domains constrained by limited data availability.

Results

Across diverse histopathology datasets, RL2 demonstrated robust performance. It exhibited monotonic responses to increasing noise levels, including blur, salt-and-pepper, and patch noises, essential for precise evaluation of image degradation quality. Empirical results underscored RL2’s capabilities, showcasing reduced computational needs while maintaining accuracy and stability across varying noise types and diffusion processes. Notably, RL2 achieved a stable value with 300 samples, contrasting significantly from traditional FID metrics, which necessitate thousands of samples for consistent performance.

Figure 2: The behavior of RL2 with increasing blur levels, indicating its efficacy in distinguishing image quality.

Figure 3: The robustness of RL2 when applied to salt-and-pepper noise, highlighting its capacity to manage image variability consistently.

Figure 4: Demonstrating RL2’s response to patch noise levels, underscoring its adaptability across diverse histopathology imagery.

Discussion

The findings underscore RL2’s potential as an efficient, scalable alternative for evaluating generative models within the histopathology domain. Its sample efficiency, domain adaptability, and computational advantages present a significant stride in optimizing image quality assessments, meeting the domain's unique demands effectively. Despite compelling performance, RL2’s adaptation to unfamiliar artifacts requires refinement, as evidenced by moderate AUC results in artifact classification tasks.

Figure 5: RL2’s monotonic response to diffusion noise levels, affirming its competency in assessing quality improvements in generative models.

Conclusions

ResNet-L2 addresses significant shortcomings in existing metrics, presenting a feasible, domain-specific method for histopathology image evaluation. Its resource-efficient computation, robustness to diverse noise types, and significant adaptability makes RL2 a valuable tool for diverse generative model assessments, especially when data scarcity imposes constraints on traditional metric application. Future developments could focus on refining RL2’s adaptation to unfamiliar artifacts and exploring its integration within broader medical imaging quality control frameworks.