Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory

Abstract: We address a fundamental challenge in Natural Language Generation (NLG) model evaluation -- the design and evaluation of evaluation metrics. Recognizing the limitations of existing automatic metrics and noises from how current human evaluation was conducted, we propose MetricEval, a framework informed by measurement theory, the foundation of educational test design, for conceptualizing and evaluating the reliability and validity of NLG evaluation metrics. The framework formalizes the source of measurement error and offers statistical tools for evaluating evaluation metrics based on empirical data. With our framework, one can quantify the uncertainty of the metrics to better interpret the result. To exemplify the use of our framework in practice, we analyzed a set of evaluation metrics for summarization and identified issues related to conflated validity structure in human-eval and reliability in LLM-based metrics. Through MetricEval, we aim to promote the design, evaluation, and interpretation of valid and reliable metrics to advance robust and effective NLG models.

• The paper introduces MetricEval, a measurement theory-based framework that rigorously evaluates NLG metric reliability and validity using statistical tools.
• It quantifies key components such as metric stability, consistency, concurrent validity, and construct validity, providing detailed insights via a summarization case study.
• Results show that metrics like G-Eval with GPT-4 offer high reliability, while others like ROUGE-4 exhibit lower consistency, emphasizing the need for refined metric design.

Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory

The paper "Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory" (2305.14889) introduces MetricEval, a framework intended to scrutinize the reliability and validity of Natural Language Generation (NLG) evaluation metrics. This framework draws on principles from measurement theory, focusing on Metric Stability, Metric Consistency, Metric Construct Validity, and Metric Concurrent Validity. These components are rigorously quantified through statistical tools to provide more meaningful interpretations of evaluation results. Here's a detailed breakdown of the framework's components and application:

MetricEval Framework Components

Reliability

Reliability in the context of NLG metrics refers to their ability to provide consistent and dependable results across different implementations and evaluations. It addresses two main aspects:

• Metric Stability: This measures the ability of a metric to yield consistent results when re-evaluating the same model outputs. Non-deterministic algorithms, such as those found in some LLM-based metrics (e.g., G-Eval 3.5 and G-Eval 4), may show fluctuations, thus affecting stability.

  Figure 1: MetricEval: A framework that conceptualizes and operationalizes four main components of metric evaluation, in terms of reliability and validity.

• Metric Consistency: This assesses how metric scores fluctuate with different datasets, aiming to measure intrinsic reliability across various data subsets. Statistical tools like the coefficient $\alpha$ provide a means to estimate this reliability coefficient.

  Figure 2: Estimated Metric Stability and Metric Consistency of popular NLG Metrics.

Validity

Validity examines whether metrics measure what they purport to measure and their applicability in making accurate inferences:

• Metric Concurrent Validity: Evaluated by comparing a metric against a validated criterion (often human judgments). This validity component is analyzed by examining correlation coefficients (e.g., Kendall's tau) between metrics such as BARTScore and expert ratings.

  Figure 3: Concurrent validity coefficients of the selected metrics in predicting the four expert-rated dimensions' factor scores. Values are based on Kendall's tau.

• Metric Construct Validity: It investigates a metric's ability to measure theoretical constructs (like coherence or fluency in generated text) through Multitrait-Multimethod (MTMM) analysis and factor analysis.

Case Study on Summarization Metrics

The paper illustrates the application of MetricEval through a case study on the SummEval dataset, consisting of summaries scored across 16 metrics. This study provided insights into both the stability and consistency of metrics.

Key Findings

• Metric Reliability: Metrics such as G-Eval with GPT-4 showcased higher stability. However, metrics using longer n-grams (e.g., ROUGE-4) demonstrated reduced metric consistency.
• Construct Validity Issues: A notable observation was the conflated validity in expert ratings of Coherence versus Relevance.
• Metric Validity Insights: Certain metrics consistently aligned with expert judgments (e.g., G-Eval and BARTScore), yet showed little differentiation across distinct dimensions, highlighting a need for refinements in prompts and metric design.

  Figure 4: Metric stability and consistency estimates for all expert- and metric-based scores.

Conclusion

MetricEval provides a comprehensive approach to evaluating NLG metrics' reliability and validity. By integrating measurement theory principles, it enables robust assessments of new and existing metrics' effectiveness, ultimately guiding improvements in NLG model evaluation practices. Future efforts could focus on expanding the framework to account for generalizability across diverse benchmarks and further refining evaluation criteria for distinct tasks.