MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task

Abstract: In this paper, we present the MetricX-24 submissions to the WMT24 Metrics Shared Task and provide details on the improvements we made over the previous version of MetricX. Our primary submission is a hybrid reference-based/-free metric, which can score a translation irrespective of whether it is given the source segment, the reference, or both. The metric is trained on previous WMT data in a two-stage fashion, first on the DA ratings only, then on a mixture of MQM and DA ratings. The training set in both stages is augmented with synthetic examples that we created to make the metric more robust to several common failure modes, such as fluent but unrelated translation, or undertranslation. We demonstrate the benefits of the individual modifications via an ablation study, and show a significant performance increase over MetricX-23 on the WMT23 MQM ratings, as well as our new synthetic challenge set.

• The paper introduces MetricX-24, a hybrid metric that combines reference-based and reference-free approaches for machine translation evaluation.
• It utilizes a two-stage fine-tuning of mT5 with Direct Assessment and MQM ratings, significantly improving accuracy across language pairs.
• Synthetic data augmentation is applied to enhance robustness, particularly in detecting undertranslation and fluency errors.

Overview of MetricX-24: Google Submission to WMT24 Metrics Shared Task

The paper "MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task" presents a detailed account of the development and evaluation of MetricX-24, a metric for assessing machine translation quality. This metric represents a progression over its predecessor, MetricX-23, and was submitted to participate in the WMT24 Metrics Shared Task. The primary feature of MetricX-24 is its hybrid architecture, enabling it to perform evaluations with and without reference translations. The key enhancements and innovative techniques devised for this iteration aim to improve robustness across multiple failure modes in machine translations.

Design and Innovation

MetricX-24 is built upon the mT5 LLM architecture, fine-tuned with both Direct Assessment (DA) and Multidimensional Quality Metrics (MQM) ratings. The fine-tuning is performed in two stages, with an initial focus on DA ratings to capture general trends, followed by an MQM-focused stage to enhance quality detection in translations. A significant departure from earlier variants is the hybrid model strategy, which integrates both reference-based and reference-free inputs to adjust more adeptly to diverse quality estimation tasks.

Hybrid Reference-Based/Free Model: This architecture allows MetricX-24 to flexibly score translations by including the source text and/or reference translations, thus accommodating scenarios where reference availability may vary. Notably, the model adapts to references of suboptimal quality by deriving scores solely from the source and candidate translation.

Synthetic Data Augmentation: The authors introduce synthetic data augmentation to bolster the metric’s ability to handle common translation challenges like undertranslation, fluent but unrelated translations, duplication, and missing punctuation. This synthetic approach addresses gaps in standard training datasets, leveraging a mixed training dataset to simulate problematic translation scenarios.

Performance Evaluation

MetricX-24 demonstrates strong results on several metrics. The inclusion of synthetic data resulted in notable improvements, particularly for identifying fluent but unrelated translation failures and undertranslation. The system-level pairwise accuracy shows substantial enhancement over MetricX-23 across diverse language pairs, with improvements of up to 11.5 percentage points in some cases, reflecting the effectiveness of the introduced enhancements.

The hybrid model, evaluated in both QE and reference-based scenarios, achieves a higher segment-level accuracy on challenging language pairs, indicating its capacity to robustly handle translations with poor references. The boost in performance on lower-quality translations underscores the metric's expanded capability, aided by the DA and MQM mixture training strategy.

Implications and Future Directions

MetricX-24's architecture and training innovations contribute significantly to the field of machine translation evaluation by mitigating deficiencies in existing metrics. Employing a hybrid model and fine-tuning with diverse annotations address practical challenges in multilingual evaluation contexts. These improvements suggest that enhanced machine translation metrics can not only provide more reliable scores but also better capture subtler quality differences without relying strictly on reference translations.

Future research could investigate more sophisticated approaches for synthetic example generation, as well as explore broadening language pair coverage in MQM data. Similarly, extending robustness through adaptive learning algorithms could play a crucial role in interpreting ambiguous translation contexts. The hybrid model approach presents a promising direction, potentially applicable to other NLP benchmarking tasks where varying data constraints exist.

Conclusion

In conclusion, MetricX-24 represents a substantive advancement in machine translation quality assessment, effectively bridging gaps left by previous approaches. Its capacity to interpret multiple input modalities and enhanced resilience against typical failure modes marks a significant contribution to developing reliable, adaptable machine translation metrics. With such developments, it is poised to influence the next generation of translation evaluation methodologies, shaping future AI-driven language technologies.