Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets

Abstract: The reliability of multilingual LLM evaluation is currently compromised by the inconsistent quality of translated benchmarks. Existing resources often suffer from semantic drift and context loss, which can lead to misleading performance metrics. In this work, we present a fully automated framework designed to address these challenges by enabling scalable, high-quality translation of datasets and benchmarks. We demonstrate that adapting test-time compute scaling strategies, specifically Universal Self-Improvement (USI) and our proposed multi-round ranking method, T-RANK, allows for significantly higher quality outputs compared to traditional pipelines. Our framework ensures that benchmarks preserve their original task structure and linguistic nuances during localization. We apply this approach to translate popular benchmarks and datasets into eight Eastern and Southern European languages (Ukrainian, Bulgarian, Slovak, Romanian, Lithuanian, Estonian, Turkish, Greek). Evaluations using both reference-based metrics and LLM-as-a-judge show that our translations surpass existing resources, resulting in more accurate downstream model assessment. We release both the framework and the improved benchmarks to facilitate robust and reproducible multilingual AI development.

• The paper introduces a robust pipeline (RITRANS) that uses USI and T-RANK methods to preserve translation context and grammatical dependencies.
• It details innovative translation ranking and self-improvement strategies that significantly outperform traditional best-of-N approaches in multilingual evaluation.
• Empirical results show improved COMET scores and enhanced evaluation reliability across diverse languages and structured benchmark formats.

Automated Translation of Multilingual Benchmarks: The RITRANS Pipeline

Motivation and Critical Analysis of Multilingual Benchmark Translation

The integrity of multilingual LLM evaluation hinges on the fidelity of localized benchmarks. Prior resources, including MuBench, Global-MMLU, and Okapi, relied substantially on dated MT tools or early-generation LLMs lacking refined instruction following and context preservation, resulting in semantic drift, answer leakage, and context-agnostic translation artifacts. The presented work identifies grammatical incongruities and semantic loss as the principal failure modes across mid-resource Eastern and Southern European target languages. These languages are characterized by dense inflectional morphology and context-sensitive grammatical features, making naïve or one-prompt automated translation strategies inadequate for robust benchmark transfer.

Existing pipelines often translate questions and answers independently, neglecting grammatical dependencies and thus jeopardizing benchmark reliability. Particularly in sentence completion tasks, answer leakage through declension and gender marking or unnatural narrative was frequently observed. These limitations expose not only the need for improved translation pipelines but also for adaptive strategies that respect idiosyncratic linguistic phenomena and preserve the task's logical and evaluative structure.

The RITRANS Framework: Architecture and Translation Methods

The proposed pipeline is a fully automated, configurable system supporting diverse translation modes—self-check (SC), Best-of-N (BoN), Universal Self-Improvement (USI), and the newly introduced multi-round Translation Ranking (T-RANK)—accommodating both classic datasets and complex QA benchmark formats. The system differentiates between simple string-to-string translation (datasets) and the structured translation of interdependent prompts and answer options (benchmarks), preserving intra-item relationships.

USI (Universal Self-Improvement) ensembles high-temperature translation candidates sampled from various prompt templates, subsequently synthesizing an improved output through a refinement LLM that selectively merges the strengths of each variant.

Figure 1: Universal Self-Improvement (USI) method workflow, ensembling multi-prompt sampled candidates and aggregating their merits into a finalized translation.

T-RANK (Translation Ranking) implements test-time compute scaling through competitive, multi-round ranking. Each translation candidate occupies all possible ranking positions via systematic permutation, significantly mitigating positional scoring bias. Successive rounds incrementally expose candidates to the LLM judge's discriminatory capabilities. The final candidate receives a targeted correction and error-culling pass informed by the comparative weaknesses of the alternatives.

Figure 2: Translation Ranking (T-RANK) method workflow, visualizing the candidate sampling, multi-round positional rotation, and competitive ranking with final candidate refinement.

Unlike prior best-of-N and fusion methods, T-RANK explicitly addresses judge model position bias and error entrenchment, surfaces subtle translation errors, and prompts for context-preserving corrections. This proves especially effective for languages or benchmark items where grammatical cohesion is nontrivial.

Empirical Results and Method Comparison

Experiments cover mass translation of MMLU, Hellaswag, ARC, and Winogrande into Ukrainian, Romanian, Slovak, Lithuanian, Bulgarian, Turkish, Greek, and Estonian. Quantitative assessments via COMET (Unbabel/XCOMET-XL, wmt23-cometkiwi-da-xl) and LLM-as-a-judge protocols provide complementary perspectives.

On standard MT sets (WMT24++, FLORES), both T-RANK and USI perform strongly, with COMET scores exhibiting consistent improvements over the baseline and classical best-of-N pipelines (e.g., USI reaching 0.843/0.945 vs baseline 0.827/0.937 for WMT24++/FLORES EN→UK). For the translation of longer, contextually complex items (i.e., benchmarks), T-RANK demonstrates diagnostic advantage in error elimination and reference preservation, resulting in more accurate and faithful localizations.

Judging with state-of-the-art LLMs in a pairwise protocol, both T-RANK and USI translations substantially outperform Global-MMLU and Okapi benchmarks—e.g., for Ukrainian MMLU, T-RANK translations win 8750 comparisons, draw 3276, and lose 2016 versus the previous standard. Model performance on translated benchmarks consistently improves: e.g., Winogrande shows +3.42% average improvement, with gains up to +3.89% for Greek, confirming that benchmark translation quality directly affects evaluation reliability of LLMs.

Methodological Innovations: Mitigating Risks and Biases

A core contribution of RITRANS is an ablation and mitigation strategy for known LLM evaluation biases, including positional bias in competitive scoring. The T-RANK framework ensures full coverage of positional permutations while optimizing for model call efficiency, allocating $2N+1$ calls per example. Prompts and evaluation templates are tailored to enforce domain-appropriate preservation, inflectional consistency, and disallow code translation errors. Configuration flexibility enables language-specific prompt engineering and method switching, a necessity for addressing divergent error distributions across benchmarks and target languages.

Empirical analysis confirms that USI is best suited for short, flat dataset items, while T-RANK excels for QA and highly structured benchmarks. COMET-based quality estimation highlights method sensitivities to item length and domain, though the authors note the metric's own limitations and emphasize the value of LLM-based and human-in-the-loop quality assessment for complex linguistic structures.

Theoretical and Practical Implications

This pipeline marks a shift from static, one-pass translation strategies towards dynamic, scalable, and evaluative self-improvement in automated benchmark localization. It operationalizes recent advances in test-time compute scaling and LLM-as-a-judge assessment, facilitating credible multilingual evaluation without the steep costs and practical limitations of manual annotation or full human verification.

Practically, the system enables rapid, high-quality expansion of multilingual evaluation resources for mid-resource languages with minimal expert intervention. The release of the framework and translated benchmarks under open licenses mitigates reproducibility and accessibility concerns.

Theoretically, these findings reinforce that translation fidelity is crucial for valid cross-lingual generalization claims. Inaccurate or contextually naive translations compromise not just reported LLM performance figures but also obscure actual cross-lingual reasoning, bias, and transfer phenomena. Methodological choices in automated translation can introduce or mask answer leakage, adversarial artifacts, or bias, directly affecting both aggregate scores and error analyses central to model development cycles.

Future Directions

Potential improvements include automatic per-instance method selection based on translation difficulty estimates, integration of robust reference-free quality metrics, and broader evaluation on open-weight and low-resource translation models. Prioritizing adaptation to even lower-resource settings and further mitigating LLM-as-a-judge biases will be necessary to generalize these advances beyond the Eastern and Southern European focus presented.

Conclusion

Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets (2602.22207) introduces a new, rigorous paradigm for automated multilingual benchmark translation, providing substantive gains in translation integrity, reliability, and evaluation fidelity. The configuration-rich pipeline, leveraging USI and T-RANK, is empirically shown to outperform prior state-of-the-art translation strategies by substantial margins across multiple benchmarks and languages, as confirmed by both automated and LLM-based evaluative metrics. The framework positions itself as the reference system for the scalable and accurate localization of LLM benchmarks, essential for both fair multilingual evaluation and robust future AI deployment.