IFMTBench: A Comprehensive Benchmark for Multilingual Translation Instruction Following

Abstract: Modern translation workflows demand more than semantic equivalence. Users routinely require models to preserve JSON or HTML schemas, honor curated glossaries, disambiguate with provided context, and match prescribed registers, often several at once. Conventional metrics such as BLEU and xCOMET capture semantic fidelity but provide little signal on constraint adherence, while general instruction following benchmarks ignore the cross-lingual nature of translation. We introduce \bench, a benchmark for multilingual translation instruction following covering seven languages, with 4,506 single-constraint and 2,838 multi-constraint items spanning six constraint dimensions and five compositional patterns with instructions issued in all seven languages. Constraints are split into a gating subset verified by deterministic checkers and a continuous subset scored by a rubric-based LLM judge, combined under a multiplicative rule that resists reward hacking. Evaluating 15 models reveals systematic gaps that prior protocols miss: Instruction following scales with size more sharply than translation quality, glossary and structured-format constraints dominate the difficulty gradient, and general instruction following rankings correlate only weakly with translation behavior. Our benchmark are available at https://github.com/Tencent-Hunyuan/Hy-MT2/tree/main/IFMTBench.

• The paper introduces IFMTBench, a benchmark evaluating translation models' instruction following with both deterministic and LLM-based scoring methods.
• It employs a three-stage data construction pipeline and a taxonomy of gating and continuous constraints across seven languages to mimic industrial workflows.
• Empirical results show that larger models excel in adhering to multiple constraints, though structural constraint enforcement remains a significant challenge.

IFMTBench: A Benchmark for Multilingual Translation Instruction Following

Motivation and Problem Formulation

Translation workflows in production environments increasingly demand constraint adherence beyond semantic equivalence. Typical translation requests often combine structural requirements (e.g., preserving JSON or HTML schemas), domain-specific glossaries, code retention, stylistic conformity, and context-sensitive disambiguation. Conventional metrics like BLEU and xCOMET generally fail to signal constraint violations, permitting outputs that are fluent yet unusable in downstream applications. General instruction following benchmarks are monolingual and ignore translation-specific constructs.

IFMTBench fills this gap by providing a comprehensive evaluation framework specifically targeting the instruction-following abilities of translation models across seven languages. It systematically formalizes translation constraints into deterministic gating subsets and continuous rubrics, applies a hybrid evaluation protocol, and decouples constraint following from translation quality.

Dataset Construction and Constraint Taxonomy

IFMTBench operationalizes translation constraints into two exclusive categories: gating constraints (binary, verifiable via deterministic checks) and continuous constraints (soft properties, scored via an LLM judge). The benchmark covers seven constraint dimensions: glossary, context, structured-data preservation, layout retention, code span maintenance, code-tag retention, and stylistic alignment. It comprises 4,506 single-constraint and 2,838 multi-constraint items, with multi-constraint patterns reflecting industrial translation workflows.

Data generation proceeds via a three-stage pipeline: meta-prompt synthesis conditioned on constraints, constraint-driven prompt generation, and expert review to ensure satisfaction and naturalness.

Figure 1: Three-stage data construction pipeline for IFMTBench, combining meta-prompt synthesis, constraint-driven generation, and expert review.

Each translation instruction is paraphrased into all seven languages, yielding parallel items that test models' multilingual instruction parsing. Source-target pairs span a balanced set of translation directions, maximizing linguistic diversity and stressing structural and lexical constraints across writing systems.

Evaluation Framework

IFMTBench's hybrid evaluation pipeline applies deterministic rule-based modules to gating constraints (structure, glossary, code, layout), while continuous constraints (style, context) are scored by rubric-conditioned LLM judgements on a normalized scale. Prompt-level instruction following score $IF$ is computed as the product of gating scores and the mean of continuous scores, enforcing hard penalties for structural failures that cannot be masked by high soft scores.

Figure 2: Hybrid evaluation pipeline combining rule validation with rubric-conditioned LLM judging, producing a multiplicative IF score per prompt.

Additionally, semantic translation quality is independently assessed using xCOMET-XXL, with both axes reported to capture quality/adherence trade-offs, and the evaluation protocol remains fully multilingual for both constraints and scoring.

Empirical Results

IFMTBench evaluates 15 models (commercial, general-purpose LLMs, and translation-specialized models) across parameter scales from 0.8B to 31B—using both dense and MoE architectures. Overall, instruction following scales more sharply with model size than does translation quality. For example, in the Qwen3.5 series, single-constraint IF rises from 40.92 (0.8B) to 82.48 (27B), while translation quality (xCOMET-XXL) increases only moderately. Multi-constraint scenarios amplify this gap, with small models failing composition while large models retain capacity.

Figure 3: Aggregate IF scores for 15 models, showing sharp scaling with parameter size and significant drop under multi-constraint composition.

Translation-specialized models (Hy-MT2) approached parity with closed-source systems (Gemini 3.1 Pro) under single constraints but trailed under composition, indicating an unresolved challenge in joint constraint tracking.

Dimension-Wise and Compositional Analysis

Per-dimension heatmaps reveal layout and code constraints to be nearly saturated—even for mid-sized models—whereas glossary and structure constraints are highly discriminative and challenging, especially under composition.

Figure 4: Per-dimension IF heatmaps for single and multi-constraint items, highlighting strong difficulty gradients across models and constraint types.

Multi-constraint composition disproportionately penalizes structural constraints, with structure dimension scores dropping more than style and glossary constraints. The multiplicative scoring rule ensures severe penalties for structural violation, aligned with practical use cases demanding integrity of output formats.

Figure 5: Degradation of IF under constraint composition, especially for structural constraints, across model scales.

Multilingual and Instruction-Language Effects

Instructions issued in different languages expose systematic performance gaps, particularly in small models. Large models with extensive multilingual pretraining demonstrate instruction-parsing robustness across languages, but low-resource/low-parameter models suffer both lower absolute scores and cross-language instability. Instruction following generalizes less uniformly than translation quality, narrowing only for models at scale.

Decoupling Quality and Instruction Following

Analysis of IF and xCOMET-XXL scores demonstrates that translation quality alone is a poor proxy for constraint adherence. Models may produce fluent outputs that ignore auxiliary instructions; conversely, frontier models may sacrifice fluency for strict adherence, which is desirable in industrial workflows. This decoupling underscores the necessity for dual-axis evaluation in practical pipeline deployment.

Comparative Benchmarking

Spearman rank correlations between IFMTBench and general instruction following benchmarks (IFEval, IFBench) are high across all models but drop significantly among top-tier models, indicating that parameter size dominates general benchmarks but fails to differentiate translation-specific instruction following in the regime relevant for ranking high-capability systems.

Implications and Future Directions

IFMTBench provides diagnostic granularity and compositional challenge in translation instruction following, critical for post-training reward modeling (GRPO, RLHF) in industrial settings. Structural constraint adherence remains a bottleneck in frontier models, motivating future research in constraint-aware post-training and reward signal engineering. Extension toward low-resource languages, richer constraint types (length, forbidden vocab), and adaptive item generation will further enhance benchmark utility. The multiplicative scoring rule’s direct compatibility with RL reward models suggests promising avenues for closing the evaluation-training loop.

Conclusion

IFMTBench offers a principled, multilingual, translation-specific benchmark for constraint following in translation models, with a taxonomy, compositional challenge, and dual-axis evaluation that exposes unaddressed gaps missed by conventional metrics and general instruction-following evaluations. Structural constraint handling and composition robustness remain open challenges that will shape future work in both model architecture and post-training protocol design.