Bilingual Evaluation Dataset

• Bilingual evaluation datasets are computational benchmarks that assess system performance across two languages using parallel or comparable data.
• They are constructed via multi-stage pipelines involving authentic data sourcing, expert translation, and stringent quality control protocols.
• These datasets support evaluations of cross-lingual tasks such as lexical similarity, translation, dialogue, and domain-specific reasoning, driving robust model development.

A bilingual evaluation dataset is a computational benchmark designed to assess system performance or linguistic phenomena across two languages, typically with parallel data or harmonized task definitions and metrics. Such datasets are foundational for probing cross-lingual transfer, lexical representation, reasoning, comprehension, generation, or domain-specific ability under truly multilingual conditions. Recent research demonstrates that the rigor and diagnostic value of bilingual evaluation datasets depend on precise multilingual curation, stringent validation protocols, and careful analysis of comparative system performance across languages.

1. Foundational Concepts and Dataset Typologies

Bilingual evaluation datasets span a wide spectrum of linguistic units, domains, and evaluation purposes. Canonical examples cover:

• Lexical-level evaluation: Datasets like MUSE for bilingual dictionary induction (BDI), which map word translation pairs across languages for intrinsic assessment of cross-lingual embeddings (Kementchedjhieva et al., 2019), and BCWS for contextual word similarity across sense units (Chi et al., 2018).
• Sentence/document-level tasks: Parallel evaluation sets for translation, e.g., DiaBLa’s English–French spontaneously written dialogues (Bawden et al., 2019), or policy-related corpora such as POLIS-Bench’s Chinese–English governmental texts (Yang et al., 4 Nov 2025).
• Conversational and dialogue datasets: BiToD, a multi-domain English–Chinese task-oriented dialogue set for joint dialogue state tracking, natural language generation, and API interaction, with explicit bilingual knowledge bases (Lin et al., 2021), and NormDial, a synthetic English–Chinese dataset probing dialogic social norm adherence (Li et al., 2023).
• Domain-specific, multi-task and reasoning datasets: BMMR (ZH–EN) for multimodal, multidisciplinary college-level reasoning (Xi et al., 4 Jul 2025), ScholarBench (EN–KO) for academic abstraction and problem-solving (Noh et al., 22 May 2025), and LC-Eval (EN–AR) for long-context reading comprehension and reasoning (Jubair et al., 19 Oct 2025).
• Task-specialized and technical benchmarks: CodeApex (ZH–EN) for programming comprehension, code generation, and correction (Fu et al., 2023), and StatBot.Swiss (EN–DE) for Text-to-SQL in statistical open data exploration (Nooralahzadeh et al., 2024).
• Safety-critical and low-resource evaluations: Qorgau, focused on LLM safety in Kazakh–Russian code-switched risk settings (Goloburda et al., 19 Feb 2025), and clinical data relation extraction in English–Turkish (Aidynkyzy et al., 14 Jan 2026).

Benchmarks are often “fully parallel” (identical content in both languages), “comparable” (analogous but not identical test sets), or “partially parallel” (significant overlap, subsets in parallel).

2. Construction Principles and Curation Protocols

Dataset construction leverages methodical multi-stage pipelines to ensure bilingual alignment, domain relevance, and annotation consistency:

• Source collection: Extraction from authentic corpora (Wikipedia, academic literature, open government, user-generated data) or controlled corpus generation via expert prompt engineering (e.g., CodeApex uses code exam archives and online judge logs (Fu et al., 2023); NormDial constructs synthetic scenarios with expert supervision (Li et al., 2023)).
• Parallelization & Translation: Human translation and expert post-editing dominate high-stakes or nuanced tasks (see UNED-ACCESS 2024, with manual Spanish–English translation and round-trip verification (Salido et al., 2024)).
• Annotation and quality control: Multi-annotator protocols, leave-one-out validation (BCWS), specialist review (medical RE (Aidynkyzy et al., 14 Jan 2026)), and spot-checks for cultural and legal adaptation (Qorgau (Goloburda et al., 19 Feb 2025)).
• Category and attribute mapping: Use of fine-grained typologies (e.g., 63–65 rhetorical “attributes” in ScholarBench (Noh et al., 22 May 2025)), and scenario-grounded task designs (POLIS-Bench’s three distinct governmental policy tasks (Yang et al., 4 Nov 2025)).

Tabular summary of notable bilingual dataset statistics:

Dataset	Languages	Instances	Domain/Tasks
MUSE BDI	5 pairs (de, da,	~1500/tst	Word translation (lexicon induction, embeddings)
BCWS	en–zh	2091	Contextual word similarity
BiToD	en–zh	7.2k dia	Multidomain task-oriented dialogue (ToD)
CodeApex	en–zh	2056	Programming: comp., gen., correction
StatBot.Swiss	en–de	455	Text-to-SQL, statistics exploration
ScholarBench	en–ko	10k+	Academic comprehension and reasoning
BMMR-Eval	en–zh	20,458	Multimodal multidisciplinary reasoning
Qorgau	kk–ru	8.1k+	LLM safety (bilingual, code-switch)
POLIS-Bench	en–zh	3,058	Policy clause, solution, compliance
LC-Eval	en–ar	7,903	Long-context QA, claim verification, MCQ

3. Evaluation Tasks, Metrics, and Error Analysis

Bilingual evaluation datasets assess both standard NLP tasks and “cross-lingual” transfer phenomena using domain-appropriate and language-sensitive metrics.

• Lexical tasks: Precision@k; Accuracy@1 for BDI (Kementchedjhieva et al., 2019):

$$\mathrm{P}@k = \frac{1}{N}\sum_{w \in D}\mathbb{I}\left(\exists i \leq k{\,\hat t_i(w) \in G(w)}\right)$$

Spearman’s rank correlation for word similarity (BCWS (Chi et al., 2018)).

• Comprehension, QA, and generation:

MCQ accuracy, Cohen’s Kappa (to adjust for chance, as in UNED-ACCESS 2024 (Salido et al., 2024)); CodeApex uses pass rates on input test cases for code generation; Text-to-SQL benchmarks leverage execution-based accuracy (strict, soft, partial; (Nooralahzadeh et al., 2024)).

• Dialogue and social norm adherence:

Macro-/micro-average $F_1$ for norm-observance detection (NormDial (Li et al., 2023)); BLEU for NLG quality when translation references are available (DiaBLa (Bawden et al., 2019), BiToD (Lin et al., 2021)).

• Safety/risk evaluation:

Safe-rate, precision, recall, $F_1$ for binary harmfulness judgments in LLM safety (Qorgau (Goloburda et al., 19 Feb 2025)), with independent per-risk-area reporting.

• Process and reasoning evaluation:

Chain-of-thought verifier step-/response-level accuracy (BMMR-Verifier (Xi et al., 4 Jul 2025)); semantic similarity (sentence embedding cosine) and LLM-judge accuracy for policy compliance (POLIS-Bench (Yang et al., 4 Nov 2025)).

Qualitative error analysis is a crucial supplement. For instance, missing gold targets in MUSE BDI artificially inflate performance gaps between systems—e.g., correcting missing inflectional variants reduced a 6.7% margin to 0.1% in English-to-Bulgarian BDI (Kementchedjhieva et al., 2019). Similarly, language-specific failure modes, such as degraded safety under code-switching (Qorgau), inform model deployment.

4. Representative Datasets and Use Cases

Several recent datasets exemplify rigorous bilingual evaluation design:

• MUSE BDI (Kementchedjhieva et al., 2019): Largest publicly used resource for word translation evaluation; affected by high rates of proper nouns and missing targets, resulting in unstable system rankings.
• BCWS (Chi et al., 2018): 2,091 English–Chinese word pairs with sentential context and human similarity scores; Spearman’s ρ upper bound from human agreement is 0.83.
• BiToD (Lin et al., 2021): 7,232 dialogues, bilingual knowledge base; evaluates end-to-end dialogue, cross-lingual transfer, and domain coverage with joint goal accuracy and API call precision.
• ScholarBench (Noh et al., 22 May 2025): 5,309 English and 5,031 Korean expert-generated academic reasoning problems; broad domain and attribute coverage; metrics span ROUGE scores, accuracy, BERTScore.
• CodeApex (Fu et al., 2023): 2,056 bilingual programming tasks (comprehension, generation, correction); uses human translation, split by language and task type.
• UNED-ACCESS 2024 (Salido et al., 2024): 1,003 Spanish/English MCQs; strict contamination control; kappa for language-normalized model comparison.
• Qorgau (Goloburda et al., 19 Feb 2025): 8,169 prompts plus code-switch subset; diagnostic for LLM safety and sociopolitical risk in Kazakh–Russian.

Applications extend across: evaluating and training robust cross-lingual models; benchmarking “real” translation/generalization versus memorization; identifying linguistic disparities or cultural biases; and, for task-specific (e.g. clinical or governmental) needs, enabling equitable and compliant automation.

5. Language, Resource, and Domain Coverage

Modern bilingual evaluation datasets span a diversity of language pairs—high-resource (e.g., en–zh, en–fr, en–es) and low-to-medium-resource (e.g., en–tr, kk–ru, ar–en)—and cover multiple domains, including:

• General semantics and world knowledge (UNED-ACCESS 2024 (Salido et al., 2024))
• Academic and scientific argumentation (ScholarBench (Noh et al., 22 May 2025))
• Policy and compliance (POLIS-Bench (Yang et al., 4 Nov 2025))
• Database querying (StatBot.Swiss (Nooralahzadeh et al., 2024))
• Safety and sociopolitical risk (Qorgau (Goloburda et al., 19 Feb 2025))
• Code and software engineering (CodeApex (Fu et al., 2023))
• Lexical vocabulary (MuLVE (Jacobsen et al., 2022))

A trend is the move toward (a) very large, balanced, and attribute-tagged coverage (BMMR-Eval 20k+ (Xi et al., 4 Jul 2025)), (b) synthetic but human-in-the-loop dialogic data (NormDial (Li et al., 2023)), and (c) multimodal, multi-task alignment with parallel reasoning paths.

Resource distribution, parallelization fidelity, and translation quality are increasingly reported; e.g., BiToD's balanced EN/ZH dialogues and knowledge base entities (Lin et al., 2021), or ScholarBench's 18.7% strictly parallel items and attribute-constrained prompts (Noh et al., 22 May 2025).

6. Methodological Challenges, Pitfalls, and Recommendations

Empirical analysis identifies substantial pitfalls and methodological recommendations:

• Test set contamination and data artifacts: UNED-ACCESS 2024 found nearly perfect model ranking correlation (Pearson r=0.98) with MMLU, even with just 1,000 exam questions, when strict source control is enforced (Salido et al., 2024).
• Proper noun overrepresentation: MUSE BDI's high rate of proper nouns (25%) renders evaluation unstable (Kementchedjhieva et al., 2019).
• Missing/ambiguous gold standards: BDI, translation, and sense similarity benchmarks can systematically overstate observed model gaps due to incomplete or narrow reference answer sets.
• Cross-lingual fidelity: Many datasets stress the need for native-translator post-editing and double annotation passes to guarantee genuine comparability, as in StatBot.Swiss (Nooralahzadeh et al., 2024) and NormDial (Li et al., 2023).
• Language/size bias in models: Across nearly all benchmarks, larger and proprietary LLMs exhibit a smaller performance gap between languages (e.g. ≤3% in EN/ES for UNED-ACCESS, vs. 37% for smallest open models (Salido et al., 2024)), but system orderings may flip under proper error filtering (see MUSE BDI (Kementchedjhieva et al., 2019)).
• Task metric selection: Reliance purely on automatic metrics (BLEU, semantic similarity) can produce misleading results; detailed task-specific metrics (step-level verification, chain-of-thought correctness) and manual error analyses are recommended.

Best practices include (i) controlling proper noun and out-of-domain content, (ii) manual verification of translation and gold-standard answer coverage, (iii) explicit reporting of filtered and discarded instances, (iv) error analysis by linguistic/inflectional category, (v) use of morphologically complete and representative references, and (vi) development of alternative tasks (e.g. semantics-based or process-based benchmarks).

7. Impact, Extensions, and Future Directions

Bilingual evaluation datasets now underpin model selection, error analysis, data-centric NLP, and transfer learning research across general and domain-specific tasks. Their continued evolution reflects three trends:

• Fine-grained, multi-attribute evaluation: Discursive, multimodal, and chain-of-thought-annotated benchmarks (e.g., BMMR-Eval, ScholarBench) disentangle surface and process abilities, and highlight discipline-specific or rhetorical limitations (Noh et al., 22 May 2025, Xi et al., 4 Jul 2025).
• Safety and compliance: Culturally aware, jurisdiction-specific safety (Qorgau) and policy compliance (POLIS-Bench (Yang et al., 4 Nov 2025)) datasets introduce new dimensions of real-world risk and constraint evaluation.
• Resource expansion and hybridization: The extension to more language pairs, multimodal contexts, code-switching, and sociocultural grounding is ongoing, with focus on robust, scalable manual and semi-automatic validation pipelines.

A plausible implication is that comprehensive multi-lingual, multi-domain evaluation will require harmonized benchmarks with explicit provenance, scenario-grounded tasks, and dual-metric scoring systems capable of capturing both superficial surface similarity and substantive correctness.

For researchers and practitioners, rigorous bilingual benchmarks—anchored to domain-representative, high-fidelity data and robust evaluation methodology—remain critical for deploying and understanding AI systems in a truly multilingual world.