Global MMLU-Lite Benchmark Overview

• Global MMLU-Lite Benchmark is a multilingual, curated evaluation suite that assesses large language models across a wide range of academic domains.
• It employs a stratified sampling methodology and a multi-stage translation pipeline, including automatic pre-translation, human post-editing, and expert review, to ensure error-free, culturally sensitive datasets.
• The benchmark integrates robust error auditing and statistical quality controls to deliver reliable cross-lingual performance metrics and highlight gaps in low-resource language evaluations.

The Global MMLU-Lite Benchmark is a curated, multilingual, and error-audited subset of the Massive Multitask Language Understanding (MMLU) evaluation suite, designed for efficient, scalable, and equitable assessment of LLMs across a representative range of academic domains and languages. It integrates lessons from translation fidelity, dataset curation, statistical sampling, and systematic error audit protocols to offer a high-fidelity resource for cross-lingual model benchmarking while controlling for translation artifacts and legacy label errors (Plaza et al., 2024, Pomerenke et al., 11 Jul 2025, Gema et al., 2024).

1. Category Selection and Dataset Construction

The construction process prioritizes balanced domain coverage, sampling rigor, and data quality. Domains are stratified to ensure representative coverage spanning STEM, social sciences, humanities, and “Miscellaneous” fields. Key selection criteria include:

• Domain Breadth: Categories such as Mathematics, Physics, US Foreign Policy, Economics, Philosophy, Art History, and Logic are sampled to span the academic spectrum.
• Difficulty Calibration: For each domain, question selection targets a deliberate mix of easy (≤ 20% failure in English, as determined by a strong English LLM such as GPT-4), medium (20–50%), and hard (> 50%) items.
• Cultural Sensitivity: Categories prone to culture-specific content (e.g., History, Law) are balanced with universal topics (e.g., Logic, Computer Science).

Stratified Sampling Protocol:

1. Partition M original MMLU categories into P thematic super-groups (typical P ≈ 4).
2. Allocate Kᵢ categories per group proportional to group size: $Kᵢ = \lfloor K \cdot (|Groupᵢ| / M) \rfloor$, enforcing $\sum_i K_i = K$.
3. Within each category, sample $nᵢ ≈ \frac{z_{α/2}^2\,p\,(1-p)}{ε^2}$, where p is the proportion of correct answers and ε the desired margin of error.
4. Validate a mix of at least 25% “hard” and 25% “easy” items (Plaza et al., 2024).

Error auditing and triage are conducted following the taxonomy of MMLU-Redux (Gema et al., 2024), distinguishing questions as:

• Retained: No error.
• Amended: Question/option clarity issues corrected.
• Discarded: Ground-truth answer errors or ambiguities.

2. Translation and Adaptation Pipeline

Translation quality is managed through a multi-stage pipeline:

1. Automatic Pre-Translation: Parallel use of Azure Translator and GPT-4, with outputs (A₁, A₂) stored for comparison.
2. Human Post-Editing: Two bilingual translators independently revise each item, handling proper names, units, idioms, and marking segments of uncertainty.
3. Expert Subject-Matter Review: Domain specialists validate semantic integrity, plausibility of distractors, and preservation of the original difficulty level; a checklist ensures terminological and contextual adaptation.
4. Quality Assurance:
   • Inter-Annotator Agreement: Measured by Cohen’s κ, requiring κ ≥ 0.75 for reliability.
   • Back-Translation: >10% of items are randomly sampled for back-translation and compared; semantic divergence >5% triggers re-revision (Plaza et al., 2024).

Adaptation guidelines enforce:

• Localized units/currency where relevant, annotated as adapted.
• Idioms and culturally bound examples replaced or clarified.
• Proper nouns are preserved (John Constable remains “John Constable”).
• Difficulty alignment: re-piloting on bilingual respondents, requiring error rates and response times within ±10% of the English pilot.

3. Error Identification, Auditing, and Quality Metrics

Quality control leverages taxonomies and formal measures:

• Hierarchical Error Taxonomy (Gema et al., 2024):
  • Type 1a: Question clarity faults.
  • Type 1b: Option clarity faults.
  • Type 2a: No correct answer available.
  • Type 2b: Multiple correct answers.
  • Type 2c: Annotated key is incorrect.
• Error Rates:
  • For question $i$, $E_i = 1$ if any error; $E_i = 0$ otherwise.
  • Subject-level: $E_s = \frac{1}{N_s} \sum_{i=1}^{N_s} E_i$.
  • Overall: $E_{tot} = \frac{1}{N} \sum_s \sum_i E_i$.
• Wilson Score Confidence Intervals: Applied per-domain to quantify error rate uncertainty.
• Translation Error Rate: $E = \frac{N_{err}}{N_{total}}$.
• Ambiguity and Meaning Shifts: Detected via cross-LLM answer discrepancies and backed by manual semantic inspection.
• McNemar’s Test: Used for statistical significance in cross-language performance, with $\chi^2 = \frac{(b-c)^2}{b+c}$, where $b$ and $c$ are discordant pairs (Plaza et al., 2024).

4. Evaluation Protocols and Scoring

Assessment standardization addresses both cross-lingual and cross-domain reliability:

• Prompting Strategy: Few-shot, in-context prompts with 3–5 English Q&A demonstrations, then the target language query presented in a fixed multiple-choice format. The leaderboard for the AI Language Proficiency Monitor exclusively reports few-shot results, based on empirical stability (Pomerenke et al., 11 Jul 2025).
• Accuracy Metrics:
  • Per-item set: $$\mathrm{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\hat{y}_i = y_i)$$
  • Per-language: $$\mathrm{Accuracy}_\ell = \frac{1}{N_\ell}\sum_{i=1}^{N_\ell} \mathbf{1}(\hat{y}_{i, \ell} = y_{i, \ell})$$
  • Macro-average: $$\mathrm{MacroAvg} = \frac{1}{L}\sum_{\ell=1}^L \mathrm{Accuracy}_\ell$$
  • Micro-average: $$\mathrm{MicroAvg} = \frac{\sum_{\ell=1}^L \sum_{i=1}^{N_\ell} \mathbf{1}(\hat{y}_{i,\ell}=y_{i,\ell})} {\sum_{\ell=1}^L N_\ell}$$
  • Weighted by language speakers: $$\mathrm{WeightedAvg} = \frac{\sum_{\ell=1}^L w_\ell\,\mathrm{Accuracy}_\ell}{\sum_{\ell=1}^L w_\ell}$$
• Relative Performance: Model accuracy decay and variance are tracked by category, by language family, and by translation origin (human/machine).
• Significance Testing: z-tests compare performance across benchmark variants or models (Gema et al., 2024).

5. Language and Coverage Strategies

Global MMLU-Lite optimizes the trade-off between coverage and evaluation efficiency:

• Core Language Set: Approximately 20 languages, including the top global languages by speaker count, five typologically diverse mid-resource languages, and five critically underserved low-resource languages (e.g., Swahili, Amharic, Yoruba) (Pomerenke et al., 11 Jul 2025).
• Task Subsetting: Typical selection includes 10–15 MMLU subject categories focusing on those with greatest performance variance, maintaining representativeness across major academic fields.
• Corpus Harmonization: Wherever available, human translations are prioritized over machine-generated versions. Script selection follows Unicode CLDR population usage.

Component	Rationale	Data Source
Language subset	Balances global speaker coverage and typological breadth	(Pomerenke et al., 11 Jul 2025)
Subject subset	Captures domain diversity, benchmarks variance	(Plaza et al., 2024, Gema et al., 2024)
Translation origin	Human if available, otherwise high-quality machine	(Pomerenke et al., 11 Jul 2025)
Error audit	Systematic removal/amendment of faulty items	(Gema et al., 2024)

6. Release, Maintenance, and Community Protocols

Sustained reliability and relevance are ensured through infrastructure and governance:

• Version Control: Hosted on collaborative platforms (e.g., GitHub), with explicit tagging (v1.0, v1.1, …) and detailed changelogs documenting item-level adjustments and error rate changes.
• Community Feedback: Structured issue templates (e.g., “Translation concern,” “Cultural mismatch,” “Ambiguity detected”) facilitate transparent triage and resolution by language/domain maintainers.
• Continuous Audits: Periodic reviews—such as quarterly random sampling of 5% of items per language—enable ongoing refinement and calibration.
• Extensibility: Scripts and configuration files for data harmonization, prompt templating, and evaluation are released for open community adaptation. Leaderboards and datasets are versioned and support automated submission workflows for new models or improved translations (Pomerenke et al., 11 Jul 2025, Plaza et al., 2024).
• Reannotation Checks: For error-rate tracking, random re-annotation of 5 questions per subject is conducted semiannually (Gema et al., 2024).

7. Benchmark Significance and Challenges

Empirical analysis of full and “lite” versions demonstrates the necessity and impact of rigorous dataset management:

• Benchmark Fidelity: Error audits such as those in MMLU-Redux reveal that about 6.5–9% of MMLU questions have errors, with some subjects (e.g., Virology) exceeding 50% (Gema et al., 2024). “Lite” protocols mitigate such artifacts, yielding truer model assessments.
• Translation Integrity: Empirical error analysis shows substantial translation-induced shifts in model answers, justifying multi-stage QA protocols and expert adaptation (Plaza et al., 2024).
• Global Comparability: The monitor indicates that models routinely perform with >80% accuracy in English, 60–75% for other major European languages, but under 40% for many low-resource languages, even with expanded parameter counts. This demonstrates persistent underperformance in underrepresented languages (Pomerenke et al., 11 Jul 2025).
• Limitations: Reduced coverage in “lite” mode can obscure language-family-specific gaps or regression in sparsely used domains—a trade-off managed by transparent reporting and modular design (Pomerenke et al., 11 Jul 2025).
• Community Role: Regular maintenance, open data, and user-driven submissions are essential for cross-lingual validity and benchmark evolution.

By combining systematic sampling, error curation, robust translation and adaptation, precise evaluation, and transparent maintenance, the Global MMLU-Lite Benchmark underpins reliable, efficient, and extensible LLM evaluation in a rapidly diversifying linguistic landscape (Plaza et al., 2024, Pomerenke et al., 11 Jul 2025, Gema et al., 2024).