ÜberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset

Abstract: Multilinguality is a core capability for modern foundation models, yet training high-quality multilingual models remains challenging due to uneven data availability across languages. A further challenge is the performance interference that can arise from joint multilingual training, commonly referred to as the "curse of multilinguality". We study multilingual data curation across thirteen languages and find that many reported regressions are not inherent to multilingual scaling but instead stem from correctable deficiencies in data quality and composition rather than fundamental capacity limits. In controlled bilingual experiments, improving data quality for any single language benefits others: curating English improves non-English performance in 12 of 13 languages, while curating non-English yields reciprocal improvements in English. Bespoke per-language curation produces substantially larger within-language improvements. Extending these findings to large-scale general-purpose training mixtures, we show that curated multilingual allocations comprising under 8% of total tokens remain remarkably effective. We operationalize this approach within an effort that produced a 20T-token pretraining corpus derived entirely from public sources. Models with 3B and 8B parameters trained on a 1T-token random subset achieve competitive multilingual accuracy with 4-10x fewer training FLOPs than strong public baselines, establishing a new Pareto frontier in multilingual performance versus compute. Moreover, these benefits extend to frontier model scale: the 20T-token corpus served as part of the pretraining dataset for Trinity Large (400B/A13B), which exhibits strong multilingual performance relative to its training FLOPs. These results show that targeted, per-language data curation mitigates multilingual interference and enables compute-efficient multilingual scaling.

• The paper demonstrates that precise, per-language data curation yields substantial cross-lingual transfer gains and improved FLOPs efficiency.
• It introduces bespoke curation pipelines with rigorous filtering, deduplication, and synthetic augmentation to overcome the curse of multilinguality.
• The study reveals bidirectional performance improvements across languages, redefining compute-performance trade-offs for multilingual models.

ÜberWeb: Data-Centric Multilingual Foundation Models at Trillion-Scale

Introduction and Motivation

The “ÜberWeb” work (“ÜberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset” (2602.15210)) addresses persistent challenges in training high-quality multilingual LLMs (MLLMs), specifically data imbalance, noisy/non-uniform data quality, and the so-called "curse of multilinguality"—the empirical degradation of per-language performance as more languages are added to a joint training mix. The authors contend that, contrary to capacity-centric theories, many performance limitations originate from remediable deficiencies in data curation and allocation, not from intrinsic scaling or zero-sum capacity trade-offs.

Consequently, the authors operationalize a data-driven curation paradigm targeting 13 diverse non-English languages and English, spanning major global scripts, resource tiers, and linguistic families. The main contributions are empirical: carefully curated per-language data pipelines and mixture strategies are shown to yield substantial, bidirectional cross-lingual transfer, mitigate interference effects, and redefine FLOP-optimal performance–compute Pareto frontiers with models trained on a 20 trillion token open corpus.

Multilingual Curation and Transfer Dynamics

Central to the paper is the interrogation of cross-lingual transfer phenomena contingent on data quality. The authors find that, in controlled bilingual settings (English + one of 13 target languages, with 3B models), curation of only the English dataset results in a mean +3.91% relative improvement for non-English evaluation tasks (MMLU, ARC-Challenge, Belebele), with 12/13 languages benefiting.

Figure 1: Performance impact of curation regimes, showing that high-quality English curation confers broad multilingual gains, and maximal gains require bidirectional, per-language curation.

However, an important asymmetry emerges: while English-centric curation substantially benefits linguistically proximate target languages (e.g., Spanish, French, German), more distant languages see diminished transfer. Using embedding-based and perplexity-based proxies for language distance, the authors establish a strong negative correlation (Pearson r = -0.62/-0.70) between similarity to English and benefit from cross-lingual transfer.

Figure 2: More similar languages to English (low embedding distance and perplexity) derive greater cross-lingual benefit from high-quality English corpus curation.

Importantly, improvements in non-English curation reciprocally enhance English performance (mean +1.21% relative, 12/13 cases), underscoring the global impact of data quality on overall model utility.

Figure 3: Enhancing non-English data quality provides measurable returns even for English downstream tasks, demonstrating bidirectional transfer.

Bespoke Per-Language Pipelines and Curse Mitigation

The study builds on these findings by demonstrating that optimal language-specific performance is realized only via bespoke, per-language curation pipelines. While global English curation uplifts aggregate non-English performance, the gap to optimal in-language performance remains wide unless each language's data—and specifically filtration, deduplication, and synthetic augmentation—is treated separately.

This result directly contradicts claims that parameter capacity or mixture size are inevitable bottlenecks. Models trained with curated mixtures suffer less from negative interference and require a substantially smaller proportion (∼8%) of multilingual tokens compared to comparable open-weight baselines. Arbitrary scaling of token or parameter count alone does not yield comparable improvements.

Efficacy of Translation and Synthetic Augmentation

A further major axis of analysis is the role of machine-translated synthetic data for low-resource languages. The experiments confirm that augmenting with arbitrarily translated English content yields only marginal gains. Translation is only effective if the source English content is high-quality and passes stringent filtering/classification, with a +5.09% relative gain in multilingual benchmarks compared to the uncurated baseline. However, even this is not a substitute for tailored, native-language curation, which produces the highest performance.

Figure 4: Score-based filtering of English source data before translation drives superior low-resource language benchmark scores versus naive translation augmentation.

Scaling to Trillion-Token Regimes: Pareto Frontier Advances

The curation approach is scaled to train models (3B and 8B parameters) on a random 1T-token subset of the 20T-token open-source corpus. With only ~80B curated non-English tokens distributed amongst 13 languages (7.75% of the corpus), models outperform strong public baselines by a factor of 4–10× in FLOPs efficiency and maintain or surpass their multilingual accuracy.

Figure 5: DatologyAI models define a new compute-performance Pareto frontier, substantially improving multilingual error rates at lower FLOPs relative to strong open baselines.

Further, the approach holds at the frontier scale, forming the multilingual component of the 17T-token Trinity Large MoE model (400B total/13B active parameters), which outperforms peer models on multilingual tasks relative to its computational budget.

Per-Language Data Efficiency and Performance

Detailed per-language evaluations show that curated, balanced mixtures maintain performance parity across diverse languages and scripts (Spanish, Hindi, Chinese, Russian, Arabic, etc.), in contrast to language-specialist models which achieve higher performance only in their focus language at the expense of other languages. Data efficiency analysis further demonstrates that DatologyAI models reach competitive per-language scores with orders-of-magnitude fewer tokens per language than comparable open-source models.

Figure 6: Per-language FLOPs/accuracy scaling for Latin-script languages; right panel shows DatologyAI’s models maintain aggregate multilingual parity.

Figure 7: Indic and Arabic-script languages see similar performance improvements, demonstrating efficacy beyond European languages.

Figure 8: CJK and Russian languages, with models retaining performance close to aggregate multilingual scores.

Figure 9: DatologyAI curation achieves high per-language performance with far fewer language-specific tokens than open baselines.

Implications and Future Directions

The findings from "ÜberWeb" have several key implications for the development of next-generation MLLMs:

• Targeted curation fundamentally alters the data–compute–performance trade-off. Strategic control over source quality in each supported language, rather than unstructured scaling, unlocks compute-efficient, broad-coverage multilinguality.
• Pareto improvements in FLOPs efficiency directly support sustainability goals and democratize LLM technologies, especially as open-weight FLOP budgets remain limited for researchers.
• Data-centric approaches supplant capacity-centric theory in practice; persistent regressions (the "curse of multilinguality") can be resolved with quality-focused interventions, not necessarily by increasing token or model size.
• Translation-based upsampling only pays off when source data is also filtered, scored, and synthetically augmented in accordance with each language’s structure and distribution.

The work motivates further research into adaptive mixture designs, compute-aware sampling strategies, and lifelong curation pipelines for both text and multimodal pretraining settings. Extending the framework to additional languages and modalities will require advances in evaluation frameworks, especially under open-world multilingual scenarios.

Conclusion

ÜberWeb demonstrates that multilingual scaling constraints are largely a function of correctable data defects rather than intrinsic model limits. Bespoke, high-quality per-language curation and mixture design yield strong and even Pareto-improving gains for multilingual models, both in low- and high-resource regimes. The bidirectional transfer dynamics observed underscore the necessity of global mixture optimization, invalidating zero-sum framings of multilingual modeling. The empirical advances—substantial improvements in FLOPs efficiency, robust per-language parity, and methodological transparency—suggest a paradigm shift toward inclusive, data-centric foundation model development.