BERT-base-multilingual-cased

Updated 13 January 2026

BERT-base-multilingual-cased is a multilingual Transformer model that pretrains on Wikipedia text across 104 languages using a shared cased WordPiece vocabulary.
It adheres to the standard BERT-Base architecture, enabling zero-shot transfer and effective cross-lingual learning through long context windows and fine-tuning strategies.
Empirical studies reveal robust performance in syntactic probing and downstream tasks, while highlighting challenges like English grammatical bias and anisotropic representation spaces.

BERT-base-multilingual-cased (commonly abbreviated as mBERT) is a Transformer-based masked LLM pretrained on Wikipedia text across 104 languages, sharing a single parameter set and WordPiece vocabulary with case sensitivity. It serves as a widely adopted model in multilingual representation learning, cross-lingual transfer, and the evaluation of typological phenomena in NLP. The following sections summarize mBERT’s architecture, pretraining, transfer behavior, linguistic properties, and performance across core linguistic and applied tasks, referencing principal results from recent arXiv literature.

1. Model Architecture and Pretraining Scope

mBERT is a direct implementation of the original BERT-Base specification, with key architectural features:

12 Transformer encoder layers ( $L=12$ )
Hidden size 768 ( $H=768$ )
12 attention heads per layer ( $A=12$ )
Feed-forward dimension 3072
Shared WordPiece vocabulary, size $\approx$ 119,000 (cased)
Total parameters $\approx$ 110–178M (a discrepancy across sources due to counting embedding table vs. transformer weights)

Pretraining is performed on Wikipedia dumps for 104 languages. The shared vocabulary is derived over all languages, preserving case distinctions. The pretraining objective is the sum of the Masked Language Modeling (MLM) loss, where 15% of tokens in each input sequence are masked and predicted, and the Next Sentence Prediction (NSP) loss, which predicts segment contiguity. Standard maximal input sequence length is 512 tokens, enabling large context windows that are shown to be crucial for the model’s eventual cross-lingual transfer capacity (Liu et al., 2020, Liu et al., 2020).

2. Pretraining Data, Tokenization, and Multilingual Vocabulary

Pretraining encompasses approximately 2.5 billion words ( $\approx$ 12.5B tokens) across typologically diverse languages ranging from high-resource (e.g., English, Spanish) to mid-/low-resource (e.g., Greek, Hindi). The single WordPiece vocabulary facilitates implicit cross-lingual signal transmission via shared subword representations, including extensive overlap in Latin-script substrings, numbers, and punctuation. English comprises the largest segment of the token pool, shaping the statistical properties of the model’s learned representations (Papadimitriou et al., 2022).

Subword overlap and case preservation increase model flexibility and serve both to anchor alignment (by shared numerals, punctuation) and to enforce finer-grained language-specific patterns, e.g., German noun capitalization (Liu et al., 2020).

3. Cross-Lingual Transfer and Representational Properties

mBERT exhibits strong zero-shot transfer capacity, as substantiated by multilingual benchmarks such as word retrieval (MRR on MUSE dictionaries) and zero-shot NLI classification (XNLI accuracy). At 200K sentences per language, performance trails static models, but with 1M+ sentences, mBERT delivers a dramatic increase in MRR and XNLI accuracy (e.g., MRR $\approx$ 0.51, XNLI Acc $\approx$ 62%) and consistently surpasses static embeddings in open-class word alignment, particularly for adjectives and nouns (Liu et al., 2020, Liu et al., 2020).

Context window size is decisive: truncating down to 20 tokens reduces word-retrieval and XNLI score by up to 30 and 10 points, respectively. For robust cross-lingual generalization, both large monolingual data per language ( $>$ 1M sentences) and long context windows ( $>$ 128 tokens) are mandatory (Liu et al., 2020).

Manipulating mBERT’s latent representations—inserting per-language bias vectors—allows direct control of output language emissions and enables unsupervised token-level translation. mBERT’s ability to switch output vocabulary space via simple mean-difference vector arithmetic is empirically validated for multiple scripts, with conversion rates approaching 100% (e.g., $en\to el$ at $\alpha=3$ ) (Liu et al., 2020).

4. Linguistic Universals and Grammatical Biases

mBERT internalizes grammatical relations that co-cluster according to the Universal Dependencies taxonomy. Structural probe analyses show that a learned subspace ( $B$ ) recovers syntactic tree distances and dependency labels across languages with high fidelity (UUAS $\approx$ 74.1%, DSpr $\approx$ 0.804 on joint subspaces). Notable universal clusters include prenominal vs. postnominal adjective modification, core arguments, determiners, and expletives (Chi et al., 2020).

Despite capturing cross-lingual structure, mBERT shows measurable “grammatical structure bias”—the tendency to reproduce grammatical properties favored in dominant training languages (principally English). For Spanish pronoun-drop and Greek subject–verb order, mBERT assigns 10–15% higher probability to English-like structures relative to strong monolingual baselines (BETO, GreekBERT), thus overproducing explicit subjects and rigid SVO order (bootstrap $p<.05$ ) (Papadimitriou et al., 2022). This structural drift can degrade fluency and register in generation or pseudo-probing tasks.

Table: Grammatical Structure Bias (English-like preference ratio $r_{model}$ )

Language	Monolingual Model $r_{mono}$	mBERT $r_{multi}$	p-value	Bias Direction
Spanish	0.93 (95% CI [0.89,0.97])	1.06 (95% CI [1.02,1.10])	$<.05$	Prefers explicit subject
Greek	0.98 (95% CI [0.94,1.02])	1.12 (95% CI [1.07,1.17])	$<.05$	Prefers SVO order

Mitigation strategies include fine-tuning on in-domain data, language-specific adapters, and rebalancing pretraining corpora (Papadimitriou et al., 2022).

5. Morphosyntactic Probing and Embedding Space Structure

Extensive probing shows mBERT encodes rich morphology: average test accuracy 90.4% across 247 tasks in 42 languages (XLM-RoBERTa: 91.8%, chLSTM: 85%). Probing setups extract the last subword vector of target words, aggregate across layers, and train a classifier atop these features. Shapley-value and masking perturbations reveal that preceding context is systematically more morphologically informative than following context (left/right context ratio $\approx$ 1.45), consistent with typological markedness—even though BERT’s attention is bidirectional (Acs et al., 2023). Cases of reversed asymmetry (postpositions, agreement) correspond to known language-specific morphological triggers.

Table: Morphological Probing—Model Accuracy

Model	Mean accuracy (%)
mBERT	90.4
XLM-R	91.8
chLSTM	85.0
fastText	83.3
Stanza	91.4

6. Embedding Space Geometry and Isotropy

Unlike monolingual BERT, mBERT’s embedding space is highly anisotropic but exhibits no “rogue” dimensions—that is, no mean activation exceeding $3\sigma$ from the mean. Anisotropy is instead distributed evenly across several principal components, with the top dimension contributing only $\approx$ 0.02–0.04 to pairwise cosine similarity (vs. 0.38 in monolingual BERT). Isotropy-enhancing cluster-based postprocessing (PCA subtraction per cluster) yields 10–20-point improvements in semantic similarity Spearman correlation and allows zero-shot transfer of the isotropy correction between languages due to shared degenerate directions (Rajaee et al., 2021).

7. Downstream Performance, Fine-tuning, and Applied Recommendations

mBERT, when fine-tuned for downstream tasks in a target language—POS/morph tagging, sentiment, NER, machine comprehension—consistently outperforms non-contextual or traditional neural baselines.

Estonian NLP: POS/morph tagging accuracy up to 98%, sentiment accuracy 70.2%, NER F1 86.5% (Kittask et al., 2020).
Indic languages: transforming vanilla mBERT to “multilingual SBERT” via synthetic NLI/STS fine-tuning raises Hindi sentence similarity Spearman $\rho$ from 0.49 to 0.75, besting alternatives (LaBSE, LASER) (Deode et al., 2023).
Sentiment/hate speech (English, Korean, Japanese, Chinese, French): accuracy 90.2% (with 8-layer freezing); XLM-RoBERTa outperforms on morphologically rich languages by 2 points (Bilehsavar et al., 6 Jan 2026).
Machine comprehension (English–Hindi QA): SQuAD F1 up to 64.5 (cross-lingual), setting new SOTA for these pairs (Gupta et al., 2020).

Layer freezing (typically the first eight layers) mitigates overfitting and preserves general multilingual representation, especially when data are scarce; unfreezing decreases mean accuracy by 2.7 points in hate speech and sentiment tasks (Bilehsavar et al., 6 Jan 2026). Explainability via LIME or SHAP confirms model decisions respect semantic cues in each language.

References

(Papadimitriou et al., 2022) Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models
(Liu et al., 2020) What makes multilingual BERT multilingual?
(Deode et al., 2023) L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT
(Rajaee et al., 2021) An Isotropy Analysis in the Multilingual BERT Embedding Space
(Chi et al., 2020) Finding Universal Grammatical Relations in Multilingual BERT
(Liu et al., 2020) A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
(Gupta et al., 2020) BERT Based Multilingual Machine Comprehension in English and Hindi
(Bilehsavar et al., 6 Jan 2026) Boosting Accuracy and Interpretability in Multilingual Hate Speech Detection Through Layer Freezing and Explainable AI
(Kittask et al., 2020) Evaluating Multilingual BERT for Estonian
(Acs et al., 2023) Morphosyntactic probing of multilingual BERT models

Summary

BERT-base-multilingual-cased represents a foundational architecture for multilingual representation learning, providing a robust, broadly deployable encoder for over 100 languages. Its strength derives from joint subword vocabulary, deep bidirectional modeling of long-range context, and empirically validated cross-lingual capacity. Notwithstanding its efficacy, users must be alert to grammatical structure bias and anisotropic representation space, applying mitigation strategies—data balancing, targeted fine-tuning, isotropy correction—as needed for fluency-critical and typologically sensitive tasks.