Masked Language Modeling (MLM)

• Masked Language Modeling (MLM) is a self-supervised pretraining method that masks input tokens and uses bidirectional context to predict the original tokens.
• It underpins architectures like BERT and extends to vision-language models through structured masking strategies and cross-modal fusion.
• Recent research shows that adaptive masking rates and curriculum designs can improve downstream metrics, with gains reported on tasks like GLUE and SQuAD.

Masked Language Modeling (MLM) is a self-supervised denoising objective in which a subset of tokens in an input sequence is corrupted—typically by replacing selected positions with a special $[MASK]$ token—and the model is trained to reconstruct the original tokens from bidirectional context. It has been one of the most prominent approaches for pretraining bidirectional text encoders because of its simplicity and effectiveness, and it has also become an essential component of vision-language pretraining (Meng et al., 2023, Verma et al., 2022). Subsequent work has treated MLM not only as a pretraining loss, but also as a family of conditional distributions that can be used for scoring, generation, link prediction, bias auditing, and style transfer, while exposing nontrivial theoretical issues concerning representation allocation, conditional consistency, and the status of MLMs as LLMs (Hennigen et al., 2023, Young et al., 2022).

1. Objective and probabilistic characterization

In the standard formulation, an input sequence $x=(x_1,\dots,x_n)$ is corrupted by masking a subset of positions $M$, and the model is trained to predict the original tokens at those positions. A common objective is

$$\mathcal{L}_{\mathrm{MLM}} = - \sum_{i\in M}\log P(x_i \mid x_{\setminus M}),$$

or, equivalently, a cross-entropy over only the masked positions (Clouatre et al., 2020, Verma et al., 2022). In BERT-style corruption, the selected tokens follow the 80/10/10 rule: 80% are replaced with $[MASK]$, 10% with random tokens, and 10% are left unchanged (Clouatre et al., 2020, Malmi et al., 2020).

The distinctive property of MLM is that prediction is conditioned on both left and right context. This differs from left-to-right causal factorization and makes MLM particularly suitable for representation learning, sentence encoding, and cross-modal fusion, since the prediction target is informed by a bidirectional context window rather than a prefix alone (Micheletti et al., 2024, Bitton et al., 2021). In vision-language settings, the same objective is applied to text while allowing the model to attend jointly to image and text, so masked words are reconstructed from both modalities (Bitton et al., 2021, Verma et al., 2022).

A central theoretical qualification is that MLMs do not explicitly define a distribution over language, i.e., they are not LLMs per se (Hennigen et al., 2023). What they provide directly is a collection of conditionals $p_\theta(x_i\mid x_{\setminus M})$ under different masking patterns. This distinction matters for sentence scoring, generation, and consistency analysis, because it means that a pretrained MLM supplies local or partially masked conditional distributions, not an explicit joint $p(\mathbf{x})$ over full sequences (Hennigen et al., 2023, Young et al., 2022).

2. Mask selection, masking rate, and curriculum design

A masking strategy can be formalized as a conditional distribution $f(M\mid T)$, and the masking rate $p$ is the target fraction of tokens to be masked (Verma et al., 2022). Uniform masking is the simplest instance, with $m_i \sim \mathrm{Bernoulli}(p)$ independently across positions, and it is parameter-free beyond $x=(x_1,\dots,x_n)$0, has no clustering bias, and is easily parallelizable (Verma et al., 2022). Other strategies introduce structure by masking whole words, spans, high-PMI n-grams, or linguistically defined categories such as nouns and verbs (Verma et al., 2022, Zeng et al., 2021).

A recurrent empirical theme is that the canonical 15% rate is not universally optimal. In vision-language pretraining with ViLT on 4 M image-caption pairs, sweeping $x=(x_1,\dots,x_n)$1 showed that downstream performance improved substantially as $x=(x_1,\dots,x_n)$2 increased, with task-dependent optima well above 15% and a median optimum of approximately 0.60 (Verma et al., 2022). The same study reports that higher masking rates improved not only MLM-related behavior but also Image-Text Matching and retrieval, suggesting that in multimodal settings heavier masking can strengthen cross-modal alignment rather than merely making token prediction harder (Verma et al., 2022).

In cross-modal pretraining with short captions, random 15% masking creates a particular inefficiency: 36% of sentences end up with zero masked tokens, and approximately 50% of the masked tokens are stop-words or punctuation (Bitton et al., 2021). Alternative strategies such as Objects, Content-words, and Top-concrete enforce at least one masked token per caption and bias mask selection toward visually grounded or semantically salient words. When pre-training LXMERT, these alternatives consistently improved VQA, GQA, and NLVR2, especially in low-resource settings, and also improved prompt-based probing designed to elicit image objects (Bitton et al., 2021).

Several works make masking explicitly time-variant. Masking Ratio Decay (MRD) uses a high initial ratio, approximately $x=(x_1,\dots,x_n)$3, and decays it linearly or with a cosine schedule over training; POS-Tagging Weighted (PTW) masking dynamically upweights parts of speech with higher smoothed MLM loss (Yang et al., 2022). At 1 M steps, cosine MRD improved GLUE by +1.7 and SQuAD by +0.7 relative to a fixed 15% baseline, and matched the baseline’s SQuAD performance at roughly 650 K steps, implying a 35% time saving (Yang et al., 2022). Concept-based Curriculum Masking (CCM) constructs an easy-to-hard schedule over concepts derived from ConceptNet; its base model reached the original BERT baseline’s final GLUE score in only 50% of total training steps and showed comparative performance with the original BERT at half of the training cost (Lee et al., 2022).

Mask selection can also be reweighted by token statistics or model difficulty. Frequency-based and dynamic loss-based weighted sampling oversample rare or hard-to-predict tokens, preserving the 15% global mask rate while changing which tokens are selected (Zhang et al., 2023). Fully-Explored MLM (FE-MLM) addresses the randomness of mask sampling itself: by partitioning a sequence into non-overlapping segments and masking one segment at a time, it maximizes Hamming distance among masks on the same sequence, reduces gradient covariance, and empirically outperforms standard random masking in both continual pre-training and pre-training from scratch (Zheng et al., 2020).

3. Architectural realizations and encoder–decoder interfaces

The baseline architectural pattern is an encoder-only Transformer whose hidden states at masked positions are projected to vocabulary logits by an MLM head (Kaneko et al., 2021, Micheletti et al., 2024). This pattern underlies BERT, RoBERTa, ALBERT, and many later variants summarized in the literature. In practice, MLM has been instantiated in encoder-only LLMs, cross-modal encoders such as LXMERT and ViLT, and domain-adapted models such as PubMedBERT and SciFive (Bitton et al., 2021, Verma et al., 2022, Micheletti et al., 2024).

A key architectural refinement is MAE-LM, motivated by the observation that $x=(x_1,\dots,x_n)$4 tokens are present in pretraining but absent at fine-tuning time (Meng et al., 2023). MAE-LM pretrains a Masked Autoencoder architecture with MLM where $x=(x_1,\dots,x_n)$5 tokens are excluded from the encoder. The encoder sees only real tokens with their original positional identity, while a small bidirectional decoder re-introduces $x=(x_1,\dots,x_n)$6 tokens and is used only during pretraining to predict the original tokens with the same cross-entropy loss. No cross-attention is required, and the decoder is discarded at fine-tuning (Meng et al., 2023). This design is explicitly intended to prevent the encoder from allocating representational capacity to $x=(x_1,\dots,x_n)$7 at the expense of real tokens.

Other variants alter the semantics of the masked slot itself. Padded MLM, used in unsupervised text style transfer, replaces a masked span with up to $x=(x_1,\dots,x_n)$8 slots consisting of $x=(x_1,\dots,x_n)$9 plus $M$0 sentinels, allowing the model to insert anywhere from zero up to four tokens without predetermining the insertion length (Malmi et al., 2020). ExLM replaces each $M$1 with $M$2 parallel clone embeddings, adds a two-dimensional rotary positional encoding over original position and clone index, and learns a transition matrix over the expanded states so that masked predictions can be aligned along paths in a directed acyclic graph (Zheng et al., 23 Jan 2025). In generation-oriented MLM systems, all positions may initially be treated as masked, and an iterative unmasking scheduler fills the most confident positions first until no $M$3 remains (Micheletti et al., 2024).

These architectural changes indicate that “MLM” now denotes a broader design space than the original single-slot replacement scheme. The common invariant is still denoising from masked context, but the encoder exposure to artificial symbols, the cardinality of the masked state, and the interface between pretraining and downstream use vary substantially (Meng et al., 2023, Zheng et al., 23 Jan 2025).

4. Scoring, inference, and downstream uses

Because MLMs do not define an explicit joint LLM, sentence scoring requires a proxy. A widely used proxy is pseudo-log-likelihood (PLL), which masks each token in turn, scores the true token under the remaining context, and sums the resulting log-probabilities (Kauf et al., 2023). However, the original PLL metric yields inflated scores for out-of-vocabulary and multi-token words because subword pieces of the same word remain visible as context. The adapted metric PLL-word-l2r masks the target subtoken and all within-word subtokens to its right, and it better satisfies theoretical desiderata, correlates more strongly with GPT-2 sentence log-likelihood, and yields consistent gains on BLiMP across BERT-base, BERT-large, RoBERTa-base, and RoBERTa-large (Kauf et al., 2023).

Bias auditing motivates a different scoring move: rather than masking some tokens, All Unmasked Likelihood (AUL) feeds the complete sentence to the MLM and predicts every token from the unmasked embedding, with AULA weighting token log-likelihoods by attention-derived salience (Kaneko et al., 2021). This was proposed because prior masked-token metrics can be unreliable when token prediction accuracy is low, when the metric does not reflect downstream use, and when high-frequency words are overrepresented in masked evaluation. On CrowS-Pairs and StereoSet, AUL and AULA were reported to detect different types of biases more accurately than prior pseudo-likelihood measures (Kaneko et al., 2021).

MLM conditionals can also be exploited in structured prediction. MLMLM performs knowledge-base link prediction by constructing a cloze context with a contiguous block of masks corresponding to the unknown entity, reading off token probabilities for each candidate entity, and ranking candidates by mean likelihood over the masked block (Clouatre et al., 2020). The method requires no new parameters beyond a RoBERTa-Large MLM head, supports single-pass inference with one forward pass per query, achieves state-of-the-art MRR on WN18RR, obtains the best non-entity-embedding-based results on FB15k-237, and can score previously unseen entities without retraining (Clouatre et al., 2020).

Generation and editing are additional inference-time uses. In unsupervised style transfer, Masker trains source and target padded MLMs, identifies spans where the two models disagree most in likelihood, deletes those spans, and lets the target MLM propose replacements of variable length (Malmi et al., 2020). In text generation, iterative MLM unmasking was compared directly with causal language modeling across medical discharge summaries, movie plot synopses, and authorship verification data. The reported result is that MLM consistently outperforms CLM in text generation across all datasets, with higher quantitative scores and better coherence, although there is no strong correlation between the quality of the generated text and downstream task performance (Micheletti et al., 2024).

5. Representation deficiency, inconsistency, and other limitations

A prominent critique concerns the role of the $M$4 token itself. Representation deficiency refers to the claim that during pretraining the special $M$5 token reserves a subspace of the model’s $M$6-dimensional hidden space, and because $M$7 never appears at fine-tuning time, those dimensions become dead for real tokens (Meng et al., 2023). The theoretical argument is rank-based: the final-layer $M$8 representations must become high-rank to support vocabulary prediction, but if the real-token subspace at any layer contained the $M$9 subspace, repeated self-attention would collapse the real-token space toward rank-1 exponentially fast, contradicting the rank-increase requirement for $$\mathcal{L}_{\mathrm{MLM}} = - \sum_{i\in M}\log P(x_i \mid x_{\setminus M}),$$0 (Meng et al., 2023). Empirically, with RoBERTa-base, the $$\mathcal{L}_{\mathrm{MLM}} = - \sum_{i\in M}\log P(x_i \mid x_{\setminus M}),$$1-effective-rank gap between representations with and without $$\mathcal{L}_{\mathrm{MLM}} = - \sum_{i\in M}\log P(x_i \mid x_{\setminus M}),$$2 grows from approximately 10–20 dimensions in layer 1 to approximately 150–200 dimensions by layer 12, while the isolated $$\mathcal{L}_{\mathrm{MLM}} = - \sum_{i\in M}\log P(x_i \mid x_{\setminus M}),$$3 representations rise from at most 5 dimensions at the input to more than 700 dimensions at the final layer (Meng et al., 2023).

A second limitation is probabilistic inconsistency. Different masking patterns can produce conditionals that cannot be derived from a coherent joint distribution when considered together (Young et al., 2022). This is not merely an abstract objection: MLMs from BERT-base to UL2-20B give different predictions to the same input question under different masking patterns, and disagreement rates on MMLU, LAMBADA, and BigBench rise as more conditionals are compared (Young et al., 2022). On a separate pairwise analysis, exact reconstruction of a two-token joint from MLM conditionals showed that a KL-based Arnold–Gokhale approach outperforms Markov random field heuristics and can even occasionally yield conditionals with lower unary perplexity than the original MLM (Hennigen et al., 2023). Together, these results sharpen the point that MLM conditionals are neither guaranteed to be mutually compatible nor trivially convertible into an explicit LLM.

A related critique concerns corrupted context semantics. ExLM identifies a corrupted semantics problem in which replacing tokens with $$\mathcal{L}_{\mathrm{MLM}} = - \sum_{i\in M}\log P(x_i \mid x_{\setminus M}),$$4 can create contexts compatible with multiple meanings, producing semantic multimodality in the masked prediction distribution (Zheng et al., 23 Jan 2025). Its repeated-MLM analysis links this phenomenon to the probability that all copies of a token are masked, and reports lower entropy and better robustness to high mask ratios when context is enhanced via clone expansion and dependency modeling (Zheng et al., 23 Jan 2025).

Evaluation can also be distorted by the masking interface. Original PLL can overrate OOV or multi-token words because within-word context leaks into the target prediction (Kauf et al., 2023). Social-bias metrics based on masking a subset of tokens can be skewed by low mask-prediction accuracy and by high-frequency-word selection bias, which motivated AUL and AULA (Kaneko et al., 2021). These critiques do not imply that MLM is unusable; rather, they specify where the conditional nature of the objective creates methodological hazards.

6. Variants, alternatives, and active research directions

Current research on MLM increasingly centers on efficiency, informativeness, and compatibility between pretraining and downstream use. Weighted masking based on PMI-derived informative relevance increases both the probability of masking informative tokens and the loss penalty for mistakes on them, yielding gains on LAMA factual recall, closed-book question answering, prompt-based sentiment analysis and NLI, fine-tuned SQuAD, and selected GLUE tasks (Sadeq et al., 2023). Weighted Sampling for BERT similarly targets rare or high-loss tokens and reports improved sentence embeddings on STS as well as improved GLUE transfer, with the dynamic loss-based sampler outperforming static frequency correction (Zhang et al., 2023).

Another direction is task-aligned masking. In Chinese machine reading comprehension, pretraining MLMs whose masking-length distribution matches the downstream answer-length distribution produced consistent relative gains: the closer the MLM’s masking-length distribution is to the MRC answer-length distribution, the better the end-task performance (Zeng et al., 2021). Concept-based curriculum masking follows a different alignment principle, starting from conceptually easy tokens and expanding through graph neighborhoods in ConceptNet to construct an easy-to-hard curriculum (Lee et al., 2022). Scheduled masking pursues alignment over training time rather than over task format, adapting the masking ratio and masked content to the model’s current maturity (Yang et al., 2022).

MLM is also no longer unchallenged as the default pretraining target. “Frustratingly Simple Pretraining Alternatives to Masked Language Modeling” shows that several token-level classification tasks can replace MLM as standalone objectives. In particular, Shuffle + Random on BERT-BASE achieved a GLUE average of 79.2 versus 77.6 for MLM and remained competitive on SQuAD, while BERT-MEDIUM with 41% of BERT-BASE’s parameters incurred only about a 1-point GLUE drop relative to BASE + MLM (Yamaguchi et al., 2021). This does not refute MLM’s usefulness, but it narrows the claim that vocabulary reconstruction is uniquely necessary for learning strong bidirectional representations.

The main research frontier is therefore not simply “better prediction of masked tokens,” but better control of what masking teaches and what it costs. This suggests several converging priorities already explicit in the literature: exclude artificial symbols from the encoder when they waste representational capacity, maximize overlap between pretraining and downstream vocabularies, track subspace allocation rather than only loss curves, enforce or exploit conditional consistency, and treat masking rate, masking content, and masking schedule as first-class hyperparameters rather than fixed defaults (Meng et al., 2023, Young et al., 2022, Yang et al., 2022). In that sense, MLM has evolved from a single pretraining recipe into a general conditional-denoising framework whose theoretical status, optimization behavior, and downstream utility remain active areas of research.