ModernBERT-PT Transformer Models

• ModernBERT-PT is a set of encoder-only Transformer models designed for efficient long-context processing and domain adaptation across diverse languages.
• The models employ tailored pretraining regimes, including dynamic masking rates and custom tokenizers, to achieve competitive benchmarks in clinical, Chinese, Turkish, patent, and Portuguese domains.
• Architectural innovations such as FlashAttention, RoPE, alternating global/local attention, and GeGLU activations contribute to 2–3× throughput improvements and enhanced downstream performance.

ModernBERT-PT refers to a set of state-of-the-art encoder-only Transformer models and pretraining methodologies that extend the ModernBERT architecture across a diverse set of languages and specialized domains, including clinical/biomedical, Chinese, Turkish, French, patent, and Portuguese corpora. The "PT" in ModernBERT-PT is not unique to Portuguese; it denotes "PreTraining" strategy for ModernBERT encoders as implemented in multiple language-specific contexts, as well as (in some literature) "Portuguese" varieties by explicit convention. These architectures leverage architectural optimizations—such as Rotary Positional Embeddings (RoPE), FlashAttention, alternating local/global attention, and normalization strategies—paired with large-scale domain/adaptive pretraining to advance efficiency and downstream performance relative to previous BERT-style, RoBERTa, or DeBERTa models (Sounack et al., 12 Jun 2025, Zhao et al., 14 Oct 2025, Türker et al., 28 Dec 2025, Antoun et al., 11 Apr 2025, Yousefiramandi et al., 18 Sep 2025, Rodrigues et al., 2023).

1. Architectural Innovations in ModernBERT-PT

ModernBERT-PT models adopt architectural paradigms that emphasize both computational efficiency and long-context modeling capacity:

• Self-Attention: FlashAttention is used for memory- and compute-efficient softmax-attention computation. Alternating global (full) and local (sliding window) attention patterns are applied, typically with one-third global and two-thirds local attention, reducing quadratic scaling at long sequence length while maintaining global information flow (Sounack et al., 12 Jun 2025, Zhao et al., 14 Oct 2025, Türker et al., 28 Dec 2025, Antoun et al., 11 Apr 2025, Yousefiramandi et al., 18 Sep 2025).
• Positional Embeddings: RoPE replaces absolute position embeddings, encoding position via token-wise complex rotations $\theta_k = 10000^{-2k/d}$ for each dimension $k$ of the head, and supporting seamless extension to >8k input tokens (Sounack et al., 12 Jun 2025, Türker et al., 28 Dec 2025, Zhao et al., 14 Oct 2025, Antoun et al., 11 Apr 2025).
• Feed-Forward Networks: GeGLU or GLU activations are used in place of standard GELU, yielding richer expressivity and optimizing compute (Antoun et al., 11 Apr 2025, Yousefiramandi et al., 18 Sep 2025). Bias terms are removed from linear layers for parameter and inference efficiency.
• Normalization and Residuals: Pre-LayerNorm is adopted for improved training stability in deeply stacked architectures (Türker et al., 28 Dec 2025).
• Tokenization: Hardware- and language-aware vocabularies (e.g., 32k BPE for Chinese, 50k for English/Turkish/clinical models) are designed to optimize embedding budgets and subword coverage. In some specialized settings (patent/NLP), tokenizer customization further boosts performance (Zhao et al., 14 Oct 2025, Yousefiramandi et al., 18 Sep 2025).

These design choices are compatible with mixed-precision (bf16/fp16) training, support context lengths up to 8,192 tokens, and enable throughput improvements up to $2-3\times$ that of standard BERT-family models at long sequence lengths (Zhao et al., 14 Oct 2025, Türker et al., 28 Dec 2025, Yousefiramandi et al., 18 Sep 2025).

2. Pretraining Regimes and Data Strategies

ModernBERT-PT implementations vary by language and domain but share several pretraining practices:

• Objectives: All variants employ standard masked language modeling (MLM) without auxiliary tasks (e.g., next-sentence prediction or replaced token detection) (Sounack et al., 12 Jun 2025, Türker et al., 28 Dec 2025, Antoun et al., 11 Apr 2025, Yousefiramandi et al., 18 Sep 2025, Rodrigues et al., 2023). Masking rates are dynamically scheduled in some settings (e.g., 30%→15% in Chinese WWM) to optimize learning over training progress (Zhao et al., 14 Oct 2025).
• Training Schedules: Warmup–Stable–Decay (WSD) or damped-cosine learning rate schedules are standard. These avoid abrupt learning rate transitions and support stable training with large batch sizes at long context (Sounack et al., 12 Jun 2025, Zhao et al., 14 Oct 2025, Türker et al., 28 Dec 2025, Antoun et al., 11 Apr 2025).
• Corpus Sources:
  • Chinese: 1.2T tokens from CCI3-HQ, CCI4, and other high-coverage Chinese corpora, with a vocabulary tuned to frequent affixes/compounds (Zhao et al., 14 Oct 2025).
  • Biomedical/Clinical: 53.5B tokens across 20 biomedical and clinical datasets and PubMed, with a two-phase adaptation (joint, then clinical specialization; mask rate 30% $\rightarrow$ 15%) (Sounack et al., 12 Jun 2025).
  • Turkish: 84.88B tokens, multi-domain. 13% non-Turkish (English/code/math) for robustness; supports 8,192-token context (Türker et al., 28 Dec 2025).
  • Patent: 30.8B tokens from 64M patent records; both base models and larger variants with custom BPE (Yousefiramandi et al., 18 Sep 2025).
  • Portuguese: 2–3.7B tokens per variant (PT-PT, PT-BR), derived from brWaC, OSCAR, Europarl, and other cleaned web/domain sources (Rodrigues et al., 2023).
  • French: CamemBERTaV2 (275B–1T tokens) or high-quality deduplicated datasets for controlled architectural comparisons (Antoun et al., 11 Apr 2025).
• Sequence Lengths: Two-stage pretraining pipelines (e.g., 1,024$\rightarrow$8,192 tokens) maintain constant tokens/update ratio to maximize hardware utilization (Zhao et al., 14 Oct 2025).

Tables included in source documents delineate token distributions and domain sampling strategies precisely for Turkish, patent, and Portuguese setups.

3. Empirical Evaluation and Benchmarks

ModernBERT-PT models are evaluated on a broad set of benchmarks in their respective domains and languages, often demonstrating state-of-the-art (SOTA) or highly competitive results.

Selected benchmark highlights:

Variant / Task	Downstream Benchmark(s)	Notable Result(s)
BioClinical ModernBERT	ChemProt, Phenotype, COS NER, Social History NER, DEID NER	ChemProt F1: base 89.9%, large 90.8% (SOTA). Phase 1 checkpoint preserves biomedical knowledge (F1 90.2–90.5%) (Sounack et al., 12 Jun 2025)
Chinese ModernBERT-PT	CLUE dev, SimCLUE STS	8,192-char/s throughput: 180k tok/s. SimCLUE+T2Ranking: 0.505 Pearson, 0.537 Spearman with 5M pairs (Zhao et al., 14 Oct 2025)
TabiBERT (Turkish)	TabiBench (8 categories, 28 tasks)	Macro-averaged: 77.58 vs. BERTurk 75.96. SOTA on QA (+9.55), code retrieval (+2.41), document retrieval (+0.60) (Türker et al., 28 Dec 2025)
French ModernBERT-PT	NER F1, QA F1/EM, classification (CLS/PAWS-X), XNLI	Slightly lower sample efficiency than DeBERTaV3, but $\sim$40–50% faster pretraining; competitive accuracy (Antoun et al., 11 Apr 2025)
Patent ModernBERT-PT	WIPO, WIPOEC, HUPD, DatasetCLV (patent classification)	Outperforms general ModernBERT on 3/4 tasks; %%%%6$\rightarrow$7%%%% faster inference than PatentBERT (Yousefiramandi et al., 18 Sep 2025)
Albertina PT-* (Portuguese)	ASSIN 2, PLUE, GLUE-like tasks (PT-BR, PT-PT)	900M models: +2.2 pts RTE accuracy, +1.9 pts STS–$\rho$ vs. BERTimbau. Variant-specific models superior to cross-variant (Rodrigues et al., 2023)

These results are aggregated without inventing new metrics or pseudocode and reflect only those explicitly found in the data.

4. Domain Specialization and Adaptation Techniques

Across all evaluated domains, ModernBERT-PT demonstrates the benefit of domain-adapted continued pretraining or from-scratch monolingual pretraining:

• Two-Phase Adaptation (Biomedical/Clinical): Phase 1 (high mask probability) avoids catastrophic forgetting of general biomedical knowledge; Phase 2 (lower mask rate) improves domain specificity and downstream clinical performance (Sounack et al., 12 Jun 2025).
• Tokenizer Customization: In patent and Chinese settings, frequent domain-specific morphemes and compounds are reflected in the vocabulary to maximize lexical coverage and reduce sequence length (Zhao et al., 14 Oct 2025, Yousefiramandi et al., 18 Sep 2025).
• Inclusion of Non-Target Languages/Modalities (Turkish, Patent): A share of tokens from English, code, and mathematics improves cross-domain robustness (Türker et al., 28 Dec 2025, Yousefiramandi et al., 18 Sep 2025).
• Ablations: Empirical studies confirm the additive impact of RoPE (+1.2 points at $L > 512$), FlashAttention ($k$02× throughput, –40% memory usage), and normalization conventions (Türker et al., 28 Dec 2025, Antoun et al., 11 Apr 2025).

A plausible implication is that architectural scaling and vocabulary tuning yield the largest SOTA leaps in technical and morphologically complex languages.

5. Sample Efficiency, Throughput, and Systems Perspective

ModernBERT-PT is explicitly designed for high-throughput, resource-cost sensitive use cases:

• Training Throughput: Pretraining is $k$140–50% faster than RoBERTa or DeBERTaV3 on matched data (French) (Antoun et al., 11 Apr 2025). TabiBERT reports up to 2.65× inference gains at $k$2 (Türker et al., 28 Dec 2025), while Patent ModernBERT-PT achieves 3× the inference speed of PatentBERT at comparable accuracy (Yousefiramandi et al., 18 Sep 2025).
• Ablation Discoveries: Skipping long-context pretraining (Stage II) in Chinese ModernBERT-PT hurts both throughput (–15%) and CLUE accuracy (–1.2) (Zhao et al., 14 Oct 2025); RoPE and dynamic WWM contribute both to speed, accuracy, and retrieval improvements.
• Fine-Tuning Considerations: ModernBERT variants require careful tuning of learning rates, as sensitivity increases with architectural change and efficiency optimization (Antoun et al., 11 Apr 2025).
• Resource Accessibility: Most variants release both "base" (e.g., 100–150M parameters) and "large" (300M–900M) checkpoints, facilitating use on commodity and scientific hardware (Türker et al., 28 Dec 2025, Rodrigues et al., 2023).

6. Licensing and Community Resources

ModernBERT-PT models and their variants are generally made available under permissive licenses:

• Accessibility: Pretrained weights, tokenizers, and training/evaluation scripts are released for all major languages and domains (Türker et al., 28 Dec 2025, Rodrigues et al., 2023, Sounack et al., 12 Jun 2025).
• Reproducibility: TabiBench (Turkish) and the CLUE/SimCLUE protocols (Chinese) ensure unified fine-tuning/evaluation. Portuguese (Albertina PT-*) provides task splits and full model cards.
• Fine-Tuning and Transfer: Full checkpoint reuse, including token embeddings and MLM head (as in Portuguese), appears to yield convergence and accuracy advantages over partial or full re-initialization (Rodrigues et al., 2023).

7. Comparative Limitations and Interpretative Remarks

• Architectural vs. Data Effects: Comparative studies with DeBERTaV3 (e.g., French) show that architectural changes alone (FlashAttention, RoPE, GeGLU) primarily benefit speed and long-context capacity, whereas raw sample efficiency remains slightly higher for DeBERTaV3-style models; benchmark saturation is a growing concern (Antoun et al., 11 Apr 2025).
• Adoption Considerations: Model choice between ModernBERT-PT and DeBERTaV3/PatentBERT/RoBERTa should be informed by task-specific sample efficiency vs. throughput/latency requirements (Antoun et al., 11 Apr 2025, Yousefiramandi et al., 18 Sep 2025).

A plausible implication is that ModernBERT-PT architectures are most advantageous in production or real-time workflows demanding both scalability and precision on large-context or domain-specific data, while DeBERTaV3 remains optimal for data-scarce or sample-efficiency-prioritized development.

---

Key references: (Sounack et al., 12 Jun 2025, Zhao et al., 14 Oct 2025, Türker et al., 28 Dec 2025, Antoun et al., 11 Apr 2025, Yousefiramandi et al., 18 Sep 2025, Rodrigues et al., 2023)