Continued Pretraining (CoPT) Explained

• Continued Pretraining (CoPT) is a technique that further trains a large, pretrained model on domain-specific data to enhance performance in targeted applications.
• It leverages various unsupervised objectives such as masked language modeling, causal prediction, and span-corruption to efficiently adapt foundation models.
• Mitigation strategies like data replay, curriculum learning, and parameter regularization are crucial to preventing catastrophic forgetting during adaptation.

Continued Pretraining (CoPT)

Continued Pretraining (CoPT) refers to the process of further training a large-scale pretrained model—typically a Transformer-based language, vision, or speech model—on new data distributions to improve adaptation to specific domains, languages, tasks, or modalities. This approach leverages the fundamental representations learned during extensive initial pretraining but infuses new knowledge or skills via additional unsupervised or self-supervised objectives, and is now a dominant paradigm for efficient domain and language adaptation across NLP, vision, and speech. In contemporary practice, CoPT encompasses diverse methodologies, from domain-adaptive LM pretraining and multilingual adaptation to specialized strategies for reasoning, chat alignment, or multi-modal input.

1. Objectives and Core Methodologies

The primary objective of CoPT is to exploit knowledge encoded in a foundation model and efficiently extend its capacity to a new target domain or distribution. The standard process is:

• Start from a checkpoint pretrained on a large, heterogeneous dataset (e.g., text corpora, web crawl, generic speech, or images).
• Further pretrain with an unsupervised or self-supervised objective (next-token prediction, masked LM, span-corruption, etc.) on new data: e.g., an in-domain, language-specific, or modality-specific corpus.
• Optionally, refine with supervised or preference-based objectives after CoPT.

Core loss functions include:

• Causal Language Modeling (auto-regressive next-token loss): $$L(\theta) = –\sum_{t=1}^N \log P_{\theta}(x_t | x_{<t})$$ (used for generative LMs and in domain-specialization) (Kawakami et al., 25 Apr 2025, Almeida et al., 14 Dec 2025).
• Masked Language Modeling: standard for encoder-based models, where random tokens or spans are masked and the model predicts them (Shi et al., 2023, Cossu et al., 2022).
• Span-Corruption: as in T5—masking contiguous sequences replaced by special sentinels, the decoder reconstructs masked content (Piau et al., 2024).
• Multi-modal/codec adaptation: models consume discrete tokens from other modalities (e.g., speech codecs) alongside standard text (Shi et al., 24 Feb 2025).

Variants include prompt-based CoPT (PCP), preference-based objectives (as in reasoning optimization), masking strategies conditioned on domain discriminativity, and mixed-modality objectives.

2. Data, Corpus, and Mixture Strategies

The construction and composition of the pretraining corpus are central to CoPT's effectiveness:

• Domain-specific Data: Target subdomain corpora (e.g., Japanese medical exam questions (Kawakami et al., 25 Apr 2025), Portuguese STEM texts (Almeida et al., 14 Dec 2025), music metadata (Tian et al., 18 Nov 2025)).
• Language-adaptation: Corpus in the new target language, often with a blend of the original high-resource language (typically English) to preserve cross-lingual capabilities and avoid catastrophic forgetting (Elhady et al., 30 May 2025, Zheng et al., 2024).
• Data Quality and Filtering: Semantic or topic filtering can yield better results than brute-force scaling. In Portuguese, a 10B STEM/Education subset outperformed a full 100B corpus (Almeida et al., 14 Dec 2025). In music, fine-grained alignment and classifier-filtered datasets improved factuality and QA performance (Tian et al., 18 Nov 2025).
• Data Mixtures and Ratios: Strategic ratios between general and target domain data are essential; mixture schedules may be dynamically adjusted based on model perplexity or other learning signals (Chen et al., 2024, Que et al., 2024).

Several works introduce formal scaling laws for selecting optimal mixture ratios, balancing general and domain-specific data to maximize performance under compute constraints (Que et al., 2024).

3. Catastrophic Forgetting and Mitigation

A central challenge in CoPT is catastrophic forgetting—the unwanted loss of previously acquired knowledge (e.g., English fluency when adapting to a new language, general reasoning capacities during domain specialization):

• Observations: Without explicit safeguards, models rapidly overwrite capabilities; e.g., omitting English data during Basque CPT collapses in-context learning accuracy by over 20 points (Elhady et al., 30 May 2025).
• Mitigation Techniques:
  • Data Replay: Interleave a portion (e.g., 10–30%) of the original pretraining corpus in each batch (Zheng et al., 2024).
  • Curriculum Learning: Include the source language or general data during a “critical period,” then phase it out (Elhady et al., 30 May 2025, Chen et al., 2024).
  • Parameter Regularization: Exponential moving average (EMA) of weights constrains rapid parameter drift (Elhady et al., 30 May 2025).
  • Loss Function Design: Preference or reward-based terms augmented with negative log-likelihood components to enforce both domain-specific alignment and preservation of reasoning skills (Kawakami et al., 25 Apr 2025).
  • Prompt and Format Injection: Instruction templates (e.g., chat tags) during InsCP ensure preservation of RLHF and conversational structure (Chen et al., 2024).

Proper monitoring involves not only tracking target-language/domain perplexity but also parameter drift and out-of-domain performance.

4. Application Areas and Empirical Impact

CoPT is employed across a spectrum of application domains:

Area	Primary Objective	Empirical Impact
Medical LLMs (Kawakami et al., 25 Apr 2025)	Japanese clinical adaptation	+0.065 accuracy, SOTA on IgakuQA
Multilingual LMs (Almeida et al., 14 Dec 2025, Piau et al., 2024)	Low-resource language adaptation	+8–28% NPM/accuracy gains
Speech (DeHaven et al., 2022, Attia et al., 2024, Shi et al., 24 Feb 2025)	Robust ASR, S2ST/TTS/ASR balance	10–36% WER reduction, S2ST enabled
Music (Tian et al., 18 Nov 2025)	Music-entertainment factual QA	0.7759 SimQA, +17% over GPT-4o
Scientific/Math Q&A (Chen et al., 2024)	Reasoning enhancement	+12 MATH, +4 SciEval, +8.8 C-Eval

In all cases, CoPT enables substantial improvements with only a fraction of the full pretraining compute. Results show that modest investments in corpus curation, scheduling, and mixture design can yield larger performance jumps than simply scaling data size.

5. Architectural and Algorithmic Innovations

Several architectural and algorithmic techniques have further expanded CoPT's capabilities:

• Prompt-aware and PCP: Incorporating prompt templates and label words directly into the CoPT objective enhances downstream prompt-tuning and zero/few-shot performance (Shi et al., 2023, Wu et al., 2022).
• Difference-Masking Strategies: Selective masking of high mutual information/low overlap tokens or regions focuses adaptation on target domain concepts (Wilf et al., 2023).
• Embedding Initialization for Multilinguality: OFA leverages aligned static word vectors and low-rank initialization for subwords to accelerate and improve large-scale multilingual adaptation (Liu et al., 2023).
• Scaling Laws: Both general (Zheng et al., 2024) and domain-specific (Que et al., 2024) scaling laws now exist for predicting optimal data/parameter allocation in CoPT under compute constraints.

In the multi-modal frontier, CoPT has enabled text LLMs to handle codec-based speech synthesis, ASR, and S2ST, using parallel vocabularies and loss heads while mitigating forgetting via data mixing (Shi et al., 24 Feb 2025).

6. Limitations, Guidelines, and Best Practices

• Data Limitations: Curated, high-quality domain data consistently outperforms larger, noisier corpora for models with sufficient scale (Almeida et al., 14 Dec 2025). For low-resource languages and domains, careful data selection is critical.
• Resource Efficiency: CoPT bypasses the need for full retraining, with compute savings of 25–50% to reach the same loss as training from scratch (Zheng et al., 2024). Instruction- and prompt-based CoPT require even fewer tokens and less compute (Chen et al., 2024).
• Hyperparameter Sensitivity: Optimal learning-rate schedules, data mixture ratios, and curriculum strategies are task- and domain-dependent. Scaling law–based prediction enables efficient grid search and avoids overfitting to particular data sizes or model scales (Que et al., 2024, Parmar et al., 2024).
• Preservation of Generalization: Best practice is to monitor performance on both the target and original domains/languages, and intervene by replay, regularization, or curriculum strategies if degradation is observed.
• Scalability and Generalizability: Benefits of CoPT are most pronounced at moderate-to-large model scales; gains plateau above ~1–3B parameters for purely monolingual adaptation (Piau et al., 2024).

7. Future Directions

Open directions in CoPT research include the design of adaptive and domain-aware masking heuristics, automated mixture scheduling based on online monitoring, unified scaling law frameworks for increasingly heterogeneous data mixtures, and cross-modal transfer strategies that can simultaneously extend LLMs' capacity in text, speech, and vision without catastrophic interference. Integration with instruction and preference optimization (e.g., RPO (Kawakami et al., 25 Apr 2025)) is likely to become standard, especially in high-stakes and multi-format applications. The full reproducibility of CoPT pipelines, including traceable mixture, curriculum, and downstream monitoring, is emerging as a leading practice (Chen et al., 2024).

---

References: (Kawakami et al., 25 Apr 2025, Elhady et al., 30 May 2025, Shi et al., 2023, Piau et al., 2024, Wu et al., 2022, Cossu et al., 2022, Almeida et al., 14 Dec 2025, Wilf et al., 2023, Tian et al., 18 Nov 2025, Zheng et al., 2024, Chen et al., 2024, Parmar et al., 2024, Liu et al., 2023, Chen et al., 2024, Que et al., 2024, DeHaven et al., 2022, Attia et al., 2024, Shi et al., 24 Feb 2025, Sun et al., 2023).