Domain-Specific Pre-training Corpus

• Domain-specific pre-training corpus is a curated text collection that adapts language models through rigorous cleaning, sampling, and tokenization specific to an industry.
• It employs tailored data sourcing, filtering, and preprocessing pipelines to ensure precise alignment with target domain characteristics.
• Using domain-adaptive strategies, these corpora enhance downstream task performance, improving results in classification, QA, and information extraction.

A domain-specific pre-training corpus is a text collection curated to match the linguistic, topical, and stylistic characteristics of a particular vertical (e.g., finance, healthcare, law, science, government, or E-commerce), and is used to pre-train or adapt LLMs or embedding architectures for enhanced performance on downstream tasks within that domain. The construction, maintenance, and optimization of such corpora involve specific data sourcing, rigorous cleaning, targeted sampling or filtering, and often specialized tokenization and supervision mechanisms to ensure maximal utility for the intended applications.

1. Definitions, Motivation, and Core Distinctions

A domain-specific pre-training corpus S is contrasted with a generic corpus G by both scale and distributional proximity to a task distribution D_t. Formally, S is domain-specific if |S| ≪ |G| and D_S is empirically closer (e.g., by n-gram coverage or expected L1-accuracy) to D_t than D_G. Empirical studies show that models pre-trained on S (alone or via domain-adaptive continual pre-training) yield representations with improved transfer, especially if S's token and n-gram statistics closely track those of target data (Gonzalez-Gutierrez et al., 30 May 2025). Notably, optimal utility occurs when S is sufficiently large (≥10M words) and distributionally aligned; otherwise, even large S may fail to yield gains over G. Domains commonly addressed include finance (Lu et al., 2023, Xie et al., 2023), healthcare (Arannil et al., 2024), government and news (Liu et al., 2023), science (Hu et al., 2022), law (Nandy et al., 2023), and industrial diagnostics (Kumar et al., 23 Nov 2025).

2. Corpus Construction Pipelines and Data Sources

Effective pipeline design typically comprises:

1. Source Identification and Collection: Data originate from crawlable web platforms, proprietary databases, government or legal repositories, corporate filings, specialized forums, knowledge graphs, or seed-driven web search (Liu et al., 2023, Lu et al., 2023, Arannil et al., 2024, Kumar et al., 23 Nov 2025). For example, MiChao-HuaFen 1.0 curated ≈70M Chinese news/government documents from 2022-vintage web portals (Liu et al., 2023), while BBT-FinCorpus aggregated 300GB from corporate announcements, research reports, news, and social media (Lu et al., 2023).
2. Domain Relevance and Authority Filtering: Inclusion criteria may enforce coverage of key subtopics (e.g., economic indicators in finance), temporal continuity, and compliance requirements (licensing, PII-safety) (Liu et al., 2023).
3. Cleansing and Preprocessing: Multi-stage pipelines typically involve
   • Keyword-based PII/spam/ad removal
   • Rule-based tag and noisy-content stripping
   • De-duplication (exact and approximate, e.g., MinHash or shingling)
   • Consistent normalization (punctuation, Unicode, width conversion, whitespace), and, when necessary, formatting to a canonical schema (e.g., Markdown) (Liu et al., 2023, Lu et al., 2023).
4. Supervisory Annotation/Labeling: For structured supervision, pipelines may distill LLM annotations into efficient classifiers to assign domains by topic or format (e.g., WebOrganizer's two-axis annotation, (Wettig et al., 14 Feb 2025)). Sentence/document embeddings may drive triplet supervision, as in FastDoc (Nandy et al., 2023).
5. Seed Generation & Synthetic Data: In scenarios lacking extensive real data, synthetic seeds are generated via prompting strong LLMs with controlled parameters, followed by nearest-neighbor mining from unlabeled corpora to harvest domain-relevant content (DoPAMine, (Arannil et al., 2024); DiagnosticSLM, (Kumar et al., 23 Nov 2025)).
6. Periodic Refresh and Versioning: High-value corpora are versioned, with regular recrawls ensuring content freshness and reproducibility; MiChao-HuaFen 1.0 maintains quarterly releases with stable schema guarantees (Liu et al., 2023).

3. Data Selection, Sampling, and Optimization

Efficient curation of domain-specific pre-training corpora often eschews naïve upsampling or random sampling in favor of more targeted, utility-driven selection:

• Importance Sampling with n-gram or Embedding Features: Target-aware pipelines featurize each document via multi-granular token counts or embeddings. Sampling probabilities are weighted by the ratio of their feature distributions in target vs. background corpora (Chang et al., 2024). The importance weight for sample x_i is $w_i = \hat{p}(z_i)/\hat{q}(z_i)$, facilitating budgeted sampling where small (1%) domain-matched subsets can match or surpass full corpus performance.
• Task-Aware Selection: Embedding-driven schemes (ETS-DACP) select samples whose representations are closest to those of the downstream task corpus, while task-agnostic sampling (ETA-DACP) prioritizes novelty (high perplexity) and syntactic diversity (POS-entropy). These methods dramatically reduce required token volume without generalization loss (Xie et al., 2023).
• Graph-Based Filtering (TextGram): Constructs a sentence similarity graph bridging in-domain anchors and candidate general examples, then applies PageRank to prioritize examples for inclusion. This can yield a 75% compute reduction with slightly higher downstream accuracy (Hiwarkhedkar et al., 2024).
• Scaling Law-Guided Mixture Optimization: The D-CPT Law parameterizes loss as a function of model size, token count, and domain/general mixing ratio, enabling prediction of optimal sampling regimes with minimal compute overhead (Que et al., 2024). Further, full cost/utility trade-off is analyzed in scaling-curve approaches, ensuring optimal resource allocation across sources and scales (Ostapenko et al., 29 Jul 2025).

4. Objective Functions, Tokenization, and Pre-training Regimes

Domain-specific corpora facilitate customized model pre-training objectives and tokenization schemes:

• Objectives: Masked Language Modeling (MLM; with or without span masking) is the most prevalent (Lu et al., 2023, Liu et al., 2023), often enhanced with domain-adaptive masking rates (e.g., increased numerical token masking in finance (Lu et al., 2023)). Some regimes add next-sentence or sentence-order prediction (policy/government texts (Liu et al., 2023)). Document-level or triplet-based objectives, exploiting metadata or taxonomy, can replace token-level MLM for orders-of-magnitude compute reduction (Nandy et al., 2023).
• Tokenization: Domain-adaptive tokenizers (vocabulary induced on S) ensure high-frequency entities, collocations, or technical terms are preserved as single tokens (e.g., financial institution names in BBT-FinCorpus (Lu et al., 2023); joint n-gram BPE merges (Chang et al., 2024)). Tokenizer adaptation is critical to maintain granularity and reduce out-of-domain artifacts.
• Pre-Training Regimes:
  • Training-from-scratch on S is viable for sufficiently large, distributionally matched corpora.
  • Domain-Adaptive Continual Pre-training (DACP/CPT) involves further MLM or causal language modeling on S, initialized from a generic model; domain-general vs. in-domain balance is set by mixture ratios, potentially optimized via scaling laws (see D-CPT Law (Que et al., 2024)).
  • Mixed-format or multi-task regimes exploit structure (e.g., infobox triples, section titles) for multi-format objectives, as in tourism domain HKLM (Zhu et al., 2021).

5. Corpus Size, Scaling Laws, and Resource Efficiency

Corpus size exerts diminishing returns on downstream performance, often logarithmic or sub-linear in scaling (Sanchez et al., 2022). Representative findings:

Corpus (D)	NCBI-disease F₁	PubMedQA Acc.	HoC F₁	Source
General (20 GB)	84.3	54.4	79.0	(Sanchez et al., 2022)
PubMedBERT (~21GB)	87.8	55.8	82.3	(Sanchez et al., 2022)
4 GB in-domain	87.7	54.9	81.1	(Sanchez et al., 2022)
8 GB	87.9	53.4	82.5	(Sanchez et al., 2022)
12 GB	88.0	55.2	81.4	(Sanchez et al., 2022)

Diminishing marginal improvement beyond ~4–8 GB in tight domains is the norm. Scaling-law–guided corpus construction (D-CPT Law (Que et al., 2024), scaling-law utility estimation (Ostapenko et al., 29 Jul 2025)) enables practitioners to predict required token budgets and allocate annotation/training resources cost-efficiently. For continual or annealed pre-training, multi-curve fits prevent erroneous point-estimate-based sourcing, crucial when rankings invert with scale (Ostapenko et al., 29 Jul 2025).

Efficient methods like FastDoc can reduce compute by 500×–4500× versus MLM, exploiting metadata/taxonomy and batch sentence-level representation (Nandy et al., 2023).

6. Downstream Impact, Benchmarks, and Applications

Domain-specific pre-training corpora consistently lead to marked improvements in in-domain downstream tasks, including:

• Text classification, QA, summarization, NER: Models pre-trained on vertical-specific corpora outperform generic baselines, with gains up to +25% MCQ accuracy (diagnostic SLMs (Kumar et al., 23 Nov 2025)), +13.6% PubMedQA (DoPAMine (Arannil et al., 2024)), and consistent F1 or accuracy boosts in government (Liu et al., 2023) or finance (Lu et al., 2023, Xie et al., 2023).
• Benchmarks: Domain-tuned corpora underlie the construction of new discipline-specific benchmarks (e.g., BBT-CFLEB for finance (Lu et al., 2023)), TCM-EXAM/TCM-EHR for Chinese medicine (Yang et al., 2023), as well as facilitating the robust evaluation of scaling and curation strategies (Ostapenko et al., 29 Jul 2025, Gonzalez-Gutierrez et al., 30 May 2025).
• Cross-domain generalization: Importance-sampled, multi-granular or domain-mixed corpora can preserve or even enhance performance on non-target tasks; domain mixing along topic/format axes further expands transfer (Chang et al., 2024, Wettig et al., 14 Feb 2025).
• Advanced applications: Multi-modal (paired image/text), procedural text, and knowledge-graph–aligned corpora extend vertical coverage and support tasks such as cross-document summarization, automation of compliance workflows, and information extraction from complex regulatory or engineering documents (Liu et al., 2023, Kumar et al., 23 Nov 2025, Zhu et al., 2021).

7. Best Practices and Open Challenges

Key recommendations include assembling sufficiently large, diversity-balanced corpora for each vertical; grounding data selection in explicit similarity measures (e.g., n-gram coverage, embedding alignment); leveraging scaling laws and mixture optimization for cost-efficiency; and maintaining transparent, versioned releases for reproducibility. Tokenization should always be adapted to the domain vocabulary, and both synthetic and real seed data are valuable, with careful filtering and deduplication.

Open challenges encompass:

• Determining optimal domain/general mixing ratios dynamically across domains and model scales (D-CPT, (Que et al., 2024)).
• Handling low-resource or highly specialized domains where seed or web-retrieved data is inherently sparse or noisy.
• Integrating structured, semi-structured, and unstructured text, including emerging multi-modal data types.
• Quantifying and mitigating catastrophic forgetting in continual domain specialization.

These advances, tied to both the theory and empirical methodologies from recent work, enable the principled construction and continual adaptation of high-value domain-specific pre-training corpora for next-generation LLMs.

---

References:

(Liu et al., 2023, Lu et al., 2023, Xie et al., 2023, Arannil et al., 2024, Ostapenko et al., 29 Jul 2025, Gonzalez-Gutierrez et al., 30 May 2025, Que et al., 2024, Yang et al., 2023, Hiwarkhedkar et al., 2024, Kumar et al., 23 Nov 2025, Sanchez et al., 2022, Wettig et al., 14 Feb 2025, Nandy et al., 2023, Chang et al., 2024, Hu et al., 2022, Zhu et al., 2021, Zhang et al., 2021, Wang et al., 2019)