In-Domain Contrastive Learning

• In-domain contrastive learning is a representation learning approach that uses domain-restricted contrastive objectives to enhance intra-class discrimination and robustness.
• Methodologies involve specialized batch construction, adversarial augmentation, and adaptive loss weighting to improve performance in classification, OOD detection, and low-resource adaptation.
• Empirical validations show that in-domain strategies yield significant gains over traditional methods, particularly in overcoming label shift and boosting domain-invariant representations.

In-domain contrastive learning is a suite of representation learning methodologies in which contrastive objectives—operating exclusively within the data of a particular domain or label space—are optimized to yield embeddings with improved discrimination, invariance, or transfer, guided by task-specific constraints. Unlike cross-domain or global contrastive learning, in-domain approaches restrict their positive/negative pair constructions, batch sampling, or loss calculations to individual classes, domains, or similarity metrics, enabling targeted improvements in intra-domain discrimination and robustness. This paradigm underpins advances in supervised detection, unsupervised adaptation, domain-invariant representation, and disentanglement, with recent methods leveraging sophisticated augmentation, loss weighting, and adaptation protocols.

1. Core Methodologies in In-Domain Contrastive Learning

In in-domain supervised contrastive learning (SCL), each mini-batch of $N$ samples $(x_i, y_i)$ is processed through an encoder to obtain normalized feature representations $s_i = f(x_i)/\|f(x_i)\|$. The loss for an anchor $i$ is computed over the set of positives $P(i) = \{p \neq i: y_p = y_i\}$, with all other in-batch samples (negatives) distributed across distinct labels. The supervised contrastive loss is

$$\ell_i = -\frac{1}{|P(i)|} \sum_{p\in P(i)} \log \left[ \frac{\exp(s_i\cdot s_p/\tau)}{\sum_{a=1,a\neq i}^N \exp(s_i\cdot s_a/\tau)} \right]$$

and the total loss is $L_{SCL} = \frac{1}{N} \sum_{i=1}^N \ell_i$ (Zeng et al., 2021).

Augmentation strategies within in-domain learning are tailored to the modality and task. For example, adversarial augmentation synthesizes hard positives via perturbation in the latent space (NLP settings), while vision methods may focus on generating additional semantically valid views or restricting augmentations to domain attributes (Li et al., 2020, Kahana et al., 2022).

Domain-wise contrastive losses restrict the negative set in InfoNCE-style objectives to examples within the same domain, effectively encouraging domain-invariant representations when domain is a sensitive or nuisance factor (Kahana et al., 2022). In the transfer context, domain-specific contrastive objectives enable within-domain discrimination without enforcing explicit inter-domain matching, often yielding more robust boundaries under label shift (Li et al., 2020).

2. Architectural and Algorithmic Details

Architectural details vary by application: text tasks commonly employ BERT, BiLSTM, or similar encoder architectures with pre-/post-projection normalization, while vision tasks utilize ResNet-based encoders initialized from large-scale pretraining. Projection heads ($g(\cdot)$ as MLP) are frequently employed to map deep features into the contrastive space, with representation dimensionalities tuned for downstream accuracy (e.g., $d=128$, $p=256$) (Zeng et al., 2021, Li et al., 2020, Mu et al., 2023).

Batch construction is typically stratified to ensure multiple in-domain samples per class or sub-domain; batch sizes of $B=128$–$512$ are empirically effective across modalities (Zeng et al., 2021, Pavlova et al., 19 Oct 2025).

Optimization follows prevalent best practices (Adam or SGD with warmup, weight decay, temperature hyperparameters $\tau \sim 0.05$–$0.1$), often organized into multi-stage protocols:

• Pre-train with contrastive objectives (optionally adversarial views/augmentations),
• Fine-tune with classification or large-margin losses,
• For domain adaptation (MOSAIC), an inaugural stage (vocabulary augmentation) is introduced, followed by joint contrastive/MLM training and final contrastive-only refinement (Pavlova et al., 19 Oct 2025).

Adaptive weighting of loss components using task-uncertainty metrics (learned temperatures $\sigma_c^2$) enables robust multi-similarity contrastive learning, suppressing the influence of noisy or ambiguous metrics (Mu et al., 2023).

3. Empirical Validation and Benchmarking

In-domain contrastive learning methods demonstrably outperform cross-entropy and unsupervised baselines in classification, OOD detection, and domain adaptation tasks.

• On the CLINC-Full benchmark, LSTM+GDA trained with SCL+CE achieves OOD Recall=66.80% and OOD F1=67.68% versus CE baseline Recall=63.72%, F1=65.23%. Few-shot settings (10% data) yield an 18.8% relative OOD F1 improvement (Zeng et al., 2021).
• In cross-domain sentiment classification, in-domain contrastive learning with BERT yields target domain accuracy improvements of 0.98–1.09% over BERT-base, with strongest relative gains under significant label distribution shift (Li et al., 2020).
• In domain-invariant representation learning (DCoDR framework), in-domain domain-wise contrastive loss achieves near-optimal invariance and informativity on Cars3D, SmallNorb, and Shapes3D, outperforming adversarial and variational baselines across invariance (e.g., Cars3D Inv=0.005) and retrieval metrics (Cars3D retrieval 97%) (Kahana et al., 2022).
• Multi-Similarity Contrastive Learning (MSCon) outperforms both supervised and unsupervised baselines on Zappos50k and MEDIC (e.g., MSCon top-1 accuracy 97.17% vs SupCon 96.95% on Zappos50k Category) (Mu et al., 2023).
• MOSAIC achieves up to 13.4% absolute improvement in NDCG@10 in extremely low-resource adaptation scenarios, demonstrating robust adaptation even with minimal in-domain data (Pavlova et al., 19 Oct 2025).

Ablation studies consistently show that the core benefit arises from the contrastive objective itself, not simply larger batch sizes or hidden dimensions. Joint objective balancing, augmentation design, and batch curation are critical to optimal performance.

4. Key Mechanisms: Augmentation, Loss Weighting, and Domain Restriction

Augmentation is central to in-domain contrastive learning:

• In NLP, adversarial augmentation leverages gradient-based perturbations in embedding space; other techniques include synonym substitution and back-translation for robust positive generation (Zeng et al., 2021, Li et al., 2020).
• In vision, augmentation must avoid spurious domain information leakage. For domain invariance, only within-domain negatives are valid; care must be taken to avoid shortcut solutions (feature suppression via collapse) (Kahana et al., 2022).

Loss weighting is advanced via adaptive scheduling and uncertainty-based reweighting. The use of learned task uncertainties ($\sigma_c^2$ in MSCon) allows dynamic suppression of unreliable similarity signals, leading to robustness against noisy supervision and improved generalization (Mu et al., 2023).

Domain-restriction mechanisms span:

• Restricting negatives/positives to in-domain samples,
• Limiting MLM losses to domain-specific vocabulary,
• Curating batch composition for within-domain variance (Pavlova et al., 19 Oct 2025, Kahana et al., 2022).

Adaptive objective weighting (e.g., $\lambda$, $\beta$, $\alpha$) is necessary; excessive weighting of secondary losses (MLM, entropy) can degrade core sentence or representation geometry (Li et al., 2020, Pavlova et al., 19 Oct 2025).

5. Practical Implementations and Adaptation Protocols

MOSAIC exemplifies a multi-stage adaptation strategy:

• Stage 1: Vocabulary expansion introduces domain-specific tokens; embeddings are initialized as means of their subwords, with encoder parameters frozen.
• Stage 2: Joint in-domain contrastive learning and masked LM, restricting the mask and denominator to domain vocabulary, governed by a joint weight $\alpha$ (optimal $\alpha=0.3$).
• Stage 3: Contrastive-only recovery to restore or preserve the sentence embedding manifold (Pavlova et al., 19 Oct 2025).

Empirically, restricting masked LM to domain tokens and finely balancing its contribution prevents collapse of the embedding space, while final contrastive recovery re-establishes global semantic discrimination. For low-resource domains, even a few thousand high-precision in-domain pairs enable substantial gains.

In multi-similarity learning, separate projection heads and per-task pseudo-likelihood weighting enable the model to integrate multiple domain-relevant similarity signals without uniform collapse or overfitting on noisy cues (Mu et al., 2023).

6. Comparative Summary of In-Domain Contrastive Strategies

Method	Domain Restriction Mechanism	Loss Structure	Domain Adaptation Feature
Supervised CL (NLP)	Positives/negatives per label	Supervised contrastive + optional adversarial	Robust to intra-class, inter-class var.
In-domain InfoNCE	Batches/domain-specific	Unlabeled CL per domain	Resilient to label shift (sentiment)
Domain-wise CL (Vision)	Negatives within domain only	InfoNCE with same-domain negative restriction	Enforces domain-invariant z
MSCon (Multi-similarity)	Negatives/positives by similarity	Multi-head, weighted contrastive	Handles overlapping/heterogeneous sim.
MOSAIC	Domain token vocab; mask restrict.	Joint InfoNCE+MLM (domain tokens only)	Extreme low-resource adaptation

7. Theoretical and Empirical Insights, Recommendations

In-domain contrastive learning achieves sharper clustering and increased inter-class separation by making intra-class variance minimization and inter-class repulsion explicit in its loss. Under domain shift, domain-wise contrastive objectives avoid overfitting or boundary drift associated with indiscriminate distribution matching. Limiting negative sampling and loss computations to the domain or class, supplemented by careful augmentation and loss balancing, addresses shortcut solutions and prevents collapse to trivial invariance (Kahana et al., 2022).

Empirical evidence across text and vision tasks consistently motivates the use of in-domain contrastive learning strategies, particularly in low-resource domains or where strong invariance/disentanglement is required. Pretraining, limited augmentations avoiding attribute leakage, and sequential optimization strategies are recommended. Adaptive loss weighting and data curation are critical to avoid either under-utilization of auxiliary signals or geometry-destroying dominance.

A plausible implication is that further advances may come from learning strategies that unify dynamic domain partitioning, automated augmentation policy learning, and continual adaptation mechanisms within the in-domain contrastive learning framework.