Privacy-Preserving Continual Pretraining

• The paper presents novel methodologies integrating DP-SGD, local DP, and federated protocols to ensure data privacy during continual pretraining.
• It introduces adaptive mechanisms like token-level DP and encrypted synthetic data to balance sensitive data protection with model utility.
• Empirical evaluations reveal stable performance with rigorous privacy guarantees, effectively mitigating catastrophic forgetting and data leakage.

Privacy-preserving continual pretraining refers to the design and deployment of machine learning models—especially large foundation models (FMs), LMs, and energy forecast models—that are continually updated or retrained as new data streams in, while rigorously mitigating privacy risks inherent to sensitive data. Current strategies integrate differential privacy (DP), local DP variants, privacy-guided memory regularization, teacher-student distillation, encrypted synthetic data, and federated/asynchronous update protocols to assure that no private raw data or identifiable features leak from participants, images, or corpora throughout the pretraining lifecycle. This article synthesizes the prevailing methodologies, theoretical guarantees, architectural innovations, empirical results, and open technical challenges in the field.

1. Conceptual Foundations and Privacy Threat Models

Contemporary continual pretraining operates over temporally evolving data distributions or dynamically expanding participant sets. Privacy threats in this paradigm include the unintended memorization of rare user data, the leakage of sensitive features via model outputs or token embeddings, and the aggregation-induced exposure present in federated or distributed settings. The gold standard for mitigation is differential privacy, with formal guarantees expressed as $(\epsilon, \delta)$-DP bounds on all learning mechanics, though variants such as local differential privacy (LDP), dχ-privacy, token-level DP, and deterministic encryption are also deployed (Yin et al., 2022, &&&1&&&, Zhan et al., 16 Sep 2025, Liu et al., 9 Jan 2026).

2. Differential Privacy Mechanisms for Continual Pretraining

Differential privacy is implemented primarily through DP-SGD, where per-example gradients are clipped and Gaussian noise (proportional to sensitivity and privacy parameters) is added before gradient aggregation. In continual pretraining, privacy bounds accumulate linearly across tasks if mechanisms are correctly composed (e.g., via moments accountants leveraging Rényi DP or Gaussian DP composition) (Desai et al., 2021, Bu et al., 2024).

A DP-SGD protocol for continual pretraining typically involves:

• Sampling a mini-batch at each step.
• Computing and clipping per-sample gradients to norm $C$.
• Aggregating these gradients and adding noise $\mathcal{N}(0,\sigma^2 C^2 I)$, where $\sigma$ is chosen to match $(\epsilon, \delta)$ under a configured privacy accountant.
• Updating model parameters with the noisy gradient.
• Tracking privacy loss over continued updates, ensuring composition and bounding sensitivity relative to all involved data or episodic memory blocks.

A rigorous moments accountant, as described by Desai et al., tracks privacy budget spent at each stage, applying sampling strategies that minimize quadratic privacy loss accumulation (Desai et al., 2021).

3. Local and Token-level Privacy: Adaptive Mechanisms

Local differential privacy (LDP), dχ-privacy, and token-level DP mechanisms grant more granular privacy control. The dχ-privacy framework scales the indistinguishability guarantee proportionally to the metric space distance (e.g., Euclidean distance of token embeddings). Formally, $$Pr[M(x) = y] / Pr[M(x') = y] \leq \exp(\eta \cdot d(x, x'))$$ where $\eta$ is the privacy parameter and $d(\cdot, \cdot)$ is the chosen metric (Qu et al., 2021).

Token-level dynamic DP as in PeCL computes each token's sensitivity, allocates per-token $\epsilon_i$, and injects calibrated Gaussian noise, balancing privacy against task importance. Sensitivity is estimated with task-conditioned contextual scores fused via $$s(t_i) = 1 - \exp(-[\alpha \cdot \text{Score}_1(t_i) + (1-\alpha) \cdot \text{Score}_2(t_i)])$$, and memory sculpting further discourages retention of sensitive knowledge in adapters and weights (Zhan et al., 16 Sep 2025).

4. Federated and Clustered Privacy-Preserving Pretraining

Federated protocols such as FedCCL decouple privacy from centralization by enforcing decentralized knowledge-sharing where no raw data or gradients traverse between clients and server. FedCCL features:

• Static, metadata-derived clustering (DBSCAN over coordinates/orientations); no dynamic reclustering phase.
• Three-tier model hierarchy: global, cluster-specific, and local models supporting personalized and cluster-specialized learning.
• Asynchronous aggregation: model updates are weighted by sample counts and integrated flexibly using non-blocking, lock-protected averaging (Helcig et al., 28 Apr 2025).
• Continual onboarding: new clients instantly benefit by pulling appropriate cluster/global models, with local fine-tuning on private data.

Federated approaches maintain utility and privacy stability under highly dynamic client populations. Empirical evaluation (MAPE energy error: FedCCL location-cluster $3.93\% \pm 0.21\%$) confirms state-of-the-art performance in solar forecasting with strict privacy guarantees (Helcig et al., 28 Apr 2025).

5. Privacy-Preserving Knowledge Retention and Replay

Catastrophic forgetting and domain-shift are central challenges in continual image and text model pretraining. Privacy-aware continual self-supervised learning frameworks leverage privacy-compliant knowledge replay buffers:

• Instead of storing raw past data, they retain latent representations (non-invertible deep features) as replay buffers, e.g. $B = \bigcup_{t'\lt t} Z^{(t')}$ where $Z^{(t')}$ are deep encodings (Tasai et al., 31 Oct 2025).
• At each stage, the model is regularized using losses over the replay buffer (latent-replay loss $\mathcal{L}_{\text{replay}}$), and feature distillation mechanisms such as Wasserstein distance knowledge distillation (WKD) and Batch-Knowledge Ensemble (BKE), which align new-stage feature distributions with previous ones.
• Full training objective combines self-supervised loss, latent-replay, and feature-distillation: $$\mathcal{L}_t = \mathcal{L}_{\text{SSL}}(D_t) + \alpha \mathcal{L}_{\text{replay}}(B) + \beta \mathcal{L}_{\text{distill}}(B, D_t)$$.

Empirical results report domain-shift robustness, e.g. J-MID SARS-CoV-2 CT evaluation ACC $0.873\pm0.03$ (latent replay + distillation), significantly surpassing standard joint or single-domain MAE baselines (Tasai et al., 31 Oct 2025).

6. Encrypted Synthetic Data for Confidentiality in Continual Pretraining

Synthetic encrypted data synthesis is a novel technique exemplified by entity-centric frameworks that systematically replace PII with ciphertext via deterministic encryption (e.g, AES-ECB) (Liu et al., 9 Jan 2026). Steps include:

• Entity extraction from corpus using NER models, regular expressions, and LLMs.
• Weighted-graph construction of entity associations using LLM-prompted scoring.
• Synthesis of question-answer/relational sentences by prompting LLMs with tuples of entities, with prior encryption of all identified PII.
• Pretraining is conducted on the fully encrypted synthetic corpus, ensuring PII is never available in plaintext to the model or pretraining environment.

While ECB encryption assures confidentiality under standard AES assumptions, equality and frequency leakage remain due to determinism. Experiments demonstrate that models pretrained on encrypted synthetic data yield substantial accuracy improvements over base and direct encryption approaches (QuALITY QA: EncS-CPT $0.600$, base $0.424$), with PII access gated by decryption keys held only by authorized data owners (Liu et al., 9 Jan 2026).

7. Empirical Performance, Utility–Privacy Trade-offs, and Practical Recommendations

Privacy-preserving continual pretraining frameworks report a robust balance between utility and privacy. Representative figures include:

• ViT-Base continual DP pretraining: $\epsilon = 8$, $41.5\%$ top-1 accuracy on ImageNet-21k, only 10% public data in warm-up stage, matching or exceeding prior DP models (Bu et al., 2024).
• Legal NLP: BERT with DP-SGD, $\epsilon \approx 5$, macro-F1 on CaseHOLD $0.636$ (vs non-private $0.617$), massive batch sizes required for utility (Yin et al., 2022).
• Token-level DP continual learning: PeCL avg accuracy $0.535$, outperforming static-DP baselines and mitigating catastrophic forgetting (Zhan et al., 16 Sep 2025).
• Catastrophic forgetting in DP-CL: careful per-task $\epsilon$ budgeting keeps utility degradation linear rather than quadratic across task sequences (Desai et al., 2021).
• Synthetic encrypted pretraining (EncS-CPT): accuracy penalty ($\approx 3$–$7$ pp vs unencrypted synthetic) offset by PII confidentiality and continued QA/training capabilities (Liu et al., 9 Jan 2026).

Best practices distilled from empirical studies include:

• Prioritize a brief, non-private public warmup to flatten curvature and increase convergence speed; then apply DP-SGD with optimal batch size and tight accounting.
• For federated/clustered settings, leverage static clustering and asynchronous update protocols to minimize coordination and ensure plug-and-play model onboarding.
• Apply sensitivity-adaptive privacy budgets (token-level DP) to balance privacy protection for sensitive tokens against overall model utility.
• Encrypted synthetic pretraining is effective for domains where raw data confidentiality is vital but persistent knowledge retention is needed.

• "FedCCL: Federated Clustered Continual Learning Framework for Privacy-focused Energy Forecasting" (Helcig et al., 28 Apr 2025)
• "Privacy-Aware Continual Self-Supervised Learning on Multi-Window Chest Computed Tomography for Domain-Shift Robustness" (Tasai et al., 31 Oct 2025)
• "Forget What's Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learning" (Zhan et al., 16 Sep 2025)
• "Natural Language Understanding with Privacy-Preserving BERT" (Qu et al., 2021)
• "Privacy-Preserving Models for Legal Natural Language Processing" (Yin et al., 2022)
• "Continual Learning with Differential Privacy" (Desai et al., 2021)
• "Pre-training Differentially Private Models with Limited Public Data" (Bu et al., 2024)
• "Continual Pretraining on Encrypted Synthetic Data for Privacy-Preserving LLMs" (Liu et al., 9 Jan 2026)
• "Mitigating Unintended Memorization in LLMs via Alternating Teaching" (Liu et al., 2022)