The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning

Abstract: Fine-tuning a LLM is a ubiquitous method for enhancing its capability on a specific downstream task. However, prior work has shown that this increase in capability comes with a cost: it can increase a model's tendency to respond to unsafe adversarial prompts, even when fine-tuning with non-adversarial data. We present the first comprehensive empirical study of this phenomenon in multilingual settings by fine-tuning Llama-3.2, Qwen3, and Gemma-3 models using benign data translated across nine languages. We find that safety outcomes are highly sensitive to both the choice of fine-tuning language and the evaluation language, with adversarial compliance rates increasing four-fold in some settings. Multilingual safety drift is decoupled from general capability metrics, and occurs heterogeneously across languages and models. Fine-tuning in non-English languages often induces smaller internal representational drifts than English, but these shifts lead models to default to either exaggerated compliance or refusal. As such, assessing fine-tuning impacts solely in English provides inadequate assurance for deployment. To facilitate further research into these cross-lingual safety blind spots, we release the Multilingual-Benign-Tune dataset and the SORRY-Bench-Multilingual evaluation suite.

• The paper demonstrates that benign multilingual LoRA fine-tuning induces heterogeneous, architecture-dependent safety drift, notably altering adversarial compliance rates.
• It employs a synthetic multilingual dataset across nine languages and evaluates safety using SORRY-Bench, TinyMMLU, and perplexity tests to quantify performance shifts.
• Findings show a decoupling between safety and capability, underlining the need for language-specific safety assessments in global LLM deployment.

The Heterogeneous Safety Impacts of Benign Multilingual Fine-Tuning

Introduction

This paper conducts an extensive empirical and mechanistic investigation into the safety impacts of benign multilingual LoRA fine-tuning on LLMs, analyzing small-scale variants (0.6B-4B parameters) of Gemma-3, Llama-3.2, and Qwen3 families. Prior work established that even non-adversarial, benign fine-tuning can degrade LLM safety, but the intersection of language, fine-tuning, and adversarial compliance—especially at the multilingual level—remains insufficiently understood. The authors probe this by evaluating over 350 models across diverse languages, quantifying the heterogeneity of safety drift post-fine-tuning and exploring its mechanistic underpinnings. Additionally, data artifacts and evaluation protocols for future research are provided.

Methodology

Fine-Tuning and Data

The experimental protocol involves constructing a high-quality, synthetic benign prompt-response dataset (1,000 prompts), translated across nine languages using Gemini 2.5 Flash and Google Cloud APIs and validated by native speakers. LoRA-based fine-tuning is conducted on six base models (two sizes per architecture), with both aggressive and conservative scaling ratios ($\alpha/r$=8, 2) to ensure robustness of drift effects. All fine-tuning is performed for one or three epochs.

Safety and Capability Evaluation

Model safety is measured using SORRY-Bench, a multilingual adversarial compliance test, evaluating both in English and in the fine-tuning ("local") language. Outputs in local languages are back-translated for scoring. Further, TinyMMLU benchmarks gauge post-fine-tuning capability, and WikiText2/Wikipedia datasets assess perplexity. Representational analyses of model internals employ PCA and directional drift in latent activation space.

Empirical Results

Heterogeneity in Safety Outcomes

Benign multilingual fine-tuning induces highly varied, language- and architecture-dependent safety drift. Small models in particular (e.g., Gemma-3-1B-IT) show drastic increases in adversarial compliance rates following as little as one epoch of fine-tuning; for instance, in some settings, compliance rates with adversarial prompts quadruple. The magnitude and direction of safety drift differ between architectures and between language of fine-tuning and evaluation. Critically, local language evaluations reveal latent vulnerabilities masked by English-only testing.

Figure 1: Differential impacts of benign multilingual fine-tuning on safety; compliance rates vary widely depending on fine-tuning language and evaluation language across model families.

Amplification Effects of Non-English Tuning

When fine-tuning is conducted in non-English languages, the resultant changes in adversarial compliance are typically more pronounced compared to English, amplifying both increases and decreases in unsafe behavior. For example, Portuguese fine-tuning in Gemma-3-4B-IT raises local-language adversarial compliance rates by 400%, while other languages and architectures show drastic declines. Amplification is bi-directional, as observed with Qwen models, where non-English fine-tuning can also magnify refusal.

Figure 2: Non-English fine-tuning amplifies the safety impact observed with English fine-tuning; local evaluations typically exhibit larger compliance changes.

Decoupling Safety from Capability

Adversarial compliance shifts are not consistently accompanied by changes in underlying model capability. Across datasets, substantial safety drift occurs with little or no corresponding increase in MMLU scores or reductions in perplexity, evidencing that "alignment" loss is not necessarily symptomatic of broader model collapse or overfitting. This decoupling is architecture-specific: some models (e.g., Gemma-3-1B-IT) display positive correlation between adversarial compliance and capability, others do not.

Figure 3: TinyMMLU scores (capability) are plotted against SORRY-Bench adversarial compliance; no consistent relationship, indicating safety-capability decoupling.

Sensitivity to Pretraining Distribution and Data Quality

The extent of safety drift following fine-tuning correlates with language resource level for small models, more so for local-language evaluations (Spearman $r=0.84$, $p=0.005$ for Llama-3.2-1B-Instruct). This effect weakens for larger models. Translation data quality, most notably for low-resource languages (e.g., Irish), also mediates the degree of drift. For Irish, high translation inaccuracy rates correspond to anomalous drift and poor local-language capability.

Mechanistic Insights

Latent Representation Shifts

Directional analysis of latent space drift between base and fine-tuned models at a critical transformer depth reveals that small representational changes—especially those induced by non-English fine-tuning—can drive models into states where safety behavior defaults are architecture-dependent. For Llama, small internal drift increases adversarial compliance; for Gemma and Qwen, small drift tends toward refusal, while large displacements help restore initial safety.

Figure 4: Euclidean distance in internal representation correlates with compliance drift; relationships are architecture-specific.

Figure 5: Representation drift directions, projected into 2D PCA, cluster by language family, with English fine-tuning consistently distinct from other languages.

Scaling Effects

Experiments with larger parameter count models (up to 14B) demonstrate qualitatively similar heterogeneity in safety drift, though with reduced absolute compliance changes and attenuated sensitivity to resource level.

Figure 6: Larger models retain heterogeneous impacts across languages, though the magnitude of drift is often reduced.

Implications and Future Directions

These findings expose a fundamental gap in current safety protocols that overwhelmingly focus on English-centric evaluation: multilingual fine-tuning precipitates significant and unpredictable safety drift, dependent on subtle interactions between language, architecture, and data properties. Practically, this implies that deployment of small, instruction-tuned LLMs for global or on-device use necessitates multilingual safety evaluation and likely language-specific guardrails.

From a theoretical perspective, the decoupling of safety from capability under benign tuning highlights the brittleness of existing alignment procedures, supporting recent lines of work suggesting that safety principles are not robustly encoded within current LLM architectures. The architectural dependence of how representational drift translates into compliance alterations points to the importance of deeper mechanistic study—potentially via causal mediation in state space—to design models intrinsically robust to small alignment perturbations.

Future directions include:

• Systematic examination of scaling laws beyond 14B parameters,
• More granular study of data quality and translation fidelity effects,
• Interventional studies on internal representations to buffer against safety drift,
• Fine-tuning within non-Indo-European languages or true low-resource contexts,
• Formalizing architectural criteria predictive of safety drift sensitivity.

Conclusion

This work establishes that benign multilingual LoRA fine-tuning induces pronounced, heterogeneous, and architecture-dependent degradation in LLM safety alignment, with drift often decoupled from model capability. The widespread reliance on English-centric safety assessment is therefore unjustified for operational deployment. Mechanistically, safety drift is mediated by subtle representational perturbations whose impacts on refusal/compliance thresholds differ across models. These results mandate multilingual safety testing and provide a concrete foundation for subsequent work on developing globally robust, alignment-preserving LLMs.

Figure 7: Lack of correlation between adversarial and non-adversarial compliance rates following benign fine-tuning, except in cases of apparent model collapse (e.g., Qwen3-0.6B).

Figure 8: Perplexity after local-language fine-tuning; Irish-tuned models exhibit particularly high variance across architectures.

Figure 9: Cosine similarity of model vectors across layers, highlighting where mechanistic divergence between base and fine-tuned models is maximal.