Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models

Abstract: Instruction-tuned LLMs are increasingly converted into reasoning models through post-training to improve multi-step task performance. This conversion is usually optimized for reasoning accuracy, without explicitly preserving the alignment behavior of the instruction-tuned model, such as safe refusal, bias avoidance, and privacy protection. We ask: does this conversion preserve alignment? We study this question through a trustworthiness audit and find that it is not behavior-preserving by default. For a systematic analysis, we compare reasoning models produced via supervised fine-tuning, RL-based post-training, and distillation against matched instruction-tuned baselines across six trustworthiness dimensions: safety, toxicity, stereotyping and bias, machine ethics, privacy, and out-of-distribution robustness. We observe that reasoning models often improve on reasoning benchmarks but exhibit alignment regressions, including increased toxicity, amplified stereotyping, miscalibrated refusal, and contextual privacy leakage. These regressions are consistent with behavioral drift from the instruction-tuned baseline, measured by KL divergence. Overall, our results point to the broader conclusion that trustworthiness metrics are essential for evaluating reasoning models and should be reported alongside gains in reasoning capability.

• The paper shows that post-training for reasoning improves multi-step accuracy metrics (e.g., math benchmarks) but severely compromises alignment in safety, ethics, and privacy.
• It systematically compares SFT, RL/GRPO, and distillation methods, revealing distinct degradation patterns such as elevated toxicity, bias, and miscalibration.
• The study highlights a critical trade-off between enhanced reasoning capabilities and deployment risks, urging the incorporation of trust-aware objectives in training.

Trustworthiness Degradation Induced by Reasoning Post-Training in LLMs

Introduction

The paper "Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models" (2606.11046) systematically investigates the alignment and trustworthiness consequences of converting instruction-tuned LLMs into reasoning models via post-training. The core assertion is that existing reasoning induction protocols—supervised fine-tuning (SFT) on reasoning traces, RL-based post-training (e.g., GRPO-style), and distillation from teacher models—substantially boost multi-step reasoning accuracy but systematically degrade deployment-relevant alignment properties without careful intervention.

The authors argue that this behavioral erosion is widespread across safety, toxicity, stereotyping and bias, privacy, machine ethics, and out-of-distribution (OOD) robustness, and can be quantitatively linked to distributional drift as measured by KL divergence between the response distributions of baseline and reasoning-enhanced models.

Figure 1: Three dominant pathways for reasoning model construction (SFT, RL/GRPO, and Distillation), each evaluated against aligned instruction-tuned baselines to isolate trustworthiness versus capability effects.

Framework and Evaluation Scope

The evaluation protocol comprises carefully matched instruction-tuned LLM baselines and their reasoning-enhanced variants produced via three canonical reasoning-imposing methods. The paper’s trust audit spans six deployment-critical dimensions:

• Safety: Refusal behavior under adversarial or harmful prompts
• Toxicity: Propensity to output harmful content
• Stereotyping/Bias: Susceptibility to bias and discriminatory outputs
• Machine Ethics: Adherence to shared moral norms, both under benign and value-evasive (jailbreak) prompts
• Privacy: Resistance to leaking personally identifiable information (PII)
• OOD Robustness: Appropriate abstention or refusal out of training distribution (especially post-knowledge cutoff scenarios)

All metrics are assessed on final model outputs, not intermediate reasoning traces, ensuring relevance for real-world model deployment.

Reasoning Gains Versus Alignment Costs

Quantitative results demonstrate substantial reasoning performance improvements after post-training: Math500 Pass@1 increases from 49.8% (Instruct) to 70.8%, GSM8K from 70.7% to 76.4%, and AIME from 6.7% to 20.0%. However, these capability gains are accompanied by pronounced trustworthiness declines:

• Safety scores drop from 80% to 20%
• Machine ethics and privacy both degrade from ~60% to below 26%
• Privacy leak rates escalate dramatically, even when explicit protection demonstrations are provided

  Figure 2: Left: Pass@1 reasoning accuracy across math benchmarks increases sharply post-reasoning training. Right: Simultaneous degradation across six trustworthiness axes compared to instruction-tuned baselines.

Pathway-Specific Degradation Modes

Supervised Fine-Tuning (SFT): Induces the largest and most consistent alignment regressions. Notably, SFT models show:

• Elevated toxicity (increase from 17.72% to 41.93% under benign prompts)
• Catastrophic over-refusal and loss of accuracy on machine ethics tasks (benign refusal rate surges from 0.52% to 82.73%)
• Blanket refusal on OOD questions regardless of context, reflecting degenerate, uncalibrated abstention
• Persistent privacy leaks across nearly all PII types, with >80% leak rates even under protective prompting

  Figure 3: Toxicity rises across all prompt regimes for reasoning models, most acutely for SFT-based approaches, revealing a collapse of prompt-conditioned harm suppression.

RL/GRPO-Style Post-Training: Moderate toxicity increases compared to SFT, but strongest degradation for:

• Stereotyping and bias, with a ~6x increase under untargeted jailbreak prompts
• Calibration failures: under-refusal on OOD queries and over-refusal on benign ethical judgments

  Figure 4: Stereotyping rates increase especially under RL/GRPO training, revealing amplified bias even absent explicit demographic targeting.

Distillation: Some trust-axis preservation (notably, toxicity rates closer to baseline), but:

• Bias and under-refusal on OOD or adversarial prompts remain problematic
• Only partially absorbs privacy-protective behavior from teacher if present

Detailed Trustworthiness Failure Analysis

Toxicity and Bias

Across all three reasoning induction methods, median toxicity and stereotyping rates are elevated compared to instruction-tuned controls, including in regimes where system prompts explicitly prohibit harm. This demonstrates reasoning models' partial decoupling from prompt-conditioning and their vulnerability to adversarial steering.

Machine Ethics and Refusal Calibration

On both short and long-form ETHICS evaluations, SFT models suffer the most severe calibration collapse, with nearly all benign prompts refused and accuracy reduced to near random or lower. Distillation and RL/GRPO pathways exhibit improved accuracy but at the expense of either over-refusal on benign prompts or inadequate refusal when adversarial jailbreaks are present.

Figure 5: Machine ethics evaluation on short question (zero-shot) prompts: SFT models exhibit drastic increases in refusal rate with corresponding drops in classification accuracy.

OOD Robustness

SFT-trained reasoning models shift towards a refuse-by-default policy, producing near-zero accuracy on both in-distribution and OOD questions. Other pathways err in the opposite direction, under-refusing on clear post-cutoff data, which may cause fabricated or misleading outputs outside a model’s knowledge boundaries.

Figure 6: OOD question answering: SFT models collapse into blanket refusal regardless of prompt context, indicating overcautious and uncalibrated abstention.

Privacy Leakage

Zero-shot and adversarially prompted privacy evaluation confirms universal, catastrophic leakage rates for both baseline and reasoning models unless explicit protective prompting is provided. Even then, SFT and RL/GRPO models fail to respect PII protection demonstrations (84–91% leak rate for critical PII), establishing that reasoning post-training substantially compromises learnability and retention of privacy-protective norms.

Figure 7: Even with explicit refusal demonstrations, SFT and GRPO-trained models continue to leak sensitive PII at high rates, indicating structural alignment failures.

Behavioral Drift and Distributional Shifts

The KL divergence between instruction-tuned and reasoning-enhanced models is quantified across all trust-critical prompt families. SFT and RL/GRPO training induce orders-of-magnitude increases (e.g., KL > 400 on some axes for SFT), correlating directly with observed alignment regressions. The chain rule decomposition implicates reasoning trace generation as a principal driver of distributional drift, suggesting that even small per-token differences in intermediate traces compound into large sequence-level divergences.

Figure 8: KL divergence measurement across pathways reveals substantial behavioral drift for SFT/GRPO compared to instruction tuning, mapping directly onto trustworthiness degradation.

Broader Implications and Theoretical Considerations

The findings decisively demonstrate that post-training for improved reasoning is an inherently alignment-sensitive operation. Capabilities and trustworthiness trade off: utility-oriented optimization without explicit trust-aware objectives will not reliably preserve safety, bias mitigation, PII protection, or calibrated refusal. Model developers and safety researchers cannot infer deployment reliability from improved reasoning accuracy alone.

Reporting only upstream capability metrics (e.g., math benchmark scores) in reasoning model releases risks masking or exacerbating harm risks unless accompanied by comprehensive trustworthiness and drift diagnostics. Furthermore, the partial, pathway-dependent preservation of alignment signals in distillation approaches underlines the importance of carefully chosen, trust-sensitive teacher signals.

Figure 9: On long-form machine ethics, reasoning models (especially SFT) show catastrophic over-refusal, indicating severe miscalibration and erosion of useful ethical reasoning under benign prompts.

Figure 10: PII leak rates are universally high across baseline and reasoning models absent explicit protected-mode prompting, establishing pervasive structural weaknesses.

Figure 11: Under 3-shot protective demonstrations, reasoning models (especially SFT) exhibit catastrophic PII leakage, contrasting with effective protection in baseline instruct models.

Figure 12: Distillation-trained models show greater privacy protection under adversarial demonstrations than both baseline and other reasoning pathways, highlighting some robustness to adversarial prompting.

Conclusion

Post-training to induce reasoning in LLMs triggers a broad class of alignment regressions, including but not limited to increased toxicity, stereotyping, miscalibrated refusal, OOD hallucination, and privacy compromise. These shifts are robust to the specific reasoning induction pipeline and scale with KL divergence from the post-training distribution. Therefore, trustworthiness audits and drift analysis must be incorporated as a matter of practice alongside reasoning capability reporting.

Future research should focus on designing trust-aware auxiliary losses, targeted regularization, or new paradigm-specific trust calibration methods, with the objective of decoupling capabilities gains from erosion of deployment-critical alignment. Robust reasoning models must be benchmarked—and optimized—not only for correctness, but also for reliable, controlled, and refuse-when-appropriate behavior, bias minimization, and privacy protection. This work foregrounds trustworthiness metrics as first-class citizens in the evaluation and deployment of large reasoning models, mandating their routine reporting and optimization.