Don't Go Breaking My LLM: The Impact of Pruning Attention Layers on Explanation Faithfulness and Confidence Calibration

Abstract: Pruning LLMs reduces memory and inference costs by removing parts of the network, producing smaller models that retain most of their accuracy. As attention layers are the most resource-intensive parts of LLMs, pruning them is a promising compression strategy. Prior work shows that up to 33% of attention layers can be pruned with minimal accuracy loss. Nevertheless, the impact of attention pruning on model interpretability, specifically faithfulness and confidence calibration, remains unstudied. To address this gap, we study how pruning attention layers affects explanation faithfulness and confidence calibration across five LLMs and eight datasets. While the pruned models often maintain high accuracy, we find that their faithfulness and calibration often degrade. Notably, faithfulness and calibration can fluctuate significantly, even when accuracy remains stable, highlighting a misalignment between model confidence, interpretability, and accuracy. Our findings suggest that layer pruning can affect LLMs' interpretability and reliability in ways not captured by accuracy and efficiency measures alone. We recommend including explainability and calibration metrics when evaluating pruned models.

• The paper shows that pruning attention layers reduces explanation faithfulness and degrades calibration, even when accuracy losses are minimal.
• It employs methods like LIME, Kernel SHAP, and metrics such as AOPC and ECE on multiple LLMs and NLP benchmarks to quantify these effects.
• The study highlights a misalignment between maintained accuracy and reduced trustworthiness, emphasizing the need for multi-objective optimization in pruning.

The Impact of Attention Layer Pruning on Explanation Faithfulness and Confidence Calibration in LLMs

Introduction

The efficiency-accuracy trade-off in LLMs has encouraged widespread adoption of pruning strategies to reduce inference costs. Among these, structured pruning targeting entire attention layers (layer-drop) has emerged as particularly attractive, allowing the removal of up to one-third of layers while minimally affecting predictive accuracy. However, the consequences of such structured sparsification on model interpretability and the trustworthiness of model confidence have remained largely unexamined. This work delivers a systematic and comprehensive analysis of how attention layer pruning alters explanation faithfulness and confidence calibration, thereby exposing critical misalignments between accuracy, interpretability, and reliability in contemporary compressed LLMs (2606.24970).

Methodology

The analysis is conducted on five open-access decoder-based LLMs (Mistral 7B, Llama-2 7B, Llama-3 8B, Qwen-3 4B, Qwen-3 8B) evaluated across eight established NLP benchmarks, including various QA, sentiment analysis, and NLI tasks. Attention layer importance is quantified via input-output cosine similarity, with pruning executed in order of least saliency as per He et al. (2026). Feature attributions are assessed using LIME and Kernel SHAP, and faithfulness is measured by comprehensiveness and sufficiency (AOPC-based metrics). Confidence calibration is quantified using Expected Calibration Error (ECE), and plausibility is appraised via human-labeled rationales (F1, precision, recall, IoU, AUPRC).

Results

Accuracy Preservation under Pruning

For all models except Qwen, up to 12 attention layers (≈1/3) can be removed with negligible or modest accuracy drops across tasks. Notably:

• The relative reduction in accuracy remains mostly below 0.13 for Mistral 7B and Llama-3 up to 12 layers removed.
• QA and BoolQ tasks exhibit monotonically degrading accuracy with increasing pruning, while sentiment/NLI tasks present non-monotonic accuracy fluctuations, correlating with label token length.
• Qwen-3 4B/8B show greater variability across tasks, with several statistically significant drops for aggressive pruning.

Degradation and Instability of Explanation Faithfulness

Pruning attention layers consistently reduces explanation faithfulness, as indicated by declines in comprehensiveness and sufficiency. Key observations:

• The overlap of the most salient input features (top-k tokens by attribution) between pruned and unpruned models decreases steadily with increased pruning, indicating altered model reasoning footprints.
• Comprehensiveness (explanation relevance) drops are particularly pronounced and often statistically significant for non-QA datasets; sufficiency is less stable but generally also declines.
• Faithfulness fluctuations with pruning do not mirror accuracy changes: scores exhibit substantial non-monotonic shifts (especially on short-label tasks), indicating a dissociation between prediction accuracy and explanation quality.
• The degradation of average model confidence, contingent on internal logit shifts post-pruning, exacerbates the reduction in explanation faithfulness.

Misalignment and Instability in Confidence Calibration

Confidence calibration is disproportionately affected by pruning, yielding increased ECE in most settings:

• ECE rises monotonically with greater pruning for most models on most tasks—calibration misalignment is sometimes severe even when accuracy remains nearly constant.
• The correlation between changes in accuracy and calibration error is weak or absent; ECE often oscillates due to logit-induced instability rather than performance drops.
• Qwen models sometimes display improved ECE post-pruning on sentiment analyses due to concurrent declines in both confidence and accuracy, artificially reducing bin-wise calibration gaps.

Plausibility Resilience

In contrast to faithfulness, the plausibility of explanations (alignment with human rationales) is largely preserved during pruning. F1, precision, recall, and IoU measures on human-annotated rationales decline only mildly, with the top-K attribution overlap robust against moderate sparsification. This suggests pruning perturbs true causal reasoning more than surface-level attribution saliency, thus potentially misleading downstream trust when only plausibility is reported.

Key Claims and Numerical Findings

• Faithfulness and calibration degrade under structured pruning even when accuracy is preserved.
• Faithfulness and calibration metrics fluctuate non-monotonically and disproportionately to accuracy, especially on tasks with short labels.
• Plausibility (by human annotation) does not appreciably decline, exposing a disconnect between human-aligned and model-aligned explanations.
• The deterioration in faithfulness and calibration is statistically significant and holds for all models across most task/dataset pairs, particularly after 8+ layers are pruned.

Implications and Future Directions

The work demonstrates that standard sparsification regimes for LLM efficiency optimization cannot guarantee the retention of crucial interpretability and reliability characteristics. As internal reasoning and output calibration are destabilized by attention layer removal, downstream deployments relying on faithfulness and calibrated confidence (security-critical, legal, and medical domains) are exposed to latent risks that are not detectable via accuracy/efficiency alone.

Practical implications:

• Accuracy is an insufficient proxy for trustworthiness in pruned models. Deployment should always monitor faithfulness and calibration as explicit objectives.
• Model selection and credentialing for high-stakes settings require multi-objective optimization: accuracy, faithfulness, calibration, and plausibility must be balanced.
• Layer importance scoring should integrate interpretability and calibration signals, and iterative pruning with rollback when metrics degrade is recommended.
• Lightweight post-hoc fine-tuning (e.g., LoRA adapters) could be explored for local recovery of calibration and faithfulness post-pruning.

Theoretically, the findings raise further research questions:

• What are optimal pruning strategies that jointly maximize efficiency and interpretability/calibration?
• How do alternative pruning schemas (unstructured, semi-structured, head-wise) compare on interpretability axes?
• What tasks or prompting formats are most/least robust to the interpretability-versus-efficiency trade-off?
• To what extent do these phenomena generalize beyond decoder-based LLMs to other architectures?

Conclusion

This work systematically documents the non-obvious and often severe detrimental effects of attention layer pruning on explanation faithfulness and confidence calibration in LLMs (2606.24970). These degradations are largely invisible from accuracy and plausibility measurements alone, indicating that current evaluation protocols for pruned LLMs are insufficient. Attention layer pruning strategies must be re-evaluated for interpretability and reliability requirements, and future research should explicitly target multi-objective pruning and post-hoc correction to ensure trustworthy model deployment.