Has This Checkpoint Been Abliterated? A Two-Signal Audit and Its Failure Map

Abstract: Can a platform tell, before deployment, whether an open-weight checkpoint has had its refusal mechanism stripped? Runtime guards cannot: they score generations, not the artifact. We combine two cheap internal signals, a reference-anchored activation refusal-gap and a weight-recovery energy of the base-to-candidate weight difference, into a threshold-free checkpoint audit. The two are negatively correlated and label-complementary: the gap supplies refusal-specificity and the weight energy supplies recall. On a 273-checkpoint registry spanning Qwen, DeepSeek-distilled Qwen, Llama, and Gemma, their z-sum separates 57 public abliterations from 37 benign fine-tunes, merges, and instruction-tunes at AUROC 0.95, significantly above either signal alone (0.84, 0.90), and a Youden-calibrated threshold transfers to held-out families at balanced accuracy 0.89 (FPR 0.11), missing only 4 of 57. We then map two failures, in order of severity: a spoofed reference evades both axes with no training (ΔW=0, \r{ho}=1 by construction), and a white-box owner trains a checkpoint past the threshold while it stays guard-unsafe and coherent. The audit is effective triage, not tamper-proofing: it presumes an attested reference, and its claims are bounded by the registry we evaluate it on.

• The paper introduces a checkpoint audit framework using an activation refusal-gap and weight-recovery energy metric to detect removals of safety mechanisms.
• Empirical evaluations demonstrate a combined AUROC of 0.95 and balanced accuracy of 0.89 across multiple LLM families, highlighting the method’s efficacy.
• The study identifies key limitations such as reference spoofing and white-box adversarial evasion, emphasizing the need for more robust attestation methods.

Two-Signal Audit for Checkpoint Abliteration: Methods, Efficacy, and Limits

Problem Setting and Motivation

Platforms and regulators increasingly face the challenge of determining whether open-weight LLM checkpoints distributed across platforms have been intentionally stripped of their refusal mechanisms (termed "abliterated") prior to deployment. Existing runtime guards operate on model outputs, not model artifacts, and are agnostic to weight-level manipulations (2607.01854). This paper introduces a checkpoint-level audit framework combining two internal signals: a reference-anchored activation refusal-gap and a weight-recovery energy metric. The practical goal is to develop a low-cost, threshold-free, and deployable screening mechanism to efficiently triage potentially abliterated checkpoints, while also analyzing its points of failure and inherent limitations.

Methods: Complementary Detection Axes

The audit harnesses two distinct but complementary signals extracted given a candidate checkpoint $M_c$ and a trusted attested reference $M_b$ (typically the originating base model or an unmodified sibling):

• Activation Refusal-Gap ($\rho$): For each model layer in a central "band," a difference-of-means direction $\hat r_\ell$ is extracted from the activations generated by a contrast set of harmful versus benign prompts. The core metric, the refusal-gap, is given by the ratio of the mean projection gap for $M_c$ relative to $M_b$. Values near 1 denote intact refusal, while values closer to 0 indicate removal of refusal. This axis is highly specific to refusal edits, but fails when the target direction is non-canonical.
• Weight-Recovery Energy ($E_1$): For a set of attention-output (o_proj) and MLP-down projection (down_proj) weight matrices, the rank-1 spectral energy fraction of the weight difference $\Delta W = W_b - W_c$ is computed and averaged. This measures whether a substantial, low-rank weight edit—characteristic of safety removal or reinforcement—occurred. $E_1$ is sensitive to any low-rank edits, including benign ones, yielding broader recall but lower specificity than $\rho$.
• Combined $M_b$0-Sum Detector: The two scores are negatively correlated across the dataset (Pearson $M_b$1), and standardization followed by summation creates a unified score: $M_b$2. This composite, threshold-free statistic leverages the refusal-specificity of $M_b$3 and the recall of $M_b$4 for effective separation.

  Figure 1: The two-signal decision boundary in the activation gap/weight energy plane, showing strong separation between abliterated and benign checkpoints.

Evaluation: Auditing on a Large Multi-Family Registry

The proposed audit was empirically evaluated on a registry of 273 checkpoints sampled across Qwen, DeepSeek-distilled Qwen, Llama, and Gemma, including both public abliterations and benign (instruction) fine-tunes, merges, and instruction-tunes as hard negatives. Behavioral ground truth labels were obtained using Qwen3Guard, stripped of chain-of-thought traces.

Main Results:

• In-Sample Performance: On the 94 checkpoints (57 abliterated, 37 benign) processed in full, the combined $M_b$5-sum achieves AUROC 0.95 (95% CI [0.90, 0.98]), which is significantly higher than either $M_b$6 (AUROC 0.84) or $M_b$7 (AUROC 0.90) alone. At the selected Youden threshold, the true positive rate (TPR) is 0.93, with a false positive rate (FPR) of 0.14.
• Cross-Family Transfer: In leave-one-family-out evaluation, balanced accuracy is 0.89 (FPR 0.11), outperforming the individual axes and an absolute (non-standardized) combination.

  Figure 2: ROC curves for the activation gap, weight energy, and combined detector, highlighting the gain from complementary signals.

• The fusion approach correctly flags 53/57 abliterated checkpoints, with the remaining 4 indistinguishable from benign edits and not adversarially hard to detect.

The complementarity is borne out in both separability and transfer across model lineages and recipes. The weight axis ($M_b$8) recovers cases where multi-directional abliteration defeats the single-direction refusal gap, while the gap protects against benign edits mis-flagged by $M_b$9.

Failure Modes and Adversarial Evasion

Two classes of failures are explicitly mapped:

• Reference Spoofing: Both metrics presuppose an honest, attested reference base. If an adversary nominates an already-abliterated model (or the candidate itself) as the reference, both axes are evaded by construction. This is an architectural limitation rooted in the chain of trust for the base model and cannot be circumvented by any internal metric.
• White-Box Adversarial Training: A targeted adversary with access to the detector and gradients can fine-tune the checkpoint to explicitly minimize both $\rho$0 and $\rho$1 while retaining harmful, non-refusing generation. In a Qwen2.5-1.5B experiment, checkpoint scores cross the detection threshold with negligible impact on general capabilities (MMLU 0.58 vs. 0.59, GSM8K 0.51 vs. 0.57 versus the base), and the model stays uncensored and coherent across 20 harmful prompts, confirming the detector's vulnerability to detection-aware evasion.

  Figure 3: Evolution of the detector score during adversarial training; the checkpoint traverses from detectable to undetectable regimes while retaining dangerous behavior.

Theoretical and Practical Implications

The audit marks a practical advance in checkpoint-level provenance analysis. The fusion of refusal-specific and direction-agnostic metrics offers a low-cost first pass, elevating the balance between TPR and FPR for both in-domain and out-of-domain checkpoint variants, demonstrating operational utility for model registries and deployment platforms.

However, the audit is explicitly not tamper-proof. Its efficacy is fundamentally limited by both the verifiability of the reference base and the inability of internal features to resist adaptive attackers with gradient access. White-box evasion, demonstrated here in a single-family setting, signals the lack of robust, model-internal "tamper-evidence" for refusal removal.

A further implication is that as adversaries are incentivized to evade known detectors, the space of abliteration will become increasingly adversarial, pushing future research toward more robust, possibly cryptographically anchored attestation and forensics, or towards hybrid approaches combining model-internal and behavioral methods.

Future Directions

• Robust Attestation: Future developments will need to harden the chain of trust for reference bases, integrating publisher-signed hashes or secure, hardware-based provenance tracking.
• Beyond Single-Direction Gaps: Exploration of multi-directional or subspace-based refusal metrics may close some gaps but will not solve evasion by adaptive white-box adversaries.
• Adversarial Training Countermeasures: Defensive research may include dynamic or stochastic probing, adversarially-aware audits, or integrating external behavioral signals.
• Label Challenges: Extensions to other harm axes (harmfulness, not just refusal) and to transferability across even more diverse architectures remain open.

Conclusion

This paper presents a formally evaluated, highly separable two-signal checkpoint audit for detecting removal of refusal mechanisms in LLMs. While the combined $\rho$2-sum of activation gap and weight-recovery energy offers substantial practical triage capability (AUROC 0.95, balanced accuracy 0.89), the analysis rigorously charts its operational envelope and points of failure. The audit's effectiveness is predicated on reference attestation, and cannot provide robust guarantees against white-box, detection-aware adversaries. The results advocate interpreting these audits as pre-deployment risk triage, not certification. The work provides a clear roadmap for both immediate practical deployment and the research agenda for checkpoint provenance and tamper-evidence going forward.

---

Reference: "Has This Checkpoint Been Abliterated? A Two-Signal Audit and Its Failure Map" (2607.01854)