Fast Multi-dimensional Refusal Subspaces via RFM-AGOP

Abstract: Steering and monitoring activations in LLMs are increasingly used for both safety and interpretability. Early work assumed behaviours are encoded along single linear directions, but recent findings suggest complex behaviours, such as the refusal to answer harmful queries, live in multi-dimensional subspaces. However, existing methods for extracting these subspaces are computationally expensive, which becomes prohibitive on reasoning models who produce long reasoning traces. By adapting the Recursive Feature Machine (RFM) algorithm -- which can be computed efficiently -- with a probe-informed initialization, we are able to identify the multi-dimensional refusal subspace in seconds, on reasoning (Qwen 3) and non-reasoning (Qwen 2.5) models. While RFM allows for faster subspace identification, it also showed better performances on the ablation task than its alternatives. More work is planned to better understand the relations between subspaces found by different methods. If confirmed, RFM could be a cheap and scalable complement to existing subspace-extraction methods in LLMs.

• The paper introduces a novel RFM-AGOP method that efficiently extracts multi-dimensional refusal subspaces from LLMs, overcoming limitations of single-vector approaches.
• It leverages recursive feature machine updates, probe-based initialization, and exponential moving averages to stabilize kernel matrix learning and ensure computational tractability.
• Experimental results demonstrate that multi-directional ablation enhances harmful query suppression in large models without degrading overall performance.

Fast Multi-dimensional Refusal Subspaces via RFM-AGOP: Methodology and Implications

Overview

"Fast Multi-dimensional Refusal Subspaces via RFM-AGOP" (2607.02396) addresses the challenge of rapidly and efficiently identifying multi-dimensional refusal subspaces in LLMs, particularly in the context of reasoning models with extensive generation traces. While prior work often assumed refusal is encoded along a single vector, this paper builds on recent geometric findings showing that refusal is generally a higher-dimensional phenomenon, especially in larger models. The central contribution is the adaptation of the Recursive Feature Machine (RFM) algorithm—augmented by Average Gradient Outer Product (AGOP) and a probe-based initialization—to efficiently extract multi-dimensional refusal subspaces with competitive accuracy and robustness, while remaining computationally tractable.

Problem Formulation and Preliminaries

The authors focus on the subspace encoding of refusal behavior in LLMs, formalized as follows: Given a transformer model, they extract the last-token residual activations for a dataset of harmful (D⁺) and harmless (Dˉ) prompts. The objective is to identify a $k$-dimensional orthonormal subspace $V = \{v_1, \ldots, v_k\}$ in activation space, such that projection onto the orthogonal complement of $V$ suppresses refusal on harmful queries while preserving response quality on benign queries.

The extraction process requires supervised contrastive data to ensure the identified subspace is specific to the refusal mechanism, as opposed to generic or confounding features. The method should also minimize hyperparameters and automatically infer subspace dimension, ensuring scalability and practicality.

Methodological Innovations: RFM-AGOP Framework

The paradigm shift from one-dimensional to multi-dimensional refusal requires algorithms capable of efficiently uncovering subspaces that cannot be captured by simple linear probes or difference-in-means (DIM) methods. The RFM-AGOP approach introduces several innovations:

• Recursive Feature Machine Adaptation: At each iteration, RFM performs kernel ridge regression based on a Mahalanobis Laplace kernel, learning both the predictor and a dynamic kernel matrix $M$. The iterative update employs AGOP, which accumulates gradient information to refine the kernel in directions that best separate harmful from harmless activations.
• Probe-informed Initialization: To address instability and suboptimal convergence with a naive identity-matrix initialization, the authors initialize $M_0$ by combining the principal direction of a trained linear probe with a rank-truncated empirical covariance of the activations. This hybrid warm start enhances convergence in practice, especially in higher dimensions.
• Exponential Moving Average of the Kernel: To further stabilize updates and dampen oscillations during training, the method incorporates an EMA on the kernel matrix, yielding smoother and more reliable convergence in empirical studies.
• Eigendecomposition for Subspace Extraction: After convergence, the top-$k$ eigenvectors of the learned kernel matrix define the refusal subspace, with signs oriented according to Pearson correlation with refusal labels.

Edits to model activations or weights are performed by projecting onto the complement of the identified subspace. Critically, the method distinguishes itself by recovering multiple meaningful directions simultaneously and demonstrating that these are necessary for adequately suppressing refusal in larger, more capable models.

Experimental Results and Empirical Analysis

Suppression Efficacy

Extensive experiments on the Qwen 3 family (1.7B, 4B, 8B, 14B) and non-reasoning Qwen 2.5 7B reveal that:

• Single-direction ablation is insufficient for large models: Whereas smaller models can be jailbreaked by ablating a single refusal direction, larger models ($\geq$8B) require ablation along at least three directions to exceed a 50% Attack Success Rate (ASR) on harmful queries.
• Performance saturates as directions increase: There is diminishing return after a certain subspace dimension, supporting that model scale correlates with refusal subspace complexity.
• Robustness across methods: RFM-AGOP consistently outperforms or matches RCO and Clusters, while being much faster; none of the baselines make single-direction interventions effective in large models.

Retention of Useful Capabilities

Evaluation on MMLU shows that ablation of refusal directions—when limited to a reasonable number—does not generally degrade general model performance, except a notable unexplained drop in the 14B model. This result indicates that refusal is at least partly disentangled from core reasoning circuits in these architectures.

Algorithmic Efficiency

One of the primary motivations for RFM-AGOP is efficiency:

• RFM-AGOP requires only a single forward pass and matrix eigendecomposition, making it tractable on commodity hardware (e.g., seconds on a consumer RTX 4090 laptop).
• RCO and similar optimization-based approaches require numerous full forward and backward passes, resulting in orders of magnitude higher computational cost (estimated $>10^{18}$ FLOPs), rendering them inefficient for large-scale or quick-turnaround analyses.

Ablation and Steerability Analysis

A detailed ablation study demonstrates that ablation along random directions does not induce refusal suppression, confirming that the identified subspace is non-trivial and task-specific. Additionally, steering experiments (adding subspace directions to activations instead of removing) show that only the principal direction reliably induces refusal; higher-order directions have weaker or inconsistent effects. This asymmetry highlights subtleties in subspace geometry and may reflect nonlinearity or redundancy in refusal encoding.

Limitations

The paper critically assesses key limitations:

• Reliability of refusal evaluation metrics remains problematic, with LLM-based judges being sensitive to prompt phrasing. The lack of a standardized, robust measurement protocol hinders reproducibility and inter-paper comparison.
• Hyperparameter sensitivity: Though functional, the current approach to kernel bandwidth and iteration selection is simplistic, and improved tuning schemes could yield better outcomes.
• Interpretability and universality: The relationship between the RFM-discovered subspace and those uncovered by unsupervised methods (e.g., sparse autoencoders) remains unresolved. Scope is limited by focus on the Qwen model family.

Theoretical and Practical Implications

These results support the hypothesis that safety-critical behaviors, such as refusal, are geometrically complex in large LLMs, reflecting scaling trends in feature disentanglement and redundancy. This comports with recent observations that harmfulness detection and refusal are encoded in distinct representational axes. Methodologically, the high efficiency and effectiveness of RFM-AGOP open the door to:

• Scalable interpretability interventions: RFM-AGOP can be incorporated into continuous monitoring and real-time steering workflows for deployed systems.
• Foundation for further scaling law discovery: Systematic sweeps across model families and sizes may reveal fundamental relationships between model capacity, behavioral geometry, and alignment strategies.
• Baseline for cross-method comparisons: Its speed and accuracy provide a strong baseline for future work on extracting, steering, and evaluating interpretability subspaces.

Future Directions

Outstanding research avenues highlighted include:

• Evaluating cross-method consistency: Comparing RFM-AGOP with SAE- or cluster-based subspaces to understand shared vs. model-specific features.
• Tracking internal geometry over reasoning traces: Analyzing how refusal subspaces evolve temporally, especially across long context windows in reasoning models.
• Standardizing refusal measurement: Community-wide effort to develop transparent, reproducible, and adversarially robust refusal metrics.

Conclusion

"Fast Multi-dimensional Refusal Subspaces via RFM-AGOP" establishes a computationally efficient, robust framework for extracting refusal-relevant subspaces in LLMs. Its findings—most notably, the necessity of multi-dimensional interventions in larger models and the practical feasibility of rapid subspace extraction—provide strong foundations for safety-aligned behavioral control and mechanistic analysis. Broader adoption and extension of these techniques will likely prove instrumental in both theoretical alignment research and operational LLM safety systems.