Contextual Linear Activation Steering of Language Models

Abstract: Linear activation steering is a powerful approach for eliciting the capabilities of LLMs and specializing their behavior using limited labeled data. While effective, existing methods often apply a fixed steering strength to all tokens, resulting in inconsistent steering quality across diverse input prompts. In this work, we introduce Contextual Linear Activation Steering (CLAS), a method that dynamically adapts linear activation steering to context-dependent steering strengths. Across eleven steering benchmarks and four model families, it consistently outperforms standard linear activation steering and matches or exceeds the performance of ReFT and LoRA in settings with limited labeled data. We therefore propose CLAS as a scalable, interpretable, and accurate method for specializing and steering LLMs.

• The paper presents CLAS, which improves upon fixed-coefficient methods by dynamically modulating steering strength via context-aware sensing vectors.
• It employs a Recursive Feature Machine probe and AGOP-driven kernel ridge regression to extract steering directions for specialized neural activation control.
• Experimental results show CLAS delivers superior accuracy (up to 99%), efficiency, and minimal interference compared to LAS, ReFT, and LoRA.

Contextual Linear Activation Steering of LLMs: Technical Summary

Background and Motivation

Linear direction-based intervention in neural activations has become a central method for behavior specialization and interpretability in LLMs. Prior frameworks, notably linear activation steering (LAS), leverage probe-extracted directions to modulate hidden states during inference, inducing or suppressing concept-specific behaviors such as reasoning or sentiment control. However, LAS employs a fixed steering coefficient across all contexts and tokens. This universality leads to inconsistent steering quality, necessitating costly grid searches for optimal coefficient selection, and ultimately impedes robustness and deployment.

The paper "Contextual Linear Activation Steering of LLMs" (2604.24693) advances this paradigm by introducing Contextual Linear Activation Steering (CLAS). CLAS dynamically adjusts steering strength as a function of both token context and model block, harnessing learnable sensing vectors to maximize control flexibility and preserve the interpretability of probe-based directions.

Methodology

CLAS operates as follows: for each transformer block, a steering direction is extracted by training a Recursive Feature Machine (RFM) probe on the block's residual stream activations, maximizing binary task label prediction. RFM utilizes AGOP-driven kernel ridge regression to identify a principal direction (the probe eigenvector) in $\mathbb{R}^k$ that linearly separates the targeted concept.

During inference (or specialized training), rather than the fixed additive perturbation $$\mathbf{h}_{l,t} = \mathbf{h}_{l,t} + \alpha \mathbf{d}_l$$ (as in LAS), CLAS modulates the steering contribution via a learned sensing vector $\mathbf{c}_l \in \mathbb{R}^{k+1}$, computing a context-dependent coefficient:

$$\mathbf{h}_{l,t} = \mathbf{h}_{l,t} + (\mathbf{c}_l \cdot [\mathbf{h}_{l,t}, 1]) \mathbf{d}_l$$

Here, $[\mathbf{h}_{l,t}, 1]$ denotes the concatenation of the activation and a bias term. The sensing vectors $\{\mathbf{c}_l\}$ are trained via next-token prediction loss on prompt-completion pairs, with all LLM weights frozen.

Steering vectors are decoupled from coefficient optimization: the probe directions are fixed, context-dependent strengths are learned. This decoupling provides robust, interpretable specialization with limited supervision.

Experimental Protocol

CLAS, LAS, ReFT, and LoRA were benchmarked across eleven steering tasks (e.g., anti-refusal, sentiment shift, step-by-step reasoning, TLDR summarization, multilingual translation) on four instruction-tuned models (Qwen2.5-7B, Llama-3.1-70B/8B, Llama-3.2-1B). Evaluation metrics include task success rate, steering interference (effect of single-task steering on remaining tasks), tuning time, and concept monitoring accuracy using linear probes.

Probe datasets consisted of prompt-concept label pairs; steer datasets paired non-task prompts with target completions, either sourced from reference data or auto-generated.

Empirical Results

Steering Accuracy:

CLAS outperforms LAS (fixed coefficient steering) on nearly every task and model, yielding higher average accuracy across ten tasks: e.g., 86.3% for Qwen2.5-7B vs. 64.0% for LAS, and 88.4% for Llama-3.1-70B vs. 75.1% for LAS. It matches or exceeds parameter-efficient rank-1 adapters (ReFT, LoRA) in limited supervision settings.

Coefficient Tuning Efficiency:

CLAS tuning (AdamW, next-token prediction) completes in minutes per task, considerably faster than LAS grid search, which often requires hours, especially for larger models.

Steering Robustness and Interference:

When steered toward a specific target, CLAS exhibits minimal interference with non-target tasks, limiting performance degradation. LoRA and ReFT frequently encounter severe drops under narrow task specialization (mean accuracy drop up to –57% in some settings). CLAS maintains high robustness, with minimal average interference ($\Delta$ accuracy < –6%).

Concept Monitoring:

Probe directions extracted via RFM (used in CLAS) yield superior concept detection accuracy (mean per-block accuracies exceeding 96% on most models and tasks), significantly outperforming directions derived from LoRA and ReFT (mean accuracies ~70%). The best-block accuracy for probe directions approaches 99% or higher, indicating high linear separability and interpretability.

Direction Orthogonality:

Cosine similarity analysis reveals that CLAS probe directions are nearly orthogonal to LoRA/ReFT fine-tuned directions, despite similar prompted behaviors, confirming that CLAS preserves mechanistic separation between concept detection and task specialization.

Rank Ablation:

Increasing LoRA and ReFT rank does not consistently improve steering or robustness, underscoring the advantage of CLAS’s low-rank context adaptation.

Theoretical and Practical Implications

CLAS substantiates the linear representation hypothesis by demonstrating that context-modulated linear directions can both reliably specialize behavior and provide monitoring signals for interpretable model operation. By decoupling direction extraction (probing) from coefficient learning (context adaptation), CLAS enables high-performance task control with minimal labeled data and lightweight computation.

Practically, CLAS supports scalable, interpretable deployment scenarios – dynamic behavior specialization without costly retraining or manual coefficient selection. Its application is extensible to broader architectures and modalities beyond LLMs, facilitating generalizable control of deep neural activations.

Theoretical implications include a more unified understanding of latent semantic encoding—linear directions are not only sufficient for concept detection but also, when contextually modulated, for precise, on-the-fly intervention. This bridges the gap between mechanistic transparency and practical fine-tuning.

Future Directions

Open research avenues include combinatorial steering for multi-concept behaviors, hierarchical sensing vector architectures for finer control granularity, exploration of alternative probe methods (nonlinear, causal), and extension to high-rank or multi-modal settings. CLAS may serve as a foundation for mechanistically transparent LLM deployment, interpretable safety interventions, and dynamic domain adaptation.

Conclusion

CLAS represents a significant technical step in LLM steering by leveraging context-dependent vector modulation of probe-extracted directions. It delivers improved accuracy, efficiency, and robustness over fixed-coefficient steering and parameter-efficient fine-tuning methods, while preserving interpretability as demonstrated in concept monitoring. These findings reinforce the viability of scalable, transparent activation engineering for model specialization and monitoring, with implications for deployment, control, and safety in advanced AI systems.