Toward universal steering and monitoring of AI models

Abstract: Modern AI models contain much of human knowledge, yet understanding of their internal representation of this knowledge remains elusive. Characterizing the structure and properties of this representation will lead to improvements in model capabilities and development of effective safeguards. Building on recent advances in feature learning, we develop an effective, scalable approach for extracting linear representations of general concepts in large-scale AI models (LLMs, vision-LLMs, and reasoning models). We show how these representations enable model steering, through which we expose vulnerabilities, mitigate misaligned behaviors, and improve model capabilities. Additionally, we demonstrate that concept representations are remarkably transferable across human languages and combinable to enable multi-concept steering. Through quantitative analysis across hundreds of concepts, we find that newer, larger models are more steerable and steering can improve model capabilities beyond standard prompting. We show how concept representations are effective for monitoring misaligned content (hallucinations, toxic content). We demonstrate that predictive models built using concept representations are more accurate for monitoring misaligned content than using models that judge outputs directly. Together, our results illustrate the power of using internal representations to map the knowledge in AI models, advance AI safety, and improve model capabilities.

• The paper presents a novel steerable framework that aggregates nonlinear features from multiple LLM layers for precise output modulation.
• The methodology employs Recursive Feature Machines (RFMs) to detect semantic vectors, outperforming benchmarks like GPT-4o in identifying hallucinations and harmfulness.
• The approach enables efficient control over models, allowing simultaneous steering of diverse concepts such as style, harmfulness, and sentiment.

Toward Universal Steering and Monitoring of AI Models

Introduction

LLMs are repositories of vast human knowledge; however, they often exhibit limitations in articulating their inherent knowledge comprehensively. These models can sometimes produce misleading outputs due to their inability to ascertain the accuracy of their own responses. This paper presents a novel methodology for detecting semantic concepts within the internal activations of LLMs. More importantly, the proposed framework not only identifies but also steers the models toward desirable outputs by employing nonlinear feature learning methods and aggregating features across multiple layers.

Figure 1: Methodology for detecting and steering concepts in LLMs by aggregating layer-wise predictors. Examples of steered generations toward concepts include harmfulness, Shakespearean/Poetic English, and dishonesty.

Methodology for Detection and Steering

The core innovation of this research lies in identifying influential linear directions using nonlinear methods across different layers of LLMs. Recursive Feature Machines (RFMs) are harnessed to pinpoint concept-specific vectors by analyzing activations through nonlinear transformations. Subsequently, these vectors are aggregated to enhance the accuracy of concept detection and facilitate precise steering toward specific outputs.

The approach involves two key components:

1. Detection: Employing RFMs to discern semantic concepts by projecting activations onto feature vectors derived from trained models. This technique has demonstrated superior performance in identifying hallucinations, harmfulness, and untruthfulness across multiple benchmarks.
2. Steering: Utilizing learned feature vectors to manipulate LLM activations, thus directing the model output toward a range of concepts encompassing human languages, programming languages, and thematic styles such as poetic or Shakespearean English.

Detection Results

The efficacy of this method in detecting a broad array of semantic concepts was evaluated using seven benchmarks, including HaluEval, AgentHarm, and TruthGen. The results manifest that aggregation with RFMs consistently outperforms previous detection methods and even the advanced GPT-4o model in several tasks.

Figure 2: Steering programming and human languages showcasing superior translation capabilities.

Steering Capabilities

Using concept vectors extracted from RFMs, the steering technique exhibited proficiency in manipulating model outputs to generate desired linguistic styles and content types. An important aspect of this research is its ability to steer multiple concepts simultaneously. For instance, merging vectors of harmfulness and poetic style results in generations that are both aesthetically distinct and conceptually aligned with targeted themes.

Figure 3: Steering instruction-tuned Llama-3.1-8B to interpret names as different identities: comparison between RFM, PCA, and Linear Regression.

Implications and Future Prospects

This paper presents a transformative approach to LLM steering, with implications for enhancing model transparency and deploying LLMs effectively across diverse applications. By providing a universal and efficient steering mechanism, these techniques offer a promising alternative to conventional fine-tuning methods, potentially revolutionizing the modulation of AI behaviors without extensive retraining.

Further research may explore refining the steering precision and exploring novel applications, including semantic disambiguation and sentiment modulation. Understanding the underlying structure of learned representations and advancing the interpretability of neural network activations are key trajectories for future developments.

Conclusion

The methodology outlined in this paper presents a significant advancement in AI model steering and monitoring, enabling unprecedented control over LLM outputs with minimal computational resources. By leveraging aggregated nonlinear predictors and steering mechanisms, this research charts a path toward more accountable, responsive, and intelligent AI systems.