Graph-Regularized Sparse Autoencoders (GSAEs)

• Graph-Regularized Sparse Autoencoders (GSAEs) are deep models that integrate graph-based regularization and sparsity to capture distributed, safety-aligned neural representations.
• They combine reconstruction loss, ℓ1 sparsity, and a Laplacian smoothness penalty over neuron co-activation graphs to enforce coherent latent feature extraction.
• Empirical results demonstrate that GSAEs boost selective refusal performance and robustness against adversarial attacks, significantly outperforming previous safety steering approaches.

Graph-Regularized Sparse Autoencoders (GSAEs) are a class of neural architectures designed to recover distributed, concept-aligned representations from deep models, notably for intervening on LLMs in safety-critical contexts. By introducing a Laplacian smoothness penalty over a neuron co-activation graph, GSAEs extend traditional sparse autoencoders (SAEs) to capture safety concepts as coherent patterns spanning multiple latent features, rather than isolating them within single dimensions. Empirical evidence demonstrates that GSAEs enable state-of-the-art selective refusal performance and robustness against adversarial prompt attacks, substantially improving upon prior safety steering methods (Yeon et al., 7 Dec 2025).

1. Model Architecture

GSAEs process pooled hidden states $h \in \mathbb{R}^d$ extracted from selected transformer layers as input. The encoder is a linear transformation followed by a ReLU activation: $$z = \mathrm{ReLU}(W^{(e)} h), \quad W^{(e)} \in \mathbb{R}^{k \times d},\; k \gg d$$ The latent code $z \in \mathbb{R}^k$ is enforced to be sparse via $\ell_1$ regularization. The decoder reconstructs the input using a linear transformation: $$\hat h = W^{(d)} z, \quad W^{(d)} \in \mathbb{R}^{d \times k}$$ For a dataset of $N$ samples, the sets $\{h_i, z_i, \hat h_i\}_{i=1}^N$ are maintained. This architecture is designed to enable distributed feature encoding while favoring sparse, interpretable activations.

2. Objective Function and Graph Regularization

GSAEs optimize a composite loss: $$L_{\text{GSAE}} = L_{\rm recon} + \lambda\,\|Z\|_1 + \mu\,\mathrm{Tr}\left((W^{(d)})^\top L W^{(d)}\right)$$ where:

• $$L_{\rm recon} = \sum_{i=1}^N \| h_i - \hat h_i \|_2^2$$ is the reconstruction error.
• The sparsity penalty $L_{\ell_1} = \lambda \sum_{i=1}^N \| z_i \|_1$ encourages most latent activations to be zero.
• The graph Laplacian term uses $L = D - A$, where $A$ is the adjacency matrix of the neuron co-activation graph (detail in Section 3), and $D$ is the diagonal degree matrix. This term enforces decoded features $v_j$ (columns of $W^{(d)}$) to be smooth with respect to neuron co-activations: $$L_{\rm graph} = \mu \sum_{j=1}^k v_j^\top L v_j = \mu\, \mathrm{Tr}\left((W^{(d)})^\top L W^{(d)}\right)$$ The overall effect is to favor features that capture smooth, distributed structure over the co-activation manifold inferred from model activations (Yeon et al., 7 Dec 2025).

3. Construction of the Neuron Co-Activation Graph

The co-activation graph $G=(V,E)$ encodes functional similarity among neurons based on their activation profiles across inputs. The construction procedure:

• Collect pooled activations $H \in \mathbb{R}^{d \times N}$
• For each neuron pair, compute cosine similarity: $$s_{ij} = \frac{\langle h_{i,:}, h_{j,:} \rangle}{\|h_{i,:}\|_2\,\|h_{j,:}\|_2}$$
• Adjacency entries are thresholded at $T$ (e.g., $T=0.6$): $$A_{ij} = \begin{cases} s_{ij} & \text{if } s_{ij} > T \ 0 & \text{otherwise} \end{cases}$$
• The degree matrix $D$ is diagonal with $D_{ii} = \sum_j A_{ij}$
• The unnormalized Laplacian $L = D - A$ is then used in the regularization term.

This process grounds concept learning in the empirical distribution of neuron co-activations, enabling the Laplacian penalty to promote structured, interpretable decompositions.

4. Runtime Safety Steering with Dual Gating

GSAEs are applied for online safety intervention with a two-stage gating system, enabling dynamic, context-dependent steering of LLM outputs.

4.1 Assembling the Spectral Vector Bank

For each decoder vector $v_i$, three scores quantify suitability for steering:

• Spectral smoothness: $E_i = v_i^\top L v_i / \|v_i\|_2^2$, then $s^{\rm lap}_i = \exp(-\beta E_i)$
• Semantic relevance: $s^{\rm sup}_i = |\theta_i|$ via a supervised linear probe discriminating harmful from benign prompts
• Causal efficacy: $s^{\rm infl}_i$ is the observed change in refusal probability when intervening along $v_i$

Latent directions are combined with final weights: $$w_i = \frac{ (s^{\rm lap}_i)^\alpha (s^{\rm sup}_i)^\beta (s^{\rm infl}_i)^\gamma }{ \sum_j (s^{\rm lap}_j)^\alpha (s^{\rm sup}_j)^\beta (s^{\rm infl}_j)^\gamma }$$ ($\alpha = \beta = \gamma = 1$ in experiments). The most salient vectors by $w_i$ comprise the "spectral vector bank" used for steering.

4.2 Input and Continuation Gating

• Input gate: A prompt is encoded to $z$ and passed to a random forest classifier $g$ yielding $\Pr_{\rm harm}$. For $\Pr_{\rm harm} > t_{\rm high}$, output is refused; for $\Pr_{\rm harm} < t_{\rm low}$, decoding proceeds unaltered; otherwise monitoring mode is entered.
• Continuation gate: At each decode step, a risk score $r_t$ (from the classifier) determines gating via hysteresis thresholds: exceeding $d_{\rm high}$ for $S_\uparrow$ steps opens the gate ($y_t = 1$), while falling below $d_{\rm low}$ for $S_\downarrow$ steps closes it ($y_t = 0$).

4.3 Steering Intervention

When the gate is open, the hidden state is updated: $$\Delta h_t = \alpha_0 \sum_{i \in \mathcal S} w_i \cos(h_t, v_i) v_i$$ where $\mathcal S$ is the set of top-$m$ vectors by $w_i$. The intervention $h_t \leftarrow h_t - \Delta h_t$ is applied prior to logits projection, modifying output probabilities to enforce safety.

5. Training Procedure and Hyperparameters

Key hyperparameters for empirical effectiveness:

• Latent dimension $k = 16d$
• Sparsity weight $\lambda = 1 \times 10^{-4}$
• Graph regularization $\mu = 1 \times 10^{-3}$
• Graph threshold $T=0.6$
• Optimizer: Adam, learning rate $1\times 10^{-3}$, batch size 32, $1 \times 10^5$ iterations
• Gate thresholds: $(t_{\rm low}, t_{\rm high}) = (0.30, 0.65)$; $(d_{\rm low}, d_{\rm high}) = (0.7, 0.9)$; hysteresis steps $(2, 3)$
• Steering strength $\alpha_0 = 2.5$

Training and intervention phases are modular, with three algorithmic phases—(1) GSAE model training, (2) spectral vector bank curation, (3) dual-gated steering—each summarized by precise pseudocode (Yeon et al., 7 Dec 2025).

6. Empirical Results and Comparative Performance

Table: Summary of Main Metrics (Llama-3 8B)

Metric	GSAE	SAE steering	SafeSwitch
Selective refusal $A_s$	$82\%$	$42\%$	$58\%$
TriviaQA (utility)	$70.0\%$	—	—
TruthfulQA (utility)	$65.4\%$	—	—
GSM8K (utility)	$74.2\%$	—	—
Robust harm-refusal rate	$\geq90\%$	$40-70\%$	$40-70\%$

GSAE steering provides a substantial increase in selective refusal score ($A_s$) over both standard SAE steering and SafeSwitch, while retaining strong task accuracy across QA benchmarks. On adversarial jailbreak tests (GCG, AutoDAN, TAP, adaptive), GSAE sustains a harm-refusal rate $\geq 90\%$, whereas prior methods degrade to $40$–$70\%$. Performance generalizes across LLaMA-3, Mistral, Qwen, and Phi model families, consistently exceeding SafeSwitch by $10$–$20$ points in $A_s$.

7. Significance and Conceptual Advances

GSAEs address limitations of prior activation steering approaches, which operationalized abstract safety concepts as single-feature phenomena. Experiments confirm that GSAEs recover smooth, distributed latent representations necessary to steer for nuanced, non-localized safety attributes (e.g., refusal, temporality), enforcing adaptive refusals while minimizing detrimental effects on benign utility. The dual-gated inference mechanism, underpinned by graph-regularized autoencoding, supports real-time control over LLM outputs in both prompt and continuation phases. This suggests a general paradigm for distributed concept operationalization in safety-critical model interventions (Yeon et al., 7 Dec 2025).