Stable Rank as Geometric Reward

• The paper introduces SR-GRPO, leveraging the stable rank metric to quantify the effective dimensionality of LLM hidden states for alignment without external rewards.
• It details a reinforcement learning procedure using Group Relative Policy Optimization with standardized geometric reward signals to distinguish high- from low-quality outputs.
• Empirical results demonstrate significant improvements in response quality and alignment benchmarks, evidencing stable rank's predictive power in LLM evaluation.

Stable Rank as Geometric Reward (SR-GRPO) represents a paradigm in LLM alignment that leverages the stable rank of internal model activations as an intrinsic, annotation-free reward for reinforcement learning. By directly quantifying the effective dimensionality of an LLM’s hidden states during generation, stable rank is utilized to distinguish high- from low-quality outputs and to optimize model behavior without human preference labels or external reward models. SR-GRPO operationalizes this geometric signal through a specialized reinforcement learning procedure—Group Relative Policy Optimization—thus providing a scalable, supervision-free avenue for LLM alignment (Tang et al., 2 Dec 2025).

1. Mathematical Foundation of Stable Rank

Stable rank is defined for a matrix $X \in \mathbb{R}^{T\times d}$, where rows correspond to $d$-dimensional hidden states across $T$ response tokens. If $\sigma_{1} \geq \cdots \geq \sigma_{\min(T,d)}$ are the singular values of $X$, then

• Total variance (Frobenius norm squared):

$\|X\|_F^2 = \sum_i \sigma_i^2$

• Dominant-direction variance (spectral norm squared):

$\|X\|_2^2 = \sigma_1^2$

• Stable rank:

$$\mathrm{sr}(X) = \frac{\|X\|_F^2}{\|X\|_2^2} = \frac{\sum_i \sigma_i^2}{\sigma_1^2}$$

Stable rank quantifies the “effective dimensionality” of the hidden-state geometry. $\mathrm{sr}(X) \approx 1$ indicates “representation collapse,” where variance is highly concentrated along one direction; larger values of $\mathrm{sr}(X)$ (up to $\min(T, d)$) reflect uniform spread of information across multiple directions. This measure is simple, unsupervised, and derived from the statistical structure of the internal activations, making it robust to overfitting and reward hacking (Tang et al., 2 Dec 2025).

2. Practical Computation in LLMs

The SR-GRPO framework specifies the following practical steps for stable rank computation:

• Layer Selection: Empirical analysis reveals that stable rank computed on the final hidden layer consistently achieves the strongest response-quality discrimination. Stable-rank signals from earlier layers yield near-random accuracy on benchmarks such as RewardBench.
• Hidden-State Extraction: For each prompt-response pair, activations are collected from the last hidden layer of a frozen reference LLM, forming a matrix $H \in \mathbb{R}^{T \times d}$.
• Computation: The Frobenius norm is obtained by summing squared entries. The spectral norm ($\sigma_1$) is approximated via a few iterations of the power method. The ratio yields the stable rank.
• Normalization and Batching: During RL updates, for each prompt, $K$ candidate responses are generated. Their stable ranks $\{r_k\}_{k=1}^K$ are standardized group-wise for advantage estimation:

$A_k = \frac{r_k - \mu}{\sigma + \epsilon}$

where $\mu$ and $\sigma$ are group mean and standard deviation, $\epsilon$ is for numerical stability.

This computation is annotation-free, purely geometric, and compatible with large-batch, high-throughput RL pipelines (Tang et al., 2 Dec 2025).

3. SR-GRPO Algorithmic Framework

SR-GRPO builds upon the Group Relative Policy Optimization (GRPO) framework, replacing external rewards with stable rank scores:

• Initialization: Policy $\pi_\phi$ starts from a reference model $\pi_\text{ref}$.
• Sampling: For each prompt $x_i$, sample $K$ responses $\{y_{i,k}\}$ from $\pi_\phi$.
• Reward Evaluation: Compute $r_{i,k} = \mathrm{sr}(H_{i,k})$ via $\pi_\text{ref}$’s final layer for each response.
• Advantage Estimation: Standardize rewards to $\{A_{i,k}\}$ within-group.
• Policy Update: Optimize the expected standardized geometric reward, penalized by the KL divergence to hold the policy near the reference:

$$J(\phi) = \mathbb{E}\left[\frac{1}{K} \sum_k \rho_{i,k} A_{i,k} - \beta D_\mathrm{KL}(\pi_\phi \Vert \pi_\text{ref}) \right]$$

where

$$\rho_{i,k} = \frac{\pi_\phi(y_{i,k} \mid x_i)}{\pi_{\phi_{old}}(y_{i,k} \mid x_i)}$$

and the gradient is

$$\nabla_\phi J(\phi) = \mathbb{E}\left[ \nabla_\phi \log \pi_\phi(a \mid s) \, r_\mathrm{sr} \right]$$

with KL penalty for stability (Tang et al., 2 Dec 2025).

This procedure leverages only internal model geometry accessed through a reference model, ensuring robustness to reward hacking since policy-induced changes in geometry are penalized via the KL constraint.

4. Theoretical Motivation

Two primary theoretical rationales support stable rank as a quality metric:

• Softmax Bottleneck Theory: Effective language modeling requires hidden states that span a high-rank manifold; low-rank (low stable-rank) representations fundamentally limit expressiveness (Yang et al. 2018, Godey et al. 2024).
• Representation Collapse: Empirical studies link low-dimensional collapse of activations to output degradation, including hallucination and repetition (Gao et al. 2019).

While SR-GRPO does not provide new theoretical bounds, it asserts that stable rank captures the trade-off between information spread and collapse in hidden-state geometry—the core phenomena implicated in generation quality. By anchoring computation to a frozen reference model, SR-GRPO is resistant to trivial exploitation of the reward signal, since the agent cannot directly manipulate the geometric evaluation space without diverging from the reference policy (Tang et al., 2 Dec 2025).

5. Empirical Assessment

SR-GRPO is extensively validated on multiple alignment and quality-benchmarking settings:

• RewardBench (zero-shot quality proxy): Using Qwen3-8B as reference, stable rank predicts human-preferred responses with 84.04% accuracy, outperforming three LLM-as-judge baselines (pointwise: 83.70%, pairwise: 71.98%, IPO: 78.02%).
• Best-of-N Decoding: Across STEM (GPQA, MMLU-redux) and mathematical reasoning tasks (MATH500, OlympiadBench, AMC23), selecting the highest stable-rank response from $N=16$ candidates yields a mean improvement of +11.3 percentage points over greedy decoding, with up to +20.5 pp on Llama-3.2-1B.
• RL Alignment (SR-GRPO): On SmolTalk2 prompts, SR-GRPO delivers +10 pp on STEM and +19 pp on mathematical reasoning on Qwen2.5-1.5B-Instruct (and similar gains for DeepSeek-R1-Distill-Qwen-1.5B), outperforming learned reward models (e.g., Skywork-Reward-V2-Qwen3-1.7B) and self-evaluation baselines (Self-Reward, Perplexity, IPO); open chat improvement is +26.2 Elo and +19 Elo, respectively.

These results demonstrate the capacity of stable rank to serve not only as a high-accuracy offline proxy for response quality but also as an effective reinforcement learning reward for end-to-end, annotation-free alignment (Tang et al., 2 Dec 2025).

6. Discussion, Limitations, and Future Directions

• Metric Correlations: Stable rank correlates moderately ($|\rho| \approx 0.2\mbox{–}0.4$) with other quality signals (e.g., semantic coherence, information density) but does not capture all aspects of quality.
• Context Length Sensitivity: Performance diminishes if activations are truncated below $\sim$512 tokens, with saturation above this threshold. Degradation is especially evident for code domains.
• Prompt Formatting Robustness: Stable rank accuracy varies by at most 3 percentage points across diverse prompt-response formatting templates, suggesting broad applicability without prompt engineering.
• Research Directions: Proposed extensions include exploration of alternative geometric invariants (subspace angles, curvature), multi-metric fusion for richer reward shaping, and longitudinal tracking of geometric signals (e.g., step-level stable-rank dynamics for chain-of-thought tasks) (Tang et al., 2 Dec 2025).

In summary, SR-GRPO demonstrates that the stable rank of LLM hidden-state activations constitutes a reliable, intrinsic metric for response quality and provides a foundation for annotation-free RL alignment. This geometric approach expands the toolkit for LLM alignment by enabling scalable, supervision-independent policy optimization grounded in internal model statistics.