Spectrum-Aware Test-Time Steering

• The paper introduces a dynamic, spectrum-aware framework that selects optimal decoding strategies by maximizing a utility function combining accuracy, token cost, and latency.
• The methodology leverages empirical mean cost models and calibrated MLP accuracy predictions, allowing per-query routing among diverse inference strategies.
• Empirical results demonstrate significant improvements in both accuracy and efficiency for LLMs and VLMs, with fast, parameter-efficient test-time adaptation.

Spectrum-Aware Test-Time Steering (STS) denotes a family of dynamically adaptive mechanisms for routing queries or inputs across a finely parameterized “spectrum” of strategies, either at the level of decoding policies in generative models or adaptation shifts in representation space, in order to optimize a utility function that jointly considers accuracy, computational cost, and latency. The unifying feature is continuous or high-resolution steering among possible compute pathways, with joint awareness of spectrum-level trade-offs. This article details two lines of recent research under the STS designation: (1) inference scaling and decoding strategy routing in LLMs (Huang et al., 11 Sep 2025), and (2) principled latent-space steering for test-time adaptation in vision-LLMs (VLMs) (Dafnis et al., 12 Nov 2025).

1. Formal Problem Setting: Dynamic Spectrum Routing

STS in LLMs formalizes the inference-time scaling problem as dynamic, per-query selection from a set $S$ of candidate strategies, $s = (m, \theta_m)$, where $m$ may be best-of-$N$ sampling, beam search, or any other decoding policy, and $\theta_m$ comprises hyperparameters such as $N$ (number of samples), beam width, and depth (Huang et al., 11 Sep 2025). For each query $x$ and strategy $s$:

• $a_s(x) \in [0, 1]$: Predicted accuracy or reward.
• $T_s(x) \geq 0$: Expected output token cost.
• $s = (m, \theta_m)$0: Predicted wall-clock latency.

A utility function is defined: $s = (m, \theta_m)$1 where $s = (m, \theta_m)$2, $s = (m, \theta_m)$3 specify user penalties for token and latency cost, respectively. The optimal strategy is

$s = (m, \theta_m)$4

Alternatively, with hard constraints, one solves: $s = (m, \theta_m)$5

This framework generalizes static approaches, treating the space $s = (m, \theta_m)$6 as a spectrum over which queries can be routed according to their predicted difficulty and cost profile.

2. Cost Modeling and Prediction Framework

The STS approach circumvents the unavailability of $s = (m, \theta_m)$7, $s = (m, \theta_m)$8 at prediction time by employing empirical mean cost models. For each strategy $s = (m, \theta_m)$9: $m$0 which are used in place of per-query estimates. For $m$1, a two-layer MLP is trained to predict likelihood of correctness, using features comprising both an embedding $m$2 of the input and contextual features $m$3 of the strategy: $m$4 Platt scaling is used for improved calibration.

At test time, for user-specified $m$5, $m$6, the utility is given by: $m$7 The final chosen strategy $m$8 maximizes this surrogate utility, after which the model is decoded via the corresponding $m$9.

Empirical analysis demonstrates that mean cost proxies produce negligible loss ($N$01–2%) relative to ground-truth costs, and that the framework is robust to the use of varied embedding backbones (Huang et al., 11 Sep 2025).

3. Algorithmic and Operational Mechanics

A canonical STS routing process for LLMs comprises the following sequence:

1. Feature Extraction: Compute a semantic representation $N$1 (e.g., Qwen2.5-Instruct, BERT, etc.) for query $N$2; concatenate with strategy features $N$3.
2. Accuracy Estimation: MLP predicts $N$4, calibrated with empirical soft labels.
3. Cost Retrieval: Look up $N$5 and $N$6 for every $N$7.
4. Utility Maximization: For each $N$8, compute $N$9, select $\theta_m$0.
5. Decoding: Apply the selected decoding method $\theta_m$1 to output the result.

Routing is thus data- and spectrum-aware: queries predicted to be hard or ambiguous are steered toward computationally intensive strategies (e.g., deep beam search), while simple queries use lightweight methods (e.g., best-of-2). The spectrum can be arbitrarily enriched with new families of decoding methods or extended to cost axes beyond tokens and latency (e.g., GPU memory, energy).

A similar paradigm is applied in test-time adaptation for VLMs (Dafnis et al., 12 Nov 2025), where the “spectrum” is a spectral subspace extracted from textual prototypes, and steering is performed by learning per-sample shifts in the principal semantic directions.

4. Spectrum-Aware Steering in Latent Space for VLMs

In STS for VLMs (Dafnis et al., 12 Nov 2025), a “spectral subspace” in the semantic embedding space is extracted from the covariance of initial class prototypes $\theta_m$2 generated by the frozen text encoder, resulting in a principal basis $\theta_m$3 ($\theta_m$4). For a test image, a single vector $\theta_m$5 is learned to generate a latent shift $\theta_m$6 that is added to all class prototypes and renormalized; this adapted set is used for prediction.

The shift $\theta_m$7 is optimized per-sample at test time to minimize the entropy of predictions across $\theta_m$8 augmented views of the input, per

$\theta_m$9

where $N$0 is the marginal probability for class $N$1 across confidence-filtered views.

Key operational properties include:

• Only $N$2 parameters are optimized; encoders are frozen.
• No backpropagation through image/encoder weights is required.
• A single gradient step suffices for near-optimal adaptation.
• Typical $N$3 is on the order of $N$4–$N$5, capturing $N$690% feature variance.

5. Quantitative Results and Trade-offs

LLM Decoding (Huang et al., 11 Sep 2025)

STS in LLMs, evaluated on NuminaMath-CoT with Qwen2.5-1.5B-Instruct and a reward model, achieves:

Setting	Max Accuracy	Token Cost (approx)	Latency (approx)
Static Beam Search	~0.45	~2000+	~60s
Static Best-of-N	<0.45	--	--
STS (Adaptive)	0.50	~2000	~40s

• STS dominates both accuracy–cost and accuracy–latency trade-offs across the spectrum $N$7.
• At low penalties ($N$8 small), most queries route to high-cost strategies; as penalties increase, routing shifts to cheaper configurations without major accuracy loss.
• Dynamic adaptation within a single method family (e.g., only beam search, varying parameters) gives 3–5% accuracy improvements at fixed cost.

Vision-LLM Adaptation (Dafnis et al., 12 Nov 2025)

STS for VLMs, using CLIP-ViT-B/16, demonstrates:

Method	OOD-avg Accuracy	Inference Time (s)	GPU Memory (GB)
Zero-Shot	57.20%	--	--
TPT (Prompt Tuning)	60.71%	0.75	17.6
STS (Single)	62.64%	0.09	1.4
STS_Ensemble	64.96%	--	--

• STS achieves significant speed (8x faster inference) and footprint (12x smaller memory) gains relative to test-time prompt tuning, while offering higher OOD robustness.
• Prompt ensembling further lifts accuracy ceiling to 64.96% OOD-avg over diverse OOD and fine-grained splits.
• Under corruptions (CIFAR10-C), STS matches or exceeds TPT.

6. Extensibility, Generalization, and Practicality

The STS framework offers several extensibility and deployment strengths:

• Spectrum Enrichment: In LLMs, $N$9 may admit new decoding paradigms (tree-of-thought, multi-model routing) without altering the routing mechanism. In VLMs, new basis selection or regularization strategies can be swapped in.
• Cost-Axis Generalization: Additional cost axes (GPU memory, energy, or external call delays) can be incorporated as new penalty terms $x$0 in the utility function, supporting mixed-objective routing.
• Real-Time Suitability: Mean-cost lookups and low-parameter probes enable practical deployment in real-time agentic and interactive settings, where wall-clock delay is as critical as token usage.
• Empirical Robustness: Predictive proxies for costs and accuracies are reliable; ablations confirm that simple feature choices and single-step adaptation suffice for near-optimal performance.
• Parameter Efficiency: In latent-space steering, only a handful of per-sample parameters need to be optimized, facilitating rapid and scalable adaptation.

7. Significance, Limitations, and Open Directions

STS represents a systematic, spectrum-aware alternative to static or parallel generation methods for test-time strategy selection, providing flexible, data-driven adjustment to per-query computational budget and required response qualities. Numerical results indicate consistent gains in both accuracy and efficiency over baselines, with low operational overhead.

However, limitations include:

• The utility maximization relies on calibration of predictors and cost accuracy; gross misestimation may lead to suboptimal routing.
• For some deployment scenarios, fine-grained latency measurement and cost estimation may require continual recalibration.
• In VLM adaptation, the reliance on entropy minimization with augmentation assumes that OOD and errorful views are filtered; severe distributional shifts not captured by the textual subspace may require deeper adaptation.

A plausible implication is that future research will extend the STS paradigm to multi-modal, multi-agent, or highly dynamic environments, possibly incorporating reinforcement learning for online utility function tuning or integrating richer spectrum structures beyond simple hyperparameter grids.