Log-Linearity in LLMs

• Log-linearity in LLMs is the nearly linear evolution of model weights and logits, enabling efficient extrapolation and precise behavioral control.
• Extrapolation techniques, including weight and logits prediction, allow skipping RL steps and achieving up to 6.1× speedup in training efficiency.
• Logit-Linear Selection leverages subtle logit shifts to fine-tune models for consistent behavior, raising implications for model security and dataset auditing.

Log-linearity in LLMs refers to the empirical and theoretical observation that both the internal parameters (weights) and output statistics (log-probabilities or logits) of an LLM evolve in a strongly linear fashion under a variety of training protocols, particularly reinforcement learning with verifiable rewards (RLVR) and preference-based fine-tuning. This linearity manifests not only in the time evolution of model weights but also in the geometric structure of context-dependent log-probabilities, enabling novel algorithms for model extrapolation, subset selection, and behavioral control. Log-linearity has important implications for model efficiency, interpretability, dataset auditing, and both intentional and unintentional behavioral manipulation of LLMs (Wang et al., 8 Jan 2026, Aden-Ali et al., 4 Feb 2026).

1. Mathematical Characterization of Log-Linearity

Log-linearity in LLMs can be formalized in two principal senses: trajectory linearity during training and low-rank logit structure across contexts.

Timewise Linearity in RLVR Training

Let $y_i(t)$ denote a model quantity at training step $t$—for example, a single model weight or token log-probability. The linearity is measured by fitting a regression $y_i(t) = a_i t + b_i$ and computing the coefficient of determination:

$$R^2_i = 1 - \frac{\sum_k (y_i(t_k) - \hat{y}_i(t_k))^2}{\sum_k (y_i(t_k) - \bar{y_i})^2}$$

with $\hat{y}_i(t) = a_i t + b_i$ and $\bar{y}_i$ the mean. For centered variables, $R^2_i$ coincides with the squared Pearson correlation between $t$ and $y_i$.

Empirically, over 80% of sampled weights and logits across LLMs exhibit $R^2 > 0.7$, with distributions tightly concentrated around 0.9, indicating nearly perfect linear evolution with training steps (Wang et al., 8 Jan 2026).

Log-Linear Context Representation

For a model $t$0, a context $t$1—composed of system prompt $t$2, user prompt $t$3, and response $t$4—admits an $t$5-approximately log-linear representation if vector embeddings $t$6 and $t$7 exist such that

$t$8

uniformly over $t$9 (Aden-Ali et al., 4 Feb 2026). In matrix notation, the output log-probability matrix $y_i(t) = a_i t + b_i$0 satisfies $y_i(t) = a_i t + b_i$1, i.e., is approximately low-rank.

Mixtures of system prompts correspond to linear addition of their embeddings in predictor space:

$y_i(t) = a_i t + b_i$2

2. Extrapolation Algorithms Leveraging Log-Linearity

The strict temporal linearity of weights and logits in RLVR post-training enables efficient step-skipping and lookahead through extrapolation:

Logits Extrapolation

Given output logits $y_i(t) = a_i t + b_i$3, $y_i(t) = a_i t + b_i$4 at steps $y_i(t) = a_i t + b_i$5, the logits at future step $y_i(t) = a_i t + b_i$6 are estimated as

$y_i(t) = a_i t + b_i$7

Sampling from $y_i(t) = a_i t + b_i$8 at inference time yields the desired extrapolated behavior.

Weight Extrapolation

Let $y_i(t) = a_i t + b_i$9 be full model parameter tensors at steps $$R^2_i = 1 - \frac{\sum_k (y_i(t_k) - \hat{y}_i(t_k))^2}{\sum_k (y_i(t_k) - \bar{y_i})^2}$$0. Extrapolated parameters are computed via

$$R^2_i = 1 - \frac{\sum_k (y_i(t_k) - \hat{y}_i(t_k))^2}{\sum_k (y_i(t_k) - \bar{y_i})^2}$$1

The resulting model $$R^2_i = 1 - \frac{\sum_k (y_i(t_k) - \hat{y}_i(t_k))^2}{\sum_k (y_i(t_k) - \bar{y_i})^2}$$2 can be directly evaluated or used as an initial checkpoint for further fine-tuning (Wang et al., 8 Jan 2026).

RL-Extra (Interleaved Extrapolation)

RL-Extra interleaves $$R^2_i = 1 - \frac{\sum_k (y_i(t_k) - \hat{y}_i(t_k))^2}{\sum_k (y_i(t_k) - \bar{y_i})^2}$$3 real RL steps with $$R^2_i = 1 - \frac{\sum_k (y_i(t_k) - \hat{y}_i(t_k))^2}{\sum_k (y_i(t_k) - \bar{y_i})^2}$$4 weight extrapolation steps in cycles of length $$R^2_i = 1 - \frac{\sum_k (y_i(t_k) - \hat{y}_i(t_k))^2}{\sum_k (y_i(t_k) - \bar{y_i})^2}$$5, updating by gradient descent if $$R^2_i = 1 - \frac{\sum_k (y_i(t_k) - \hat{y}_i(t_k))^2}{\sum_k (y_i(t_k) - \bar{y_i})^2}$$6, else by extrapolation. This augments RLVR compute efficiency while mitigating long-range drift.

3. Subset Selection and Behavioral Control via Log-Linearity

Log-linearity extends beyond temporal trajectories to the interaction between dataset structure and context-elicited behavior. The Logit-Linear Selection (LLS) method operationalizes this:

LLS Algorithm

Given a teacher model $$R^2_i = 1 - \frac{\sum_k (y_i(t_k) - \hat{y}_i(t_k))^2}{\sum_k (y_i(t_k) - \bar{y_i})^2}$$7, preference dataset $$R^2_i = 1 - \frac{\sum_k (y_i(t_k) - \hat{y}_i(t_k))^2}{\sum_k (y_i(t_k) - \bar{y_i})^2}$$8, and target system prompt $$R^2_i = 1 - \frac{\sum_k (y_i(t_k) - \hat{y}_i(t_k))^2}{\sum_k (y_i(t_k) - \bar{y_i})^2}$$9, each example is scored by the logit shift:

$\hat{y}_i(t) = a_i t + b_i$0

The top $\hat{y}_i(t) = a_i t + b_i$1 quantile of $\hat{y}_i(t) = a_i t + b_i$2 defines a subset $\hat{y}_i(t) = a_i t + b_i$3. Fine-tuning a student model $\hat{y}_i(t) = a_i t + b_i$4 on $\hat{y}_i(t) = a_i t + b_i$5 (using DPO or similar) causes $\hat{y}_i(t) = a_i t + b_i$6, even when queried without system prompt, to mimic the behavior induced in $\hat{y}_i(t) = a_i t + b_i$7 by $\hat{y}_i(t) = a_i t + b_i$8.

A summary of LLS steps is provided below:

Step	Operation	Notes
1	Compute $\hat{y}_i(t) = a_i t + b_i$9 for all examples	Requires two forward passes per example
2	Normalize (optional)	Adjust for varying token lengths
3	Filter and sort	Keep $\bar{y}_i$0, select top $\bar{y}_i$1 fraction
4	Fine-tune on subset $\bar{y}_i$2	Using standard preference alignment

4. Experimental Verification and Quantitative Results

Linearity and its algorithmic consequences are validated across diverse LLMs, datasets, and tasks.

Trajectory Linearity in RLVR

• For DeepSeek-R1-Distilled-Qwen-1.5B on four RLVR benchmarks (AIME '24, AIME '25, MATH-500, LiveCodeBench), $\bar{y}_i$3 for both weights and logits is sharply concentrated at $\bar{y}_i$4, with $\bar{y}_i$5 of sampled tokens exhibiting $\bar{y}_i$6 (Wang et al., 8 Jan 2026).
• Median $\bar{y}_i$7 generalizes across model sizes (1.5B–8B), architectures (Qwen, Llama), and RL algorithms (GRPO, GSPO, Reinforce++).

Efficiency Gains from Extrapolation

• Weight extrapolation, projecting from $\bar{y}_i$8, $\bar{y}_i$9 to $R^2_i$0 matches the baseline RL performance on AIME-24 (Avg@64 $R^2_i$1) at fraction of compute.
• Logits extrapolation, e.g. from steps $R^2_i$2 to $R^2_i$3, yields an absolute $R^2_i$43% gain over direct RL at $R^2_i$5 steps.
• RL-Extra with $R^2_i$6 achieves a $R^2_i$7 speedup with no loss in final accuracy.

Logit-Linear Selection Behavioral Experiments

• Animal preference induction: Models fine-tuned via LLS reference “You really love [animal]s,” mentioning the [animal] at rates comparable to system-prompted baselines, despite zero [animal] mentions in the fine-tuning subset.
• Instruction-following and translation: LLS fine-tuning for translation elicits pure target-language outputs (e.g., Spanish, Chinese), with no source-language leakage in $R^2_i$8.
• Persona shift: LLS using “You are an evil ruler…” induces “evil” completions at rates of $R^2_i$9–$t$0, equivalent to or exceeding system-prompted reference models. Random subsets confer no effect (Aden-Ali et al., 4 Feb 2026).

5. Implications for Model Understanding, Dataset Design, and Security

The emergence of log-linearity as a fundamental regularity in LLMs substantially alters the landscape of post-training, behavioral auditing, and safety.

• In RLVR, most training steps merely amplify trajectories set early, with minimal qualitative novelty beyond initial trends.
• Linear extrapolation—either in logits or weights—can reliably “skip” hundreds of RLVR steps, or extend to steps where direct RLVR becomes unstable due to entropy collapse.
• LLS demonstrates that coherent behaviors (preferences, personas, instruction compliance) can be implanted through carefully chosen subsets lacking obvious semantic correlates—posing both opportunities (e.g., efficient watermarking, fine-grained control) and risks (e.g., stealth backdoors, dataset poisoning).
• Standard heuristics like keyword removal may fail, as ensemble linear effects from innocuous-appearing records accumulate in model logits.

A plausible implication is the need for new linear-algebraic dataset auditing tools to detect and attribute subliminal logit-space behaviors, as well as the development of defenses such as null-space regularization or adversarial prompt probing targeting undesired linear shifts (Aden-Ali et al., 4 Feb 2026).

6. Universality and Theoretical Perspective

Log-linearity appears to be a universal phenomenon across architectures (Qwen, Llama), model sizes (1.5B–8B), and post-training RL algorithms, particularly under low-noise optimizers such as AdamW with small learning rates and large batch sizes. Both mechanistic interpretability and the literature on spurious correlation in deep models intersect with the observed low-rank, log-linear structure: small, distributed correlations in high-dimensional data can produce macroscopic behavioral changes under gradient-based fine-tuning. Exploiting log-linearity is poised to become standard practice in efficient, robust LLM development and curation (Wang et al., 8 Jan 2026, Aden-Ali et al., 4 Feb 2026).