Learning to Reason in 13 Parameters

Abstract: Recent research has shown that LLMs can learn to \textit{reason}, often via reinforcement learning. Some work even trains low-rank parameterizations for reasoning, but conventional LoRA cannot scale below the model dimension. We question whether even rank=1 LoRA is necessary for learning to reason and propose TinyLoRA, a method for scaling low-rank adapters to sizes as small as one parameter. Within our new parameterization, we are able to train the 8B parameter size of Qwen2.5 to 91\% accuracy on GSM8K with only 13 trained parameters in bf16 (26 total bytes). We find this trend holds in general: we are able to recover 90\% of performance improvements while training $1000x$ fewer parameters across a suite of more difficult learning-to-reason benchmarks such as AIME, AMC, and MATH500. Notably, we are only able to achieve such strong performance with RL: models trained using SFT require $100-1000x$ larger updates to reach the same performance.

• The paper demonstrates that reasoning performance on math benchmarks is achieved with as few as 13 updated parameters using reinforcement learning.
• It introduces a novel TinyLoRA architecture that utilizes weight tying and fixed random tensors to minimize the parameter update footprint.
• The study reveals that larger models require drastically fewer updates, reducing compute and memory costs while maintaining high accuracy.

TinyLoRA: Achieving Provable Reasoning with Minimal Parameter Updates

Introduction

The study "Learning to Reason in 13 Parameters" (2602.04118) explores the minimum requirements for effective reasoning finetuning in LLMs. Historically, approaches such as Low-Rank Adaptation (LoRA) have been adopted to reduce the scale of trained parameters from billions down to millions, with the belief that substantial update footprints are essential for task-specific adaptation. This work directly challenges that assumption by introducing TinyLoRA—a parameterization that achieves competitive reasoning accuracy on established math benchmarks while training only 13 parameters out of nearly 8 billion. The study conducts rigorous controlled experiments to assess the differential efficacy of supervised finetuning (SFT) versus reinforcement learning (RL) in extremely low-parameter regimes, hypothesizing and empirically verifying RL’s greater parameter efficiency.

Motivation and Background

Existing parameter-efficient finetuning methods, including LoRA (Hu et al., 2021), VeRA (Kopiczko et al., 2023), and LoRA-XS (Bałazy et al., 2024), generally operate at scales ranging from ten thousand to several million parameters. Applications leveraging LoRA variants find pragmatic benefits in reduced GPU memory usage and communication overheads, permitting both scalable serving (multi-tenancy of adapters) and enhanced personalization (Chen et al., 2023). However, the absolute parameter counts remain large relative to more theoretical results on model capacity (Cuccu et al., 2018), and do not exploit the inherent overparameterization and low intrinsic dimensionality of LLMs (Aghajanyan et al., 2020). The scope of adaptation achievable with extremely low-bit updates is thus poorly understood, particularly under RL objectives.

TinyLoRA: Design and Technical Innovations

TinyLoRA generalizes LoRA’s decomposition, scaling the trainable part down to a single parameter by using projections through fixed random tensors and aggressive weight tying. Instead of employing $r \times r$ matrices for each adapted module (as in LoRA-XS), TinyLoRA employs a single vector $\mathbf{v} \in \mathbb{R}^u$ and projects it via a random tensor $P \in \mathbb{R}^{u \times r \times r}$:

$$W' = W + U\Sigma\left(\sum_{i=1}^u v_i P_i\right)V^\top$$

Here, $U, \Sigma, V$ constitute the truncated SVD of $W$. Full block-level weight tying collapses the total trainable parameter count to as low as $u = 1$. The authors demonstrate that this ultra-low-capacity parameterization suffices to drive model behavior close to fully finetuned performance—given appropriate learning objectives and sufficiently pre-trained base models.

Key Experimental Findings

RL Requires Far Fewer Parameters than SFT

Empirically, reinforcement learning via Group Relative Policy Optimization (GRPO) yields strong performance with dramatically reduced updates; supervised finetuning (SFT) fails in the tiny update regime. On GSM8K—a standard math word problem benchmark—the Qwen2.5-7B-Instruct achieves:

• 91% accuracy with only 13 finetuned parameters under RL.
• Comparable performance to full finetuning with 120 parameters.
• SFT, by contrast, requires $100\text{--}1000\times$ more parameters to approach similar results and nearly saturates at baseline performance with tiny updates.

Figure 1: TinyLoRA rapidly approaches full-finetuned accuracy on GSM8K with minimal parameter updates; dashed lines show baselines.

Scaling Laws: Larger Models, Smaller Required Updates

The study finds a monotonic relationship between model size and minimal required update size for near-optimal performance. Bigger base models permit adaptation with fewer finetuned parameters.

Figure 2: Larger models require smaller parameter updates to reach high performance thresholds (e.g. 95% of peak performance).

Robustness Across Multiple Math Reasoning Benchmarks

TinyLoRA’s parameter efficiency generalizes beyond GSM8K. Across diverse math reasoning tasks (MATH500, AIME, AMC, Minerva Math), using under 200 parameters retains 87% of absolute performance improvement relative to full RL finetuning.

Figure 3: TinyLoRA performance trajectories during RL on MATH, validating learning capacity with <1KB updates.

Performance Ablations and Hyperparameters

The paper provides extensive ablation analyses on frozen rank $r$, trainable projection dimension $u$, and weight tying $n_{\text{tie}}$. Optimal performance is realized with minimal $r$ (typically $r=2$), prioritizing expressivity per module ($u$) over increased parameter sharing ($n_{\text{tie}}$).

Figure 4: Performance sensitivity to frozen rank $r$ and trainable rank $r_{trainable}$, indicating diminishing returns for larger $r$.

Figure 5: Tradeoffs between numbers of tied layers and trainable ranks in Qwen2.5-3B-Instruct; less sharing with higher $u$ is preferable.

Figure 6: Further ablation on tied layers and ranks $r$ across the 3B backbone, reinforcing the parameter allocation guidelines.

Contradictory and Strong Claims

• The authors claim "learning to reason can be achieved with as few as a single parameter update, if RL is used as the learning objective and the base model is sufficiently strong."
• SFT is fundamentally less information-dense than RL; RL’s capacity for performance-enhancing updates is demonstrably maximal at far smaller scales, due to cleaner signal separation via reward annotations.
• Scaling laws invert classical intuition: larger models require less adaptation to hit performance ceilings, pointing toward a future where ultra-parameter-efficient personalization is viable at trillion-parameter scales.

Theoretical and Practical Implications

The technical and conceptual leap enabled by TinyLoRA reframes the bottleneck for reasoning adaptation in LLMs. Rather than investing in millions of distinct adapters and brute-force parameter sweeps, practitioners can favor RL-based objectives and modular ultra-efficient parameterizations, dramatically reducing compute, storage, and deployment costs. This advance paves the way for large-scale multi-tenancy and personalized serving architectures, as the practical memory and communication loads of adapter storage and transmission collapse by orders of magnitude.

The findings also suggest that extensive model knowledge—acquired during pretraining—is programmable via minimal bit-level updates, particularly in domains like mathematical reasoning where reward signals are clearly defined and verified. The study raises open questions about broader generalization (beyond math), the limits of RL-based adaptation in creative and open-ended domains, and the theoretical reconciliation of overparameterization with intrinsic model dimensionality.

Speculations on Future Directions

• Adaptive and dynamic serving: TinyLoRA-like adapters could enable real-time, client-specific reasoning profiles with near-immediate swap-in and swap-out capabilities across model-serving stacks.
• Generalization to non-mathematical domains: The contrast between math tasks and others (science, language generation, etc.) warrants investigation into RL’s signal density and informativeness; future work could probe whether similar efficiency gains are achievable outside math.
• Fine-grained control of LLM behaviors: Minimal adapters may facilitate programmatic style, persona, or task control, potentially governed by RL over adaptive reward landscapes.
• Meta-learning and continual adaptation: RL-based tiny updates could integrate naturally with meta-learning protocols and lifelong learning, reducing catastrophic forgetting and optimizing memory footprints.

Conclusion

The study introduces TinyLoRA, demonstrating that reasoning adaptation in LLMs can be realized with updates orders of magnitude smaller than previous standards—down to 13 parameters (26 bytes) in practical RL settings. This challenges prevailing assumptions about necessary capacity for complex task adaptation and reveals RL’s superior information density in tiny parameter regimes. As model sizes continue to scale, such approaches are poised to fundamentally reshape practices for efficient, scalable, and adaptive AI deployment.