Downstream Capabilities Scaling Laws

• Downstream capabilities scaling laws are formulas that quantify the relationship between model size, pretraining data, and fine-tuning data in determining performance on target tasks.
• They integrate multiple factors—such as data, compute, and parameter count—to identify regime transitions where distillation outperforms fine-tuning or vice versa.
• Empirical findings, validated with benchmarks like ImageNet, provide practical guidelines for resource allocation and optimizing transfer learning strategies.

Downstream Capabilities Scaling Laws

Downstream capabilities scaling laws characterize how model performance on target tasks—often differing from the pretraining objective—changes as a function of model size, pretraining data, compute, and fine-tuning or transfer data. In contrast to classical scaling laws for upstream loss, downstream scaling laws must capture additional phenomena such as adaptation under data limitations, knowledge transfer, phase transitions, and task-specific regimes. Recent research establishes both practical predictive laws for mainstream models and quantifies the boundaries where predictability fails or where non-trivial transitions govern performance.

1. Mathematical Forms of Downstream Scaling Laws

Scaling laws for downstream capabilities extend classical power-law models by incorporating multiple axes—typically pretraining dataset size ($P$), model parameter count ($M$), and downstream (fine-tuning) dataset size ($N$). For visual transfer learning tasks, downstream error rate $E$ and cross-entropy loss $L$ are empirically described by (Yang et al., 17 Apr 2025):

$$E(P,\,M,\,N) \;=\; E_{\infty} \;+\; \lambda_{p}\,P^{-\alpha} \;+\; \lambda_{m}\,M^{-\beta} \;+\; \lambda_{f}\,N^{-\gamma}$$

$$L(P,\,M,\,N) \;=\; L_{\infty} \;+\; \omega_{p}\,P^{-\alpha''} \;+\; \omega_{m}\,M^{-\beta''} \;+\; \omega_{f}\,N^{-\gamma''}$$

where $E_{\infty},L_{\infty}$ are irreducible error/loss floors, and $\alpha,\beta,\gamma$ are empirically fitted exponents. For instance, on ImageNet-100, values are $$\alpha \approx 0.62,\, \beta\approx4.88,\, \gamma\approx0.38$$.

These multi-term forms generalize to other domains. For LLMs, accuracy $Q$ on downstream benchmarks at fixed token-to-parameter ratio is fit by (Krajewski et al., 9 Dec 2025):

$-\log Q = \frac{A}{C^{\alpha}}$

and, allowing model size $N$ and dataset size $D$ to vary,

$$-\log Q = \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

where $C$ is total training compute (FLOPs), $A,B,\alpha,\beta$ are benchmark-specific. These forms provide accurate descriptions over wide ranges of budget and size, with predictability contingent on surpassing certain data and compute thresholds.

2. Distillation Boundary Theory and Critical Regimes

A central result in data-constrained visual transfer is the "distillation boundary theory" (Yang et al., 17 Apr 2025). When training a large student (size $M_t$) via distillation from a teacher ($M_s$), error admits a four-term law:

$$E_{\rm distill}(P,\,N,\,M_{t},\,M_{s}) = E_{\infty}' + \lambda_{p}'\,P^{-\alpha'} + \lambda_{f}'\,N^{-\gamma'} + \lambda_{s}'\,M_{s}^{-\beta'} + \lambda_{t}'\,M_{t}^{-\eta'}$$

Empirically, fitted exponents on ImageNet-100 are $$\alpha'\approx0.702,\,\beta'\approx5.840,\,\gamma'\approx0.338,\,\eta'\approx2.053$$. A key implication is the existence of a critical downstream data threshold $N_c$:

$$N_{c} = \Theta\bigl(M_{t}^{(\beta'-\beta)/(\gamma-\gamma')} M_{s}^{(\beta-\beta')/(\gamma-\gamma')} P^{(\alpha'-\alpha)/(\gamma-\gamma')}\bigr)$$

For $N<N_c$, distillation yields lower error; for $N>N_c$, base model fine-tuning dominates. This formalizes the transition between regimes where knowledge transfer is beneficial and when task-specific adaptation overtakes inherited information.

3. Empirical Exponents, Task Dependence, and Regimes

The scaling exponents $(\alpha,\beta,\gamma)$ vary with task, dataset, and architecture. For visual benchmarks:

Dataset	$\alpha$	$\beta$	$\gamma$
ImageNet-100	0.620	4.882	0.377
TinyImageNet	0.412	5.086	0.359
CIFAR-100	0.609	1.797	0.587
CIFAR-10	10.129	4.975	0.331

These values quantify the sensitivity of downstream error to each scaling variable. Notably, exponents for distillation (e.g., $\gamma' = 0.338$) are generally lower than for base models ($\gamma=0.377$), reflecting the improved data efficiency induced by knowledge transfer. The critical boundary $N_c$ shifts gradually with model size and pretraining scale, dictating regime transitions for optimal training pipelines.

4. Distillation vs. Direct Fine-Tuning: Interpreting the Two Regimes

Two operational regimes are delineated by the scaling theory (Yang et al., 17 Apr 2025):

• Distillation Superiority ($N<N_c$): In low-data regimes, student's gradients are prone to overfitting. The teacher's guidance (softened logits or feature alignment) lowers variance and enhances generalization. Empirically, distillation can yield up to $1\%$ lower error in data-scarce conditions.
• Pretraining Dominance ($N>N_c$): As more task-specific data become available, the student's optimization can surpass the teacher's limitations. Continued distillation then restricts achievable error floors; base-model adaptation becomes preferable.

Figures 8 and 9 in (Yang et al., 17 Apr 2025) illustrate this, with error curves crossing over at data thresholds predicted by the analytic $N_c$.

5. Practical Guidelines and Implications for Model Scaling

Derived scaling laws provide actionable recommendations for practitioners:

• Employ distillation when downstream data $N$ falls substantially below the analytically predicted $N_c$; this yields best sample efficiency and error rates per data point.
• For $N \gg N_c$, forego distillation in favor of direct adaptation, allocating computational resources to more task-specific epochs or augmentation rather than to expensive teacher-student setups.
• In deployment scenarios with uncertain $N$, a sweep over possible fine-tuning set sizes can empirically localize the regime boundary by observing where the distillation advantage vanishes.
• Distillation incurs a 2–5× increase in training cost, thus should be reserved for sub-critical data regimes where it demonstrably aids performance.

This bifurcation clarifies optimal resource allocation and resolves competing intuitions about the value of knowledge inheritance vs. task-specific adaptation.

6. Broader Context and Limitations

Downstream scaling laws as developed in (Yang et al., 17 Apr 2025) bridge a key gap in the literature, which previously emphasized large-scale pretraining but lacked robust predictive tools for adaptation under data constraints. The demonstrated three-term power laws, analytic regime transitions, and empirical validations provide a foundation for computational planning and model design in vision applications requiring efficient transfer.

However, proposed forms remain empirical and task-dependent: the precise values of scaling exponents—and the location of $N_c$—must be fitted to observed data for each deployment case. The framework is validated for parameter scales of $2.5\,$M–$38\,$M and fine-tuning data up to $10^6$ samples; extrapolation beyond this range requires additional confirmation.

7. Significance for Scaling-Laws Research

The concept of downstream capabilities scaling laws establishes a unified predictive language for transfer learning regimes, formalizing how inherited knowledge and task-specific adaptation interact as a function of accessible data, compute, and model size. The introduction of the distillation critical threshold $N_c$ resolves a longstanding ambiguity in transfer efficiency and delineates optimal strategies for data-constrained and data-rich regimes. This framework sets the stage for further investigations into broader modalities, multi-stage adaptation, and the principled design of future transfer and distillation algorithms (Yang et al., 17 Apr 2025).