Skip Connection Layer Fine-Tuning

• Skip Connection Layer Fine-Tuning is a parameter-efficient adaptation paradigm that uses layer- and class-wise skipping to reduce fine-tuning complexity while enhancing generalization.
• It employs techniques like layer-wise freezing, dynamic class token pruning, and skip-connection injection to restrict gradient flow to critical segments, achieving significant speed-ups.
• Empirical results show that SCL FT can match or surpass full fine-tuning accuracy while updating only a small fraction of parameters and substantially lowering memory usage.

Skip Connection Layer Fine-Tuning (SCL FT) is a parameter-efficient adaptation paradigm leveraging skip connections or layer skipping to induce effective transfer, efficient computation, and improved generalization in deep pre-trained models. SCL FT encompasses a family of approaches, exemplified by Skip Tuning for vision-LLMs (VLMs), skip-augmented transfer methods for neural collapse alignment in classification, and block-level skip adaptation in LLMs. These methods systematically reduce the computational and memory cost of classical fine-tuning—while often improving accuracy—by modifying gradient flows, freezing substantial model portions, and restricting parameter updates to vital segments of the network (Wu et al., 2024, Li et al., 2022, Pathak et al., 18 Jul 2025).

1. Core Principles and Motivations

Fine-tuning large pre-trained models traditionally involves propagating features and gradients through all layers. In models with $N$ layers (e.g., ViT-B/16 with $N \sim 12$) and, for multi-class problems, $M$ class tokens (100–1000 in VLMs), this results in a computational and memory complexity scaling as $C_\text{full} = N \times (C_V + M \cdot C_T)$, where $C_V$ and $C_T$ are per-layer costs for vision and text branches, respectively (Wu et al., 2024). Empirical analysis shows:

• Feature Sensitivity decays in shallow layers: Most adaptation occurs in the deepest layers; shallow layers exhibit near-zero feature sensitivity for target tasks.
• Class-token Gradient Dependence is sparse: Only a small token subset drives meaningful gradients per sample.

Thus, reducing the length and width of the Feature-Gradient Propagation Flow (FGPF) by skipping early layers and class tokens is both computationally efficient and empirically robust. SCL FT thus includes: (1) freezing and caching early layers (layer-wise skipping), (2) selecting or dropping class representations dynamically (class-wise skipping), or (3) injecting skip connections from intermediate representations for efficient adaptation (Wu et al., 2024, Li et al., 2022, Pathak et al., 18 Jul 2025).

2. Methodological Realizations

2.1 Layer-wise Skipping (LSkip)

LSkip statically skips the first $\omega$ layers of the encoder(s), freezing their weights and caching their outputs. Only the remaining $N-\omega$ layers are fine-tuned. The forward pass for an input $x_i$ is:

• $v_i^\omega = E_V[1:\omega](x_i)$ (cache for image encoder)
• $\tilde v_i^N = E_V[\omega+1:N](v_i^\omega)$ (fine-tune only these layers) Analogous steps hold for text prompt encoders. Gradients flow only through the unskipped layers, formally via a binary skip indicator $I_l$, ensuring that parameters for $l \leq \omega$ remain fixed (Wu et al., 2024).

2.2 Class-wise Skipping (CSkip)

CSkip dynamically prunes the set of class tokens during loss computation for a given sample. For each image, class tokens are ranked by similarity, and only the top-$r \cdot M$ (with $r \in (0,1]$) are deterministically kept; the remainder are sampled with exponentially decaying probability controlled by $\lambda$:

• $p_j = 1$ if $o_j \leq rM$, else $p_j = \exp[-\lambda(o_j - rM)]$ Reducing $r$ improves speed and memory but eventually degrades accuracy. Typical robust ranges are $r \in [0.4, 0.7]$, $\lambda \sim 0.3$ (Wu et al., 2024).

2.3 Skip-Connection Based Fine-Tuning

SCL FT as in (Li et al., 2022) introduces a learnable skip branch from an intermediate (“$\ell$th”) layer to the penultimate feature:

• Let $z_i = \phi_1(x_i)$ (output of the $\ell$th layer for $x_i$)
• $h_i = \phi_2(z_i)$ (penultimate layer output)
• $h'_i = h_i + P z_i$ (where $P$ is a small projection, typically a $1 \times 1$ conv or linear) Only $P$, $\theta_\ell$ (the $\ell$th layer's weights), and the classifier $W, b$ are updated; the rest of the network is frozen.

2.4 Long Skip Connections in Transformers

Block-level skip adaptation, such as Solo Connection (Pathak et al., 18 Jul 2025), augments LLMs by introducing sparse, low-rank trainable maps between non-adjacent decoder blocks, leveraging homotopy-inspired gating for smooth adaptation. The main architecture introduces trainable scalars and vectors ($\alpha_t$, $v$) interpolating between zero and the new task signal, while all original block parameters remain frozen.

3. Mathematical Formulation and Complexity

3.1 Complexity Analysis

Combining LSkip and CSkip:

$$\text{Time}_\text{SkipTuning} \approx (N-\omega) \cdot (C_V + rM C_T) \ \text{Mem}_\text{SkipTuning} \approx (N-\omega) \cdot (A_V + rM A_T)$$

The speedup factor, assuming $N=12$, $\omega=6$, $M=100$, $r=0.5$, and $C_T \sim C_V$, is $\approx 4\times$, in line with 4–15$\times$ empirical accelerations (Wu et al., 2024). In skip-based PEFT methods, trainable parameter counts are reduced by over 90% compared to full FT, e.g., SCL FT uses 6–8% in vision backbones and Solo Connection uses less than 1% in GPT-2 Medium (Li et al., 2022, Pathak et al., 18 Jul 2025).

3.2 Neural Collapse Alignment

For classification, the SCL FT objective can optionally add a regularizer $R_{NC}$ driving within-class feature variance (Σ_W) to zero and between-class distances (Σ_B) high via canonical Neural Collapse-1 metric:

$$NC_1 = \frac{1}{K}\mathrm{Tr}[\Sigma_W (\Sigma_B)^+]$$

Minimizing $NC_1$ aligns learned features with the classical simplex ETF geometry, improving generalization and stability, especially under few-shot regimes (Li et al., 2022).

4. Empirical Performance and Benchmarks

Skip Connection Layer Fine-Tuning demonstrates compelling empirical gains:

Method	Base ACC	New ACC	Harmonic ACC	Time (s)	Mem (MB)
Full FT	84.99	74.26	79.27	1002	1846
LoRA	80.53	74.73	77.52	910	1580
CLIP-Adapter	74.48	73.81	74.14	888	1784
SkipTuning (SCL FT)	85.04	77.53	81.11	239	404

(Wu et al., 2024)

• On cross-dataset and domain generalization, SkipTuning outperforms adaptive prompt-tuning methods (CoOp, PromptSRC) on target accuracy, with a 10–50$\times$ reduction in compute and memory.
• In image classification transfer, SCL FT consistently matches or exceeds full FT, especially in fine-grained and low-data regimes. For example, ResNet18 pre-trained on ImageNet achieves: SCL FT (8.27% params tuned) $\to$ 93.11% (CIFAR-10); full FT (100% params) $\to$ 92.11% (Li et al., 2022).
• In NLG with GPT-2, Solo Connection slightly outperforms LoRA despite 25–60% fewer parameters and achieves near-state-of-the-art scores, using $\sim$0.07% of full FT parameter count (Pathak et al., 18 Jul 2025).

5. Hyperparameters, Ablations, and Implementation

• Layer skip depth $\omega$: Optimal at $\omega = N/2$; skipping too many layers degrades accuracy (Wu et al., 2024).
• Class keep ratio $r$: Robust in $[0.4, 0.7]$; default $0.5$.
• Decay $\lambda$ (CSkip): Default $0.3$.
• Projection $P$: Single-layer suffices; increasing depth gives diminishing returns (Li et al., 2022).
• Learning rates: Classifier/skip branch $0.1$; adapted backbone $0.01 - 0.05$.
• Shared modules in Solo Connection: Shared encoder/decoder reduce redundancy; optimal skip span is between adjacent blocks (Pathak et al., 18 Jul 2025).

Implementation recommendations include caching shallow features in FP16, index-selecting only surviving classes per batch, and progressive adjustment of layer skip and class-drop parameters to match GPU constraints (Wu et al., 2024).

6. Applicability, Trade-offs, and Limitations

SCL FT is most effective on classification and matching tasks in vision and language, significantly reducing overfitting in scarce-data transfer. Key limitations:

• Aggressive skipping ($\omega \to N$, $r \to 0$) degrades performance.
• For dense prediction (detection, segmentation), skipping early layers can harm spatial fidelity (Wu et al., 2024).
• Benefits from class-dropping diminish as $M \to 1$ or for negative mining tasks.
• Effectiveness depends on diversity of features in the pre-trained backbone; extremely different downstream domains may require deeper adaptation (Li et al., 2022).
• In block-level skip adaptation, activation storage for long skips slightly increases memory; skip span must be tuned for very deep architectures (Pathak et al., 18 Jul 2025).

7. Theoretical Significance and Future Directions

SCL FT operationalizes the insight that transferability is maximized—and overfitting is minimized—by restricting fine-tuning to high-salience segments of the network, enforced either structurally (skip/freeze) or parametrically (skip-connection branch), supplying “just enough” capacity for neural collapse and class alignment (Li et al., 2022). Homotopy-inspired gating provides additional regularization and stability in very deep transformer architectures (Pathak et al., 18 Jul 2025).

Future directions include adaptive layer-skipping per instance, integration with dynamic class-mining schemes, and exploration of multi-branch or hierarchical skip structures for generative or dense-prediction models. Empirical evidence suggests that mature pre-trained representations enable SCL FT to serve as the foundation for the next generation of parameter-efficient transfer mechanisms across vision, language, and multi-modal domains (Wu et al., 2024, Li et al., 2022, Pathak et al., 18 Jul 2025).