Prefill-Only Pruning (POP)

• Prefill-Only Pruning (POP) is a stage-aware inference strategy that prunes transformer operations during the prefill phase to reduce compute cost while preserving decode accuracy.
• It employs techniques such as block/layer removal, N:M activation sparsity, and online structural pruning, achieving up to 33% prefill reduction with minimal accuracy impact.
• By exploiting the asymmetry between parallel prefill and sequential decode stages, POP accelerates inference without retraining, making it valuable for diverse deep learning applications.

Prefill-Only Pruning (POP) is a class of stage-aware inference strategies in deep learning that prunes model structure or activations exclusively during the computationally intensive prefill phase—where context tokens are processed in parallel—while preserving the full model for the decode phase. POP encompasses a spectrum of algorithmic approaches spanning block/layer removal, activation sparsity, channel pruning, and online context-adaptive inference, all unified by the principle of targeting the prefill stage to achieve substantial reductions in inference latency and compute cost, typically without retraining the model and with controlled accuracy loss.

1. Conceptual Framework and Motivation

POP exploits an inherent asymmetry in transformer-based generative models between the prefill and decode stages. During prefill, the entire prompt is processed in parallel to populate the key–value (KV) cache; during decode, subsequent tokens are generated one at a time via autoregressive steps. Analysis of transformer layer importance via virtual (residual) gates and Fisher–Taylor sensitivity reveals that deep layers are critical for decode (next-token prediction) but largely redundant in prefill (context encoding). Classical stage-agnostic pruning—removing the same substructure from both phases—can collapse accuracy for open-ended generation due to excessive loss of representation power in decode. POP circumvents this by focusing compute reduction solely on prefill, where the multiplicity of input tokens provides higher amortization of any approximation error and yields greater theoretical and observed speedup per unit structure pruned (He et al., 3 Feb 2026).

2. Core Methodologies

POP admits multiple realizations, each leveraging different axes of the model for prefill-specific reduction:

1. Block/Layers Pruning via Importance Estimation: Methods such as PD-disaggregation pruning (Zhang et al., 29 Aug 2025) and virtual gate analysis (He et al., 3 Feb 2026) rank layers by context sensitivity or redundancy, then suppress computation in the least important blocks for prefill only. For example, pruning the final third of transformer layers during prefill—identified via second-order Taylor/Fisher approximations of loss increase—preserves accuracy in decode while reducing prefill computational cost by up to 33%.
2. N:M Activation Sparsity: Amber Pruner (An et al., 4 Aug 2025) implements structured N:M activation sparsity by generating a binary mask on activations in each linear projection during prefill. A weight-aware scoring function (Robust-Norm) selects the maximally salient entries within each group, while sensitive layers are detected and exempt from sparsification via a relative perturbation threshold to protect accuracy.
3. Fine-grained, Online Structural Pruning: Partition-guided Online Pruning (Chen et al., 6 Feb 2026) partitions each channel (e.g., in MLPs) into retained, candidate, and pruned regions based on prefill-aggregated importance scores. Only candidate channels are dynamically mask-selected at each decode step, ensuring context-adaptive accuracy with bounded overhead.
4. Pre-training Mask Generation: Fisher–Taylor Sensitivity (FTS) (Navarrete et al., 17 Feb 2025) allows one-shot pruning before any training, using a criterion combining gradient and empirical Fisher (curvature) at initialization. While historically positioned as "pruning before training" (PBT), its selective mask generation over initial weights can conceptually realize POP in architectures where prefill computations dominate downstream cost.

The algorithmic pipelines universally involve three stages: (1) importance computation on a calibration dataset or prompt-induced activations; (2) mask or schedule selection for prunable substructures; (3) enforced pruning during prefill, reverting to full computation in decode for open-ended generation or downstream tasks.

3. Mathematical Formulation and Algorithmic Details

For block/layer pruning, let $L$ denote the total layers, and $\mathcal{B}\subset\{1, \ldots, L\}$ the subset of pruned layers. Importance scores per layer $l$ can be estimated by context sensitivity via:

$$I_l^{\text{prefill}} \approx \mathbb{E}\Bigl[\left(\frac{\partial \mathcal{L}}{\partial g_l^{\text{prefill}}}\right)^2\Bigr]$$

where $g_l$ is a gate on residual or branch output, evaluated on calibration data (He et al., 3 Feb 2026). The final schedule typically prunes the last $pL$ layers ($p\sim \frac13$) during prefill, skipping none for decode.

For N:M activation sparsity, the activation tensor $X \in \mathbb{R}^{B \times L \times D}$ is masked by $M \in \{0,1\}^{B \times L \times D}$, with:

$\sum_{i=1}^M M_{b, \ell, g \cdot M + i} = N$

per group $g$. Amber Pruner scales channel activations by weight-norm scores $f_j$ and selects the top-N per group, per position, at inference (An et al., 4 Aug 2025).

Partition-guided online pruning (Chen et al., 6 Feb 2026) uses:

$I_{i,k} = |W_{i,k}| \cdot \| X_{:,:,k} \|_2$

to compute channel importances, partitions output channels into retained/pruned/candidate based on quantiles, and then further prunes candidates at each autoregressive step with negligible extra FLOPs due to candidate set size being small ($\epsilon \ll r$ relative to overall pruning ratio $r$).

4. Empirical Results and Performance Trade-offs

Multiple studies report consistent prefill latency reductions, often 1.26–1.37×, for modest pruning ratios ($\sim$13–33%), along with negligible accuracy loss on standard zero-shot, few-shot, or generative tasks:

Method/Model	Prune Ratio	Prefill Speedup	Avg Accuracy Loss	Notable Results
POP (Llama-3.1-8B)	31.25%	1.36×	<2%	Matches Wanda, much higher than SliceGPT, ShortGPT (He et al., 3 Feb 2026)
Amber Pruner (8:16)	50%	1.3–1.6×	<1% (zero-shot)	Over 55% MAC savings, 0.5–1% generation loss (An et al., 4 Aug 2025)
POP-partitioned	20%	1.14×	—	2.85% MLP overhead at 20% (Chen et al., 6 Feb 2026)
PD-disaggregation	13.6%	1.26×	~2.5 pp	Also sees 5× bandwidth decrease with KV cache pruning (Zhang et al., 29 Aug 2025)

Ablation studies emphasize that deep-layer targeting, independent KV projection, and boundary handling are each essential; alternate shallow or interleaved pruning, or dropping boundary steps, severely compromises accuracy or open-ended generation quality. The N:M activation sparsity approach is particularly effective in linear projections—empirically, activation tensors from prefill stages contain significant near-zero "whiteness," increasing amenability to semi-structured sparsification.

5. Theoretical Analysis and Complexity Considerations

The prefill stage processes all $N$ prompt tokens in parallel, leading to complexity $O(NL d^2)$, where $L$ is the number of transformer layers and $d$ the hidden width. By pruning $pL$ layers only in prefill, POP achieves complexity:

$O\left(N (1-p) L d^2 \right)$

with decode cost untouched, and speedup increases with larger prompt lengths $N$. Errors introduced during prefill can compound linearly with $N$, motivating careful construction of pruning masks and hybrid techniques (e.g., independent KV projection and boundary processing).

Partition-guided online pruning reduces overall inference FLOPs by $(1-r)$, with a negligible overhead due to candidate masking ($<4\%$ of MLP FLOPs at typical $\epsilon$). It requires neither retraining nor offline calibration, enabling immediate deployment.

6. Extensions, Variants, and Hardware Implications

Recent frameworks extend POP to joint quantization (Outstanding-sparse (An et al., 4 Aug 2025)), MoE/VLM variants (He et al., 3 Feb 2026, Chen et al., 6 Feb 2026), token-aware cache bandwidth reduction (Zhang et al., 29 Aug 2025), and true online context-conditioned pruning (Chen et al., 6 Feb 2026). Integration with post-training quantization and programming of candidate-only online partitions further reduces memory bandwidth and achieves additional latency savings.

POP is hardware-friendly, requiring only minor changes in batched GEMM kernels. However, further throughput scaling may depend on native support for structured (N:M) sparsity, fused quantization, and top-k selection at the microarchitecture level. Speculatively, future LLM hardware could integrate per-lane top-N selection logic and efficient support for dynamic candidate masking—this suggests that the co-design of POP algorithms and specialized accelerators will play an increasingly significant role (An et al., 4 Aug 2025).

7. Limitations and Open Directions

POP does not reduce peak memory usage as the full model must be present for decode. It is most efficient in compute-bound scenarios (long context, high-resolution input). In settings with extreme domain shift, static prefill importance partitioning may underperform adaptive or dynamic recalibration. Future work includes extension of channel partitioning to attention heads, joint optimization of sparsity and quantization, device-specific kernel specialization, and stage-aware pruning for multi-stage, heterogeneous inference architectures (Chen et al., 6 Feb 2026).

In summary, Prefill-Only Pruning (POP) constitutes a suite of inference-time model reduction techniques centered on stage-specific, context-sensitive, and often training-free sparsification. By exploiting the structural redundancy of the prefill processing pathway while reserving model expressivity for decode, POP enables substantial reductions in computational cost and latency with minimal impact on downstream performance, substantiated across large language, vision-language, and mixture-of-experts models (An et al., 4 Aug 2025, Zhang et al., 29 Aug 2025, He et al., 3 Feb 2026, Chen et al., 6 Feb 2026).