Video Patch Pruning: Efficient Video Instance Segmentation via Early Token Reduction

Abstract: Vision Transformers (ViTs) have demonstrated state-ofthe-art performance in several benchmarks, yet their high computational costs hinders their practical deployment. Patch Pruning offers significant savings, but existing approaches restrict token reduction to deeper layers, leaving early-stage compression unexplored. This limits their potential for holistic efficiency. In this work, we present a novel Video Patch Pruning framework (VPP) that integrates temporal prior knowledge to enable efficient sparsity within early ViT layers. Our approach is motivated by the observation that prior features extracted from deeper layers exhibit strong foreground selectivity. Therefore we propose a fully differentiable module for temporal mapping to accurately select the most relevant patches in early network stages. Notably, the proposed method enables a patch reduction of up to 60% in dense prediction tasks, exceeding the capabilities of conventional image-based patch pruning, which typically operate around a 30% patch sparsity. VPP excels the high-sparsity regime, sustaining remarkable performance even when patch usage is reduced below 55%. Specifically, it preserves stable results with a maximal performance drop of 0.6% on the Youtube-VIS 2021 dataset.

• The paper introduces Video Patch Pruning (VPP), a framework that achieves early token reduction in ViTs using temporal mapping for improved video instance segmentation.
• It leverages a Mapping-Selective Module with Gumbel-Softmax to dynamically prune up to 40% of patches while preserving foreground integrity.
• Empirical results on Youtube-VIS datasets show VPP outperforms image-based methods with only a 0.6% drop in AP at a target patch keep ratio of 55%.

Video Patch Pruning: Efficient Video Instance Segmentation via Early Token Reduction

Introduction and Problem Setting

Recent advances in Vision Transformers (ViTs) have established them as a dominant backbone for video instance segmentation (VIS) due to their high modeling capacity for both spatial and temporal dependencies. However, the quadratic complexity of the self-attention mechanism with respect to the input size leads to significant memory and computational demands, limiting practical deployment, especially on edge devices. While patch pruning has been used as an efficiency mechanism, prior approaches focus on sparsification only in deeper layers, with early layers remaining fully dense due to insufficient foreground selectivity.

Video Patch Pruning: Concept and Architecture

This paper introduces Video Patch Pruning (VPP), an online patch pruning framework designed to overcome the aforementioned limitations by enabling effective early-stage token reduction for ViTs in VIS. VPP exploits temporal prior information by mapping high-level features from the previous frame onto the current frame, thus facilitating accurate early pruning and better foreground-background discrimination, even when feature selectivity is low in the initial layers.

The core component is the Mapping-Selective Module (Map-SM), a differentiable module that computes correspondences between patches of consecutive frames using both feature similarity and spatial locality constraints. With the assistance of a Gumbel-Softmax based association matrix, Map-SM propagates highly foreground-selective representations from deeper layers of the previous frame to earlier layers in the current frame. This enables the dynamic construction of binary pruning masks immediately after the first transformer block, removing up to 40% of input patches at this early stage while ensuring foreground integrity.

Figure 2: Visualization of the Mapping-Selective Module (Map-SM), which aligns patches across frames to facilitate early pruning based on temporal context.

To guarantee coverage of foreground instances and accommodate dynamic visual content, VPP introduces a sparsified background activation using controlled Gumbel noise, which allows for the detection of new object instances and mitigates collapse to sub-optimal solutions. Successive pruning stages in deeper layers use dynamic Selective Modules (SM) to adjust patch density adaptively.

Foreground Selectivity and the Limits of Image-based Pruning

Through an ablation on Foreground Selectivity Score (FGS), the study empirically confirms that early ViT layers display poor objectness awareness, leading to almost random pruning decisions if relying solely on per-frame features. Only features from layer 6 and deeper achieve robust discrimination between foreground and background patches. This property reveals the fundamental limitation of image-based patch pruning methods, motivating the need for the proposed temporal guidance.

Evaluation and Results

The framework is benchmarked on the Youtube-VIS 2019 and 2021 datasets using the ViT-Adapter backbone augmented by Mask2Former and ROVIS for segmentation and tracking, respectively. VPP enables an average patch keep ratio (PKR) as low as 40% with minimal AP loss—substantially outperforming all baselines, including SViT, DynamicViT, and TPS. At a goal PKR of 55%, VPP achieves only a 0.6% loss in AP, compared to ≥3% for image-based strategies. Notably, VPP produces denser activation maps for foreground regions and achieves higher Intersection-over-Instance (IoI) across object size distributions.

Figure 3: Comparison of pruning masks generated by image-based (SViT) and video-based (VPP) patch pruning at 60% patch reduction. VPP leverages temporal priors for improved foreground coverage.

Figure 1: Qualitative examples of the initial pruning masks from Map-SM: background regions are aggressively suppressed, while patches on object instances are retained.

At high sparsity (e.g., 40% PKR), VPP maintains robust AP, mAP_bounding-box, and mAP_segm, and achieves up to 8.6% higher IoI than strong image-based baselines, indicating superior maintenance of instance information under compression. Moreover, FLOP analysis and throughput (FPS) measurements establish that VPP substantially reduces computation relative to dense and image-only approaches, with improved wall-clock inference rates at iso-accuracy.

Figure 6: Layerwise patch activation density: VPP achieves early-stage pruning while maintaining sufficient density in deep layers for effective representation, as opposed to aggressive late-stage compression in image-based methods.

Adaptation in Dynamic/Discontinuous Video

The framework demonstrates fast mask adaptation following abrupt scene changes, recovering correct foreground coverage within a single frame after a context reset, further validating the robustness and practical efficiency of the Map-SM temporal mapping mechanism.

Figure 8: Mask adaptation after a scene switch—foreground patches are reallocated efficiently by VPP immediately after scene transitions, demonstrating temporal awareness.

Ablation: Module Placement and Dynamic Sparsification

Module placement studies highlight the necessity of introducing patch pruning after, not before, the first block, and the value of staged dynamic pruning in deeper blocks to balance representational capacity against efficiency. VPP also adaptively scales the mask resolution according to object size, maximizing patch savings without losing critical information for small, information-rich instances.

Figure 7: Temporal evolution of pruning masks across a video sequence; VPP tracks moving objects by dynamically updating foreground patch allocation.

Practical and Theoretical Implications

VPP’s architecture- and task-agnostic pruning mechanism enables deployment of high-throughput, low-latency VIS models on limited-resource platforms without major accuracy trade-offs. The framework expands the landscape of viable neural compression strategies in dense prediction settings, making structured early-layer token sparsification practical for the first time. Theoretically, the results establish that effective foreground selection in ViT is fundamentally dependent on temporal context at early layers, refuting the assumption that purely spatial or per-frame metrics suffice.

Future Directions

Opportunities for further development include extending the Map-SM concept to more diverse temporal video understanding tasks, integrating adaptive learnable thresholds for target sparsity, and designing hardware accelerators that natively exploit the structured patch dropping patterns produced by VPP. Investigating the synergy with quantization and structured pruning in the context of end-to-end multitask ViTs is also warranted.

Conclusion

Video Patch Pruning (VPP) introduces a temporally informed, early-stage patch pruning mechanism for ViTs in video instance segmentation, yielding state-of-the-art computational efficiency at negligible accuracy cost. Strong empirical evidence on large-scale video benchmarks demonstrates the method’s ability to surpass the sparsity limits of competing image-based approaches while delivering robust foreground coverage. This work provides a significant advancement for efficient dense prediction in video, with direct implications for low-latency vision systems and edge deployment.

Figure 4: Example VPP pruning masks on diverse videos: temporal consistency in patch allocation enables accurate, efficient segmentation and tracking across time.