Hierarchical Geometric Pruning

• Hierarchical geometric pruning is a method that applies geometric and combinatorial criteria to systematically reduce network complexity.
• It leverages layer-wise optimization techniques to allocate sparsity while maintaining a high count of linear regions and model expressivity.
• In visual matching, hierarchical candidate pruning ranks feature tokens to significantly accelerate processing with minimal accuracy loss.

Hierarchical geometric pruning refers to a class of techniques that leverage the combinatorial and geometric structure of deep neural networks or candidate sets to guide the removal of parameters, activations, or tokens in a structured, layer-wise—or otherwise staged—fashion. These approaches exploit hierarchical organization and geometric criteria to maximize resource savings (such as memory or computation) while minimizing the loss of functional complexity or predictive accuracy. Two distinct research threads exemplify the concept: layer-wise geometric pruning in deep ReLU networks, and hierarchical candidate pruning in detector-free visual matching pipelines. Across both, the unifying principle is the strategic, geometry-informed selection of pruning schedules that preserve a task-relevant notion of expressivity.

1. Geometric Complexity in ReLU Networks

In deep networks with ReLU activations, every hidden neuron partitions its input space into two half-spaces (activated versus inactivated), rendering the overall function piecewise affine. The domain is divided into maximal connected linear regions where the activation pattern of all neurons is fixed, with each region corresponding to an affine map. The quantity of linear regions is a natural proxy for the network's piecewise linear complexity, intimately tied to its capacity to approximate complex decision boundaries. Empirically, as the number of linear regions decreases—particularly due to pruning—test accuracy often remains stable until a critical threshold, after which both region count and accuracy precipitously drop. This suggests that carefully designed pruning protocols must account for geometry to prevent catastrophic loss of expressivity (Cai et al., 2023).

2. Theoretical Foundations for Hierarchical Geometric Pruning

Cai et al. (2023) develop a theoretical upper bound on the number of linear regions remaining in a sparsified ReLU MLP by extending prior hyperplane arrangement results to the regime of random pruning. For each layer $l$ with width $n_l$ and density $p_l$ (the fraction of surviving weights), they define a recurrence for the expected number of regions $R(l,d)$ as a function over layers and input dimensionality:

• The recurrence accounts for the probability $P(k \mid R,C,S)$ that a pruned weight matrix has rank $k$, evaluated using binomial or random matrix models.
• Each rank-$k$ residual layer induces up to $\sum_{j=0}^{\min(k,d)} C(n_l, j)$ regions, forwarding reduced dimensions to the next layer.
• The cumulative upper bound $B(p_1, \ldots, p_L) = R(1, n_0)$ serves as a differentiable surrogate objective for allocating sparsity across layers under a fixed global parameter budget.

This theoretically grounded surrogate allows efficient outer-loop optimization schemes for hierarchical (layer-wise) density allocation, providing precise guidance on where pruning least impacts expressivity as measured by linear region count (Cai et al., 2023).

3. Pruning Strategies Guided by Geometry

With a closed-form upper bound on the expected linear regions, hierarchical geometric pruning employs optimization to set per-layer densities $(p_l)$ to maximize $B$ under a global density constraint. In practice, for two-layer MLPs, the protocol consists of:

• Evaluating extreme schedules (min/max for $p_1$ and corresponding $p_2$) and a uniform case ($p_1=p_2=p$).
• Quadratic interpolation to identify the maximizing $p_1$ in the feasible interval or grid search if the maximum lies at the boundary.
• Applying the schedule, pruning, and fine-tuning the network.

This approach consistently improves test accuracy by 1–3 percentage points relative to uniform pruning, particularly in architectures with varying fully connected layer sizes and moderate global densities (10–30%). Empirically, the actual number of linear regions (measured or estimated) also falls more slowly, confirming that the expressivity loss is mitigated (Cai et al., 2023).

4. Hierarchical Candidate Pruning in Detector-Free Matching

In local feature matching, the HCPM (Hierarchical Candidates Pruning for Efficient Detector-Free Matching) method demonstrates the application of hierarchical geometric pruning for computational acceleration. HCPM inserts a hierarchical pruning module between feature extraction and transformer-based matching in a LoFTR-style pipeline, conducting:

• Self-pruning: Each coarse feature token is ranked by an informativeness score computed by an MLP+sigmoid, and only the top-$k$ tokens (parameterized by a fixed ratio $\alpha=0.5$) are retained.
• Interactive-pruning: The retained tokens undergo $N_c=4$ rounds of self-cross attention, after which DICS modules use geometric co-visibility cues (learned with Gumbel-softmax for differentiability) to further suppress tokens unlikely to produce matches across views.

This sequential reduction results in quadratic complexity scaling reductions (by factors of $\alpha^2$ and then $\beta^2$ for successive stages), with empirical speedups exceeding 8-fold at the transformer stage, while match accuracy degrades by less than 1–2% AUC. Ground-truth depth or co-visibility masks supervise both pruning stages (Chen et al., 2024).

5. Mathematical and Algorithmic Formulation

The two principal instantiations of hierarchical geometric pruning adopt distinct but rigorous algorithmic frameworks:

Domain	Core Quantity Controlled	Pruning Mechanism
ReLU MLPs (Cai et al., 2023)	Expected max linear region count $R(l,d)$	Layerwise density allocation via $B$
Detector-free Matching (Chen et al., 2024)	Candidate set size (token count) at each stage	Self and interactive hierarchical stages

For ReLU networks, the recurrence for $R(l, d)$ incorporates combinatorial enumeration of active neurons and random matrix rank, while for feature matching, the pruning stages employ explicit ranking, hard masking, and attention masking with learned geometric criteria.

6. Implementation and Empirical Results

In neural architecture pruning (Cai et al., 2023), experiments across MLPs (widths 100–400), LeNet, and AlexNet on MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 demonstrate that hierarchical pruning schedules derived from the geometry-inspired bound yield systematic gains over uniform pruning. Gains are maximal in networks with highly imbalanced parameter distributions and at moderate sparsity. The method reliably preserves a higher count of linear regions per fixed parameter budget.

For HCPM (Chen et al., 2024), quantitative evaluation on HPatches (homography estimation) and MegaDepth-1500 (relative pose) reports speed-ups of 36–57% (homography) and 17–32% (pose) over LoFTR, with negligible accuracy loss. The hierarchical pruning enables runtime reductions mainly by eliminating tokens from homogeneous or uninformative image regions (such as sky or foliage) and focusing later attention on co-visible, geometrically consistent matches.

7. Assumptions, Limitations, and Theoretical Scope

Both lines of work make simplifying assumptions for tractability:

• For network pruning, the bound assumes uniform random survival probabilities for weights, though typical real-world pruning uses magnitude-based schemes. The approximation focuses on rank-based loss and does not account for stable neuron effects or detailed sign patterns. The geometric bound is an upper bound, using the classical hyperplane arrangement formula, not an exact enumeration.
• For matching pruning, informativeness and geometric consistency are learned via compact heads and cross-view consistency masks; the effectiveness depends upon the accurate learning of these criteria.

A plausible implication is that for highly structured or application-specific architectures, further refinements to these geometric proxies may yield even tighter integrity between reduced computational cost and preserved model complexity. Nevertheless, these schemes provide a highly efficient, theoretically motivated, and empirically validated paradigm for allocating sparsity or reducing candidate sets in hierarchical, geometry-aware fashion.