SortNet: Neural Sorting Frameworks

• SortNet is a collection of neural architectures that integrate explicit or implicit sorting operations to enable modularity, learned ranking, and certified robustness.
• Empirical evaluations demonstrate that SortNet achieves high accuracy and efficiency across multiple applications, including modular DNNs, top-K pooling for point clouds, and robust comparators.
• The framework supports resource-adaptive inference through sub-model selection and enhances performance in adversarial settings via guaranteed Lipschitz properties.

SortNet refers to a set of neural network architectures and algorithmic frameworks, originating independently in multiple research streams, that integrate sorting operations or score-driven selection within deep learning. The principal instances of SortNet—distinguished by their context and technical formulation—include: (1) a modular DNN training and deployment framework for dynamic accuracy-compute tradeoffs (Valipour et al., 2023); (2) a learned, permutation-invariant local feature extractor for point clouds (Engel et al., 2020); (3) a neural comparator-based learning-to-rank approach (Rigutini et al., 2023); and (4) a Lipschitz neural network architecture achieving certifiable robustness via order-statistics (Zhang et al., 2022). Though differing in technical specifics and application domains, each SortNet instance leverages the concept of “sorting” either explicitly (by manipulating rank orderings) or implicitly (by exploiting ordered information for neural computation or resource allocation).

1. Modular Deep Neural Network Training with SortedNet (Valipour et al., 2023)

SortedNet introduces a unified, scalable methodology for training a single deep neural network family in which all possible sub-models, defined by truncation across architectural dimensions (e.g., depth, width, attention heads), share the master parameter tensor. Each sub-model corresponds to selecting the first $b_j$ units along each of $K$ dimensions $D=\{\mathrm{Dim}_1,\ldots,\mathrm{Dim}_K\}$. Formally, given the parameter set $\theta^{(n)}$ of the full network, a sub-model at iteration $t$ is

$$\theta_t^\star = \bigcap_{j=1}^K \theta_{\mathrm{Dim}_j\downarrow b_j^t}(n)$$

with each $b_j^t \sim P_{B_j}$ a discrete distribution over allowed indices.

During training, random sub-models are sampled per-iteration, and standard loss (e.g., cross-entropy) is accumulated—either (a) over the sampled sub-model alone or (b) over all nested sub-models within it—using gradient accumulation across $g_{\mathrm{acc}}$ draws before each optimizer step. The total parameter storage remains $O(|\theta|)$, enabling the family to scale to hundreds of sub-models with a single checkpoint.

Inference and deployment are search-free: for a resource budget (FLOPs, latency), the largest feasible sub-model is selected by picking the largest prefix $b_j$ indices within the constraint, as the ordering is monotonic in both compute and accuracy. Experiments demonstrate that SortedNet can concurrently train up to 160 sub-models for MobileNetV2, with each achieving at least 96% of full-model accuracy. For LLMs (LLaMA-13B), SortedNet enables self-speculative decoding, providing up to 1.63$\times$ speed-up with only 1–2% accuracy drop. Gradient accumulation is critical for convergence; increasing $g_{\mathrm{acc}}$ from 1 to 4 on CIFAR-10 improved mean sub-model accuracy by over 3%. Compared to prior approaches (e.g., OFA, Slimmable, DynaBERT), SortedNet is architecture-agnostic, multi-dimensional, and does not require costly neural architecture search or distillation (Valipour et al., 2023).

2. SortNet as Learned Top-K Pooling in Point Transformers (Engel et al., 2020)

Within the Point Transformer architecture, SortNet replaces traditional symmetric set pooling with a learnable, permutation-invariant Top-$K$ selection mechanism for point cloud feature extraction. Given $N$ input points $P = \{p_i\}$ and corresponding latent features $X \in \mathbb{R}^{N\times d_m}$, SortNet operates as follows:

1. Self-Attention: Compute contextual features $$X' = \mathrm{LayerNorm}[X+\mathrm{MultiHead}(X,X,X)]$$.
2. Score Regression: Map $X'$ via an MLP to obtain scores $s \in \mathbb{R}^N$, $s_i = \mathrm{MLP}_\mathrm{scores}(x_i')$.
3. Top-K Selection and Sorting: Select indices with the $K$ largest $s_i$, order them descendingly.
4. Local Neighborhood Aggregation: For each selected $p_{i^j}$, aggregate features from its spatial neighborhood.
5. Formation of Output Tensor: Concatenate $p_{i^j}$, $s^{(j)}$, and the local aggregation to form an ordered list $F^L \in \mathbb{R}^{K\times d_m}$.

Multiple independent SortNets can be run in parallel, their concatenated outputs forming a $(M\cdot K)\times d_m$ tensor. SortNet ensures permutation invariance by virtue of all operations before sorting being permutation equivariant, and the final sorted order depending only on content, not input order. Ablations demonstrate that SortNet-selected points yield significantly higher classification accuracy (~83%) than random or FPS selection (60–74%) on ModelNet40, and that the mechanism supports high robustness to rotations and spatial permutations (Engel et al., 2020).

3. Neural Comparator-Based Learning-to-Rank with SortNet (Rigutini et al., 2023)

This instance of SortNet formalizes learning-to-rank as learning a symmetric neural comparator $f: \mathbb{R}^d\times\mathbb{R}^d \to \{0,1\}$ from pairwise preferences $(x_i, x_j, t_{ij})$, where $t_{ij} \in \{\succ, \prec\}$ indicates which object should rank higher. The architecture consists of:

• Input: Concatenated pair $[x; y] \in \mathbb{R}^{2d}$.
• Hidden Layer: $H$ pairs of neurons $(i, i')$, logistic activation with enforced weight-sharing:
  • $v_{x_k,i'}, v_{y_k,i}$ obey $v_{x_k,i'}=v_{y_k,i}$
  • Output units also weight-share and output $N_\succ(x,y) = N_\prec(y,x)$.

A universal approximation theorem ensures that any symmetric two-output function can be approximated by the weight-sharing SortNet comparator. The training algorithm is incremental: each iteration grows the training set by incorporating the most informative mis-ranked pairs. This avoids quadratic scaling in the number of training pairs.

After training, the learned comparator is used as the comparison function in any standard $O(n\log n)$ sorting algorithm (e.g., mergesort). If transitivity is violated (a consequence of neural ranking not enforcing total orders), sorting stability may vary slightly with input shuffles. On the LETOR benchmarks (TD2003, TD2004), SortNet achieves MAP/NDCG comparable to or exceeding classic learning-to-rank baselines such as RankSVM and ListNet, especially on TD2004 (Rigutini et al., 2023).

4. Lipschitz-Bounded Neural Networks with SortNet Layers (Zhang et al., 2022)

SortNet is presented as a neural network architecture composed of “Sort neurons” engineered to be 1–Lipschitz with respect to $\ell_\infty$ norm, thus guaranteeing certified robustness against adversarial perturbations:

• Sort Neuron: $y = w^\top \mathrm{sort}(\sigma(x + b))$ where $\|w\|_1 \leq 1$ and $\sigma$ is 1–Lipschitz (e.g., $|t|$). The sorting operation $\mathrm{sort}(\cdot)$ fully orders the vector.
• Layer Stacking: Each layer applies such neurons, ensuring overall 1–Lipschitzness by induction.
• Certified Radius: For classification, the certified robustness radius $r_{\mathrm{cert}} = \tfrac{1}{2}\text{margin}(x)$, computable from the margin of outputs.

Training utilizes a stochastic dropout-max approximation to the sorted weighting, maintaining unbiasedness and tractability. Full-sort is necessary: GroupSort and $\ell_\infty$-distance nets are nested special cases but exhibit reduced expressivity for Boolean function representation or require greater depth. Empirically, SortNet attains state-of-the-art deterministic $\ell_\infty$ certified robustness on MNIST, CIFAR-10, TinyImageNet, and ImageNet 64x64 at substantially reduced training and certification cost relative to IBP methods, with certified accuracy on MNIST at $\epsilon=0.1$ of 98.14% (SortNet) versus 97.73% ($\ell_\infty$-distance nets) and much faster runtime (Zhang et al., 2022).

5. Commonalities and Distinctions Across SortNet Variants

While named identically, each SortNet variant targets a unique technical challenge: model subspace modularity (Valipour et al., 2023), permutation-invariant geometric feature learning (Engel et al., 2020), universal comparator learning for ranking (Rigutini et al., 2023), and expressive Lipschitz networks (Zhang et al., 2022). The linking thread is leveraging sorting—either of indices, scores, or activations—to structure computation, promote invariance, or guarantee theoretical properties.

Notably, the architectures in (Zhang et al., 2022) and (Engel et al., 2020) both exploit order statistics (sort, Top-$K$) to overcome expressivity or invariance bottlenecks in prior neural designs. Weight-sharing and symmetry properties in the comparator-based SortNet allow universal approximation in learning-to-rank. SortedNet for modular DNNs departs from the others by treating sorting as prefix-index truncation to form a nested sub-model family, but achieves resource-adaptive inference and storage advantages not seen in the other settings.

6. Empirical Performance and Application Scope

The reported implementations of SortNet demonstrate domain-leading or competitive results within their respective areas. SortedNet (Valipour et al., 2023) achieves ≥96% of full-model accuracy for up to 160 sub-models in MobileNetV2, outperforms Slimmable and nested Dyn approaches, and supports LLM decoding acceleration. SortNet in Point Transformer (Engel et al., 2020) advances state-of-the-art on ModelNet40 and ShapeNet part segmentation, with robust invariance to spatial and rotational transformations. Comparator-based SortNet (Rigutini et al., 2023) matches or surpasses RankSVM, ListNet, AdaRank, and RankBoost on LETOR datasets. Lipschitz SortNet (Zhang et al., 2022) achieves higher certified robustness and substantially reduced computational cost compared to previous Lipschitz or interval-bound propagation architectures.

7. Significance and Theoretical Insights

SortNet architectures inform critical aspects of deep learning model design:

• Parameter-sharing and modularity for scalable sub-model training (Valipour et al., 2023).
• Permutation-invariance without loss of local detail in geometric deep learning (Engel et al., 2020).
• Symmetric, universal comparator function construction for ranking problems (Rigutini et al., 2023).
• Exact Lipschitz constant control and robust Boolean function representation in adversarially robust learning (Zhang et al., 2022).

These properties position SortNet as a foundational technical pattern exploitable in modularity, invariance, sorting, and robustness contexts.

---

References:

• "SortedNet: A Scalable and Generalized Framework for Training Modular Deep Neural Networks" (Valipour et al., 2023)
• "Point Transformer" (Engel et al., 2020)
• "SortNet: Learning To Rank By a Neural-Based Sorting Algorithm" (Rigutini et al., 2023)
• "Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective" (Zhang et al., 2022)