KNN-Based Feature Distance Approach

• KNN-based feature distance approach is a technique that computes geometric dissimilarities between feature vectors to enable robust neighbor search in various learning tasks.
• It integrates classical metrics with adaptive and learned measures, enhancing performance in high-dimensional, noisy, or nonconvex environments.
• Practical strategies, including weighted Minkowski distances, autoencoder mappings, and deep metric learning, provide affine invariance and improved classification accuracy.

A KNN-based feature distance approach refers to a broad class of methodologies in which pairwise (or more generally, point-to-class or point-to-distribution) dissimilarities between feature vectors constitute the geometric basis for neighbor search and subsequent supervised, unsupervised, or semisupervised tasks. The choice and construction of the distance function, or feature mapping in which distances are computed, fundamentally determines the classifier’s statistical properties, robustness, interpretability, and efficacy under high-dimensional, nonconvex, or noisy regimes. Contemporary KNN-based feature distance methods encompass not only classical metrics but also robustified, adaptive, learned, and ensemble composite measures, often paired with explicit or implicit feature transformations to induce affine invariance, skewness adaptation, or local reliability guarantees.

1. Classical and Robust Distances: The Foundation

The original k-nearest neighbor (kNN) classifier employs a symmetric, positive-definite distance metric—typically Euclidean, Mahalanobis, Minkowski or member of the Lp family ($d(x, y) = (\sum_i |x_i - y_i|^p)^{1/p}$)—directly on observed feature vectors. However, this baseline metric is often inadequate in high-dimensional, noisy, or non-Gaussian settings due to its sensitivity to scale, outliers, and irrelevant/uninformative features.

To combat these limitations, robust alternatives have been proposed:

• Bagdistance via halfspace depth: For $x \in \mathbb{R}^p$, the halfspace (Tukey) depth is defined as

$$\mathrm{HD}(x;P) = \inf_{\|u\|=1} P\{y : u^\mathsf{T}y \ge u^\mathsf{T}x\}$$

The bagdistance,

$$\mathrm{bd}(x;P) = \begin{cases} 0,&x=\theta\ \frac{\|x-\theta\|}{\|x_B-\theta\|},&x\neq\theta \end{cases}$$

is affine-invariant, robust to outliers, and generalizes norm structure to permit sensitivity to distributional asymmetry (Hubert et al., 2015).

• Skew-adjusted projection depth (SPD): Outlyingness is measured by

$$\mathrm{SPD}(x;P) = \frac{1}{1 + \sup_{\|u\|=1} \mathrm{AO}_1(u^\mathsf{T}x;\,P_{u^\mathsf{T}Y})}$$

adapting for skewed classes.

Both distances can be used in a DistSpace transform, which maps points into a new feature vector of class-wise distances and applies classical kNN in this transformed space, yielding state-of-the-art robustness to skewness and affine transformations (Hubert et al., 2015).

2. Feature Weighting and Feature-wise Adaptivity

The curse of dimensionality and the problem of irrelevant features motivate KNN variants with explicit per-feature weighting. These approaches assign data-driven relevance weights to individual feature dimensions:

• Weighted Minkowski distance:

$$d_w(x, y) = \left( \sum_{i=1}^d w_i |x_i - y_i|^p \right)^{1/p}$$

with nonnegative weights $w_i$, typically normalized ($\sum w_i = d$). Weights $w_i$ are derived from a univariate "fitness" score $\delta_i$ for each feature, such as class-separation adjusted by within-class spread. The final weights interpolate between pure discriminative weighting ($x \in \mathbb{R}^p$0) and uniform ($x \in \mathbb{R}^p$1) (Mollah, 23 Oct 2025).

• Feature importance from ensembles: Out-of-bag error increases from a random forest yield per-feature importances, normalized via z-scores and used as weights in a weighted Euclidean metric (Bhardwaj et al., 2018).
• Univariate discriminability-based schemes: Other information-theoretic or statistical criteria (e.g., ANOVA, Gini, Fisher scores) can be substituted, provided they rank features by classification relevance (Mollah, 23 Oct 2025).

Weighted distances substantially improve accuracy, especially in high-dimensional or small-sample regimes where many features are irrelevant or noisy.

3. Nonlinear Feature Space Transformations and Metric Learning

Beyond shallow feature weighting, recent KNN-based methods employ nonlinear transformations or explicitly learned metrics:

• Autoencoder mapping: The AEkNN approach learns an undercomplete autoencoder (AE), yielding a low-dimensional, information-rich code $x \in \mathbb{R}^p$2, then computes distances in the code space. This reduces "hubness," increases discriminability, and improves computational efficiency (Pulgar et al., 2018).
• Deep metric learning: DNet-kNN initializes a deep encoder using stacked restricted Boltzmann machines (RBMs) and fine-tunes with a large-margin loss that enforces kNN decision boundaries in the code space. The objective directly minimizes margin violations among triplets $x \in \mathbb{R}^p$3 (in-class neighbor, impostor), resulting in improved clusterability and generalization, with state-of-the-art results on large vision benchmarks (0906.1814).
• Adaptive Nearest Neighbor (ANN) framework: Continuous, differentiable surrogates for the empirical KNN risk (via log-sum-exp and softmax approximations) enable direct metric learning. This general formalism subsumes LMNN, NCA, and pairwise-constraint schemes, yielding faster and often higher-accuracy solutions (Song, 2019).
• Ridge-regression KNN dissimilarity: A simple regression learns a transformation mapped only on labeled points to reduce the variance of the labeled set and minimize "spatial centrality," efficiently reducing hubness and matching the accuracy of established metric learners (Shigeto et al., 2018).

4. Affinity Measures and Distances for Nonstandard Data

Alternate feature distance approaches are tailored to the data's geometric or statistical structure:

• Coincidence similarity-derived dissimilarity ($x \in \mathbb{R}^p$4), robust for skewed densities:

$x \in \mathbb{R}^p$5

with dissimilarity $x \in \mathbb{R}^p$6 (Benatti et al., 2024). Scale-invariant and advantageous for right-skewed distributions.

• Dimensionality-Invariant Similarity Measure (DISM): Per-dimension, piecewise-bounded distance,

$x \in \mathbb{R}^p$7

yielding robustness to outlier features and consistent improvement in 1-NN accuracy (Hassanat, 2014, Prasath et al., 2017).

• Choquet-integral subset-weighted distance: Aggregates interactions among feature subsets using a monotone measure $x \in \mathbb{R}^p$8:

$x \in \mathbb{R}^p$9

granting invariance to duplicates, redundancy, and enabling higher-order (nonlinear) relationships (Theerens et al., 1 Apr 2025).

• Z-distance (reachable distance) for KNN: Penalizes interclass pairs via class centers,

$$\mathrm{HD}(x;P) = \inf_{\|u\|=1} P\{y : u^\mathsf{T}y \ge u^\mathsf{T}x\}$$0

ensuring that intraclass distances are always less than interclass, promoting superior separation (Zhang et al., 2021).

• Instance and neighborhood reliability: DW-KNN applies exponential distance weighting and multiplies by a precomputed reliability score (local label agreement) for each neighbor, improving stability and interpretability (Pathak et al., 28 Nov 2025).

5. Practical Algorithmic Strategies and Empirical Findings

Implementation of KNN-based feature distance methods requires careful algorithmic engineering, choice of hyperparameters, and attention to computational cost:

• Preprocessing: Feature-wise normalization is generally a prerequisite before weighted or non-Euclidean distances unless the method is intrinsically scale-invariant (Bhardwaj et al., 2018, Mollah, 23 Oct 2025).
• Feature selection: Fusion of multi-domain statistics (time, frequency, time-frequency) followed by robustness-scored selection can dramatically boost accuracy in domain applications (e.g., bearing fault diagnosis), leveraging thresholds on combined intra/interclass scatter and noise deviation metrics (Chaleshtori et al., 25 Sep 2025).
• Complexity analysis: Most plug-and-play weighted distances retain $$\mathrm{HD}(x;P) = \inf_{\|u\|=1} P\{y : u^\mathsf{T}y \ge u^\mathsf{T}x\}$$1 query cost with negligible overhead per distance. Metric learning procedures can incur $$\mathrm{HD}(x;P) = \inf_{\|u\|=1} P\{y : u^\mathsf{T}y \ge u^\mathsf{T}x\}$$2 or more but many methods offer closed-form or fast iterative solvers (Shigeto et al., 2018, Song, 2019).
• Empirical performance:
  • Robust depth/distance-transformed spaces (DistSpace + kNN) outperform standard kNN especially on skewed or nonconvex class shapes (Hubert et al., 2015).
  • Weighted KNN approaches deliver substantial gains in high-dimensional, small-sample domains such as gene expression classification (>10% absolute improvement) (Mollah, 23 Oct 2025, Chaleshtori et al., 25 Sep 2025).
  • Advanced distance measures (e.g., Hassanat, Lorentzian, Clark, DISM) rank highest in comprehensive empirical studies on diverse UCI datasets, consistently exhibiting strong resilience to feature-level noise (Prasath et al., 2017, Hassanat, 2014).

Distance/Method	Robustness/Key Setting	Empirical Outcome (context)
Bagdistance/DistSpace + KNN	Affine-invariant, robust	Best on skewed/nonconvex classes
Weighted Minkowski (W-KNN)	Feature weighting, any p	+10-13% acc. (high-d, gene expr.)
Coincidence dissimilarity	Scale/skew invariant	Beta index +0.2 vs Euclidean (skewed)
AEkNN autoencoder	Dim. reduction, code-space	11/14 datasets: +acc., +AUC
DISM / Hassanat	Bounded per-dim, outlier-rob.	Highest accuracy, noise resilience
DW-KNN	Double weighting (dist, val.)	0.8988 mean acc., lowest CV std

6. Specializations and Modern Extensions

Beyond classical and robust measures, specialized KNN-based feature distance methodologies target domain-specific or high-dimensional challenges:

• Community detection: KNN-defined medoid-shift stabilizes cluster boundaries and accelerates convergence, outperforming classic radius-based methods (Hou et al., 2023).
• Semi-supervised learning: Pseudo-labeling using balanced top-k KNN in an embedding space, leveraging cosine distance and confidence fusion, improves rare-class identification and early cycle stability (Botzer et al., 2023).
• O(1) k-NN distance estimation: Neural approaches (PivNet) predict all k-NN distances for a query using grid pivots and feedforward networks, attaining low error and microsecond-scale inference, suitable for large-scale proximity analytics (Amagata et al., 2022).

7. Guidelines and Theoretical Properties

The selection and construction of feature distances in KNN-based learning critically influences classifier properties:

• Affine invariance: Depth-based, projection-based, and some subset-weighted approaches provide invariance to nonsingular linear transformations (Hubert et al., 2015, Theerens et al., 1 Apr 2025).
• Robustness to outliers: Depth, bounded (DISM/Hassanat), and reliability-weighted (DW-KNN) approaches resist the distortive influence of spurious data (Hassanat, 2014, Pathak et al., 28 Nov 2025).
• Skewness adaptation: Bagdistance (halfspace) and skew-adjusted projection depth explicitly accommodate class-tail asymmetry.
• Dimensionality and redundancy: Choquet integration, feature importance weighting, and regression-mapped transformations mitigate the curse of dimensionality and avoid performance deterioration under duplicate/correlated features (Theerens et al., 1 Apr 2025, Mollah, 23 Oct 2025).
• Empirical risk consistency: Metric-learning frameworks with continuous surrogates provide direct optimization of KNN risk, admitting broader solution spaces and unifying earlier approaches (Song, 2019).

In practical terms, robust, bounded, or adaptively weighted distances (e.g., DISM, Hassanat, weighted Minkowski, bagdistance), especially when combined with feature selection or dimensionality reduction, yield superior and more reliable KNN performance across high-dimensional, noisy, and domain-specific learning scenarios (Hubert et al., 2015, Mollah, 23 Oct 2025, Prasath et al., 2017, Hassanat, 2014).