Median-of-Means Tournament Approach

• The median-of-means tournament approach is a robust statistical framework that partitions data into blocks and aggregates medians to mitigate outlier effects and heavy-tailed distributions.
• It achieves minimax optimal risk bounds with exponential concentration by leveraging finite moment conditions and robust pairwise comparison techniques.
• The method is adaptable to high-dimensional, regularized, and non-Euclidean settings, extending its practical utility to pairwise losses and robust machine learning applications.

The median-of-means (MOM) tournament approach is a robust statistical learning and estimation framework designed to provide optimal accuracy and confidence under minimal moment conditions, particularly in the presence of heavy-tailed data and outliers. By leveraging random data partitioning and robust estimation via medians of means, the MOM tournament overcomes the fragility of classical empirical risk minimization (ERM), attaining minimax optimal rates and high-probability concentration bounds for excess risk. The method admits direct generalization to regularized and high-dimensional settings, to pairwise and U-statistic losses, and to estimation in general metric and non-positively curved spaces.

1. Foundations and Motivation

Classical ERM procedures suffer from significant vulnerabilities under heavy-tailed distributions or contamination: their confidence degrades polynomially rather than exponentially with increasing confidence level, and their concentration rates become sub-optimal or unreliable (Lugosi et al., 2016). MOM estimators, especially when embedded in a tournament or elimination framework, provably retain sub-Gaussian concentration under only finite $L_q$ moment conditions, thereby addressing these deficiencies.

The MOM tournament approach partitions the data into several blocks, computes empirical means per block, and then aggregates statistics (risks, losses, distances) via medians, rather than means. By orchestrating pairwise risk comparisons across blocks, and eliminating poorly performing candidates through multiple "tournament" rounds, the MOM tournament ensures robustness even when an $O(1)$ fraction of the blocks is arbitrarily corrupted (Lugosi et al., 2017, Yun et al., 2022).

2. Core Algorithmic Structure

The canonical MOM tournament for regression or statistical learning consists of a structured multi-phase procedure (Lugosi et al., 2016, Lugosi et al., 2017, Yun et al., 2022):

1. Partitioning: Split the sample into $n$ blocks (typically $n \sim \log(1/\delta)$ for confidence $1-\delta$), each of equal or nearly equal size.
2. Distance Oracle: For each candidate pair $(f,h)$ (with $f,h$ from the candidate function class $F$), compute blockwise means of risk differences or distances and aggregate via medians. Abandon matches where empirical separations are below a threshold.
3. Tournament Elimination: For each allowed pair, compare robust blockwise excess risks. Declare winners based on majority of blocks. Surviving candidates are those that do not lose any match.
4. Final Selection/Champions League: Further refine among survivors using finer-grained blockwise statistics and select any function that wins all remaining duels.

This median-of-means elimination strategy can be re-cast as an optimization: minimize the maximal median-of-means block excess risk over all possible opponents, i.e., $$\phi(f) = \max_{g\in F} \operatorname{Med}_j \{ \mathbb{B}_{f,g}(j) \}$$, and select $\hat f = \arg\min_{f \in F} \phi(f)$ (Lugosi et al., 2017).

3. Theoretical Guarantees and Optimality

The MOM tournament guarantees optimal minimax rates for excess risk and estimator error, with high (exponential) confidence, under weak moment assumptions. For regression or prediction in $\mathbb{R}^d$ under quadratic loss, and assuming only finite $q$-th moments $(q > 2)$, the procedure yields, with probability at least $1 - \exp(-cn \min\{1, (r/\sigma)^2\})$, an estimator $\hat f$ with

$$\|\hat f - f^*\|_{L_2} \leq C r, \quad R(\hat f) - R(f^*) \leq C r^2,$$

where $r$ is the critical radius determined by localized complexity of $F$ and the noise level $\sigma$ (Lugosi et al., 2016, Lugosi et al., 2017).

In regularized settings (tournament LASSO, SLOPE), analogous rates are achieved for both $\ell_2$ and $\ell_1$ (or SLOPE) norm errors, with the failure probability decaying exponentially in $n$ (Lugosi et al., 2017, Kwon et al., 2018). For instance, in sparse linear regression,

$$\|\hat t - t_0\|_2 \lesssim \sigma \sqrt{ \frac{s \log(ed/s)}{N} }, \quad R(\hat t) - R(t_0) \lesssim r^2$$

with confidence at least $1 - 2\exp( -c N \min\{1, \lambda^2/\sigma^2 \} )$, under heavy-tailed noise (Lugosi et al., 2017).

For general metric spaces with non-positive curvature, the geometric median-of-means tournament replaces empirical Fréchet means and achieves exponential concentration for the estimated mean (e.g., GMOM estimator) under only a second-moment assumption: $$\Pr \bigl( d(\hat\mu_{\mathrm{GMOM}}, \mu^*) > \varepsilon \bigr) \leq 2 \exp\big( -c n \varepsilon^2 \big)$$ for spaces such as Hadamard, 2-Wasserstein, or SPD matrix spaces (Yun et al., 2022).

4. Generalizations: Regularization, Ensembles, and Pairwise Losses

The MOM tournament framework naturally incorporates regularization by including penalty terms (e.g., $\lambda \|f\|_{\mathrm{reg}}$) in pairwise blockwise risk comparisons. Hierarchies of nested candidate classes, parameterized by complexity or sparsity, allow automatic model selection and oracle inequalities for adaptivity (Lugosi et al., 2017, Kwon et al., 2018).

In ensemble and model selection scenarios, the MOM tournament robustly compares candidate models or hyperparameter configurations via blockwise medians of empirical risks, orchestrated in elimination brackets. This methodology provides a robust alternative to cross-validation, consistently selecting viable estimators in the presence of contaminated or heavy-tailed data and automatically identifying clean, informative subsamples for final training (Kwon et al., 2018).

Extension to pairwise/U-statistic losses is achieved by constructing blockwise, median-of-U-statistic estimators for pairwise risk. The theoretical performance guarantees—exponential concentration in excess risk—are preserved when blocks are constructed either via partition or random sampling without replacement (SWoR), enabling robust metric learning, pairwise ranking, and similar tasks (Laforgue et al., 2022).

5. Empirical and Computational Considerations

Empirical studies confirm that MOM tournament-based estimators remain robust under substantial contamination or heavy-tailed distributions, where standard ERM or regularized methods fail catastrophically (e.g., cross-validation collapses under a single hard outlier) (Kwon et al., 2018). Tournament LASSO and SLOPE, for example, attain oracle accuracy and properly filter out harmful outlying blocks (Lugosi et al., 2017, Kwon et al., 2018).

Sample complexity matches minimax lower bounds: for excess risk $O(r^2)$ at confidence $1-\delta$, the total sample size is $$N \asymp \mathrm{Complexity}(F)/r^2 + O(\log(1/\delta))$$ (Lugosi et al., 2017). Computationally, naïve implementations require $O(|F|^2 n)$ block comparisons for finite $F$; for infinite $F$ or high-dimensional settings, practical implementations rely on discretization (nets), approximate nearest-neighbor selection, or convex optimization heuristics. For certain classes, e.g., convex classes with appropriate geometry, coordinate descent and blockwise optimization are employed, but worst-case polynomial-time guarantees remain largely open (Lugosi et al., 2017, Lugosi et al., 2017, Kwon et al., 2018).

6. Robustness: Outliers and Heavy Tails

The central robustness property of the MOM tournament derives from the median's breakdown point: as long as less than half of the blocks are contaminated, overall estimator concentration is unaffected. This extends to blockwise U-statistics and regularized estimators, enabling robust learning in extreme contamination models (Lugosi et al., 2017, Laforgue et al., 2022).

This mechanism remains effective across settings:

• Heavy-tailed input and output distributions ($L_q$–$L_2$ norm equivalence, $q > 2$; only finite second moments for geometric-MOM).
• Outlier models: Arbitrary contamination of up to a fraction of blocks does not impact the exponential tails of the estimator's error.

7. Extensions to General Metric Spaces and Non-Euclidean Settings

The MOM tournament principle generalizes seamlessly to non-Euclidean and non-vector spaces via the geometric median-of-means framework (Yun et al., 2022). By employing blockwise empirical risk comparisons under suitable contrast functions (e.g., $d^2$), and leveraging metric-geometric inequalities (quadruple/CN and variance-type), one achieves exponential concentration for Fréchet mean estimation in non-positively curved spaces under only a second-moment condition.

Applications span:

• Hyperbolic spaces
• Symmetric positive definite matrix manifolds
• Wasserstein space

This robustification is significant due to the failure of naive empirical means (e.g., empirical Fréchet means) to achieve more than polynomial confidence without strong tail assumptions.

• "Risk minimization by median-of-means tournaments" (Lugosi et al., 2016)
• "Regularization, sparse recovery, and median-of-means tournaments" (Lugosi et al., 2017)
• "A MOM-based ensemble method for robustness, subsampling and hyperparameter tuning" (Kwon et al., 2018)
• "A remark on 'Robust machine learning by median-of-means'" (Lugosi et al., 2017)
• "On Medians of (Randomized) Pairwise Means" (Laforgue et al., 2022)
• "Exponential Concentration for Geometric-Median-of-Means in Non-Positive Curvature Spaces" (Yun et al., 2022)