Friedman & Nemenyi Tests Overview

• The Friedman test is a nonparametric omnibus procedure that ranks and compares multiple treatments across datasets to test for performance equivalence.
• The Nemenyi mean-ranks test serves as a post-hoc method to compare all pairs following Friedman’s test but can yield paradoxical results due to pool-dependence.
• Alternatives like pairwise tests and S-plots offer pool-independent comparisons and controlled Type I error, enhancing inference clarity.

The Friedman and Nemenyi tests are core nonparametric methodologies for analyzing and comparing multiple treatments or algorithms under a randomized complete block design. Their primary application is the statistical comparison of several methods across multiple data sets, with particular relevance in areas such as machine learning, psychology, and medicine. After an initial omnibus hypothesis test (Friedman), post-hoc analyses such as the Nemenyi mean-ranks test are commonly employed to determine sources of significant differences. However, the dependence of Nemenyi post-hoc inferences on the entire set of treatments and accompanying paradoxes have recently prompted scrutiny and recommendations for alternative pairwise procedures.

1. The Friedman Test: Omnibus Nonparametric Comparison

The Friedman test is used to detect differences among $m$ algorithms (treatments) evaluated on $N$ datasets (blocks). For each dataset $j$, the outcomes $X_{ij}$ ($i=1,...,m$) are ranked, yielding $R_{ij}$; average ranks replace raw performance scores, with ties handled via average-ranking. Each algorithm’s sum of ranks is $R_i = \sum_{j=1}^N R_{ij}$, with mean rank $\overline{R}_i = R_i/N$. The null hypothesis $H_0$ asserts “all $m$ algorithms perform equivalently,” that is, $$X_{1j} \overset{d}{=} X_{2j} \overset{d}{=} \cdots \overset{d}{=} X_{mj}$$ for all $j$.

The Friedman statistic is

$$\chi^2_F = \frac{12N}{m(m+1)}\sum_{i=1}^m \overline{R}_i^2 - 3N(m+1).$$

Alternatively, in terms of $R_i$,

$$S = \frac{12}{N m (m+1)}\sum_{i=1}^m (R_i - N(m+1)/2)^2.$$

For large $N$, $\chi^2_F$ is approximately $\chi^2$-distributed with $m-1$ degrees of freedom. In practice, the test serves as a robust, nonparametric alternative to repeated-measures ANOVA for arbitrary, not necessarily normal, data (Benavoli et al., 2015, Elamir, 2022).

2. Nemenyi Mean-Ranks Post-hoc Test

Upon rejection of the Friedman test’s omnibus null, the Nemenyi test is traditionally used for all $C = m(m-1)/2$ algorithm pairs. For any algorithms $i$, $j$, the mean-rank difference $|\overline{R}_i - \overline{R}_j|$ forms the test statistic, evaluated versus a critical difference (CD):

$$CD = q_{\alpha; m, \infty} \sqrt{ \frac{m(m+1)}{6N} },$$

where $q_{\alpha; m, \infty}$ is the upper $\alpha$–quantile of the Studentized range distribution with $m$ treatments and infinite degrees of freedom. Algorithms $i$ and $j$ are declared significantly different at family-wise level $\alpha$ if $|\overline{R}_i - \overline{R}_j| > CD$. This controls the family-wise error rate across all $m(m-1)/2$ comparisons and is operationally analogous to Tukey-Kramer procedures for parametric ANOVA (Benavoli et al., 2015).

3. Critique of the Mean-Ranks Test and Its Dependence on Algorithm Pool

A foundational critique, detailed by Benavoli, Corani, and Mangili (2016) (Benavoli et al., 2015), is that Nemenyi’s mean-ranks test produces decisions for any pair $(A, B)$ contingent on the presence, absence, and relative ordering of all other algorithms involved in the ranking. This can lead to paradoxical scenarios:

• In an experiment where $A$ and $B$ each win on half the cases, two-algorithm tests (sign, Wilcoxon, t-test) yield nonsignificance. However, introducing additional poor-performing algorithms can inflate $|\overline{R}_A-\overline{R}_B|$ sufficiently to exceed $CD$ and declare a significant difference.
• On real-world datasets, the decision on a given pair can flip between significant and non-significant solely due to the composition of the algorithm pool. For example, in UCI data, algorithm pair $C_2$ vs $C_4$ was shown to change significance status depending on which other classifiers were included [(Benavoli et al., 2015), Table 4].

This pool-dependence means mean-ranks tests cannot guarantee control of maximum Type I error when equivalent algorithms are present, as also discussed by Fligner & Killeen (1984).

4. Alternative Two-Algorithm Post-hoc Procedures

To address the pool-dependence flaw, tests evaluating only the paired performances of $A$ and $B$ are recommended. These include:

a) Sign Test: For each dataset $j$, set $d_j = +1$ if $A_j > B_j$, $-1$ if $A_j < B_j$, $0$ if tie. $S =$ number of $+1$s among non-ties. Under $H_0$, $S \sim \text{Binomial}(n, 1/2)$. Large $n$ allows normal approximation:

$z = \frac{S - n/2}{\sqrt{n/4}}$

to compare to standard normal quantiles.

b) Wilcoxon Signed-Rank Test: For dataset $j$, compute $\delta_j = X_{Aj} - X_{Bj}$; discard ties. Rank $|\delta_j|$ among nonzero values, sum ranks $r_j$ for positive $\delta_j$. Under $H_0$ (symmetric differences),

$z = \frac{T - n(n+1)/4}{\sqrt{n(n+1)(2n+1)/24}}$

for large $n$. Both tests require family-wise correction (e.g., Bonferroni, Holm) over all $m(m-1)/2$ pairs (Benavoli et al., 2015).

5. Recent Developments: S-Statistics and Graphical Interpretation

Recent work (Elamir, 2022) proposes a graphical “S-plot” approach that simultaneously provides the global Friedman test and local post-hoc indications with drastically fewer comparisons. For $G$ treatments and $B$ blocks, each treatment $g$ has a score

$S_g = \frac{(R_g - \bar{R})^2}{ B G (G+1) /12 }$

where $\bar{R} = B(G+1)/2$ is the expected rank sum under $H_0$. The sum $F = \sum_{g=1}^G S_g$ recovers the classical Friedman statistic.

The distribution of $S_g$ is well-approximated via gamma moments matching:

• $E[S_g]$, $\text{Var}(S_g)$, and third moment $M_3(S_g)$ derived from those of $F$.
• Fitted Gamma($a$, $\beta$) with $a=4/\gamma_1(S_g)^2$, $\beta = a/E[S_g]$, matching mean and skewness.
• The threshold $DL = Q_{\text{Gamma}}(1-\alpha_{PT}; a, \beta)$ provides Bonferroni-adjusted familywise Type I error.

The S-plot visualizes each $S_g$; treatments with $S_g > DL$ are significant contributors to rejection. This reduces testing from $G(G-1)/2$ to $G$ with controlled error rates and delivers immediate interpretive insight (Elamir, 2022).

6. Empirical Validation and Practical Recommendations

Simulation studies have demonstrated that both the classical Friedman and S-statistic procedures maintain empirical Type I error within Bradley's robustness bounds across a range of $G$ and $B$ for both normal and exponential data, with accuracy improving as $B$ increases. Real-data applications (e.g., class size effects on children’s questions, per Gibbons & Chakraborti) confirm that the S-plot precisely identifies the dominant treatments responsible for global rejection, reducing the reliance on multiple pairwise post-hoc tables (Elamir, 2022).

Practical guidelines:

• Apply the Friedman test as the omnibus procedure for multiple-treatment, multiple-block designs.
• Avoid the classical Nemenyi mean-ranks test; its results for a pair may depend irrationally on other treatments present.
• Prefer pairwise comparisons based exclusively on two-algorithm tests (Wilcoxon signed-rank if symmetry plausible, else sign test), with appropriate correction for multiple comparisons (Benavoli et al., 2015).
• Consider global-to-local visualization approaches such as S-plots for succinct interpretability and error control.

7. Summary Table: Properties and Critique

Method	Pairwise Test Pool Dependence	Familywise Error Control	Number of Comparisons
Friedman + Nemenyi	Yes (dependent)	Yes (nominal)	$O(m^2)$
Friedman + Pairwise (Wilcoxon/Sign)	No (independent)	Yes (Bonferroni/Holm)	$O(m^2)$
S-Statistic/S-Plot [Editor’s term]	No (per-treatment)	Yes (Gamma approx, Bonferroni)	$O(m)$

The core limitation of the mean-ranks test is its statistical dependence on the composition of the entire set of algorithms, which undermines its relevance for pairwise inference. Alternative approaches leveraging either pairwise-only tests or S-statistical visualizations achieve more interpretable, pool-independent, and statistically valid post-hoc inference (Benavoli et al., 2015, Elamir, 2022).