Permutational Rademacher Complexity (PRC)

• Permutational Rademacher Complexity (PRC) is a complexity measure for transductive learning that quantifies the supremum deviation between test and training empirical averages.
• It employs symmetrization over all equiprobable train-test splits to derive tight, data-dependent risk bounds in fixed finite sample settings.
• PRC refines classical complexity measures by controlling empirical processes under sampling without replacement, ensuring sharper generalization guarantees.

Permutational Rademacher Complexity (PRC) is a complexity measure specifically tailored to the transductive learning setting, where a fixed finite data population is partitioned into labeled training and unlabeled test sets via sampling without replacement. PRC captures the supremum of deviations between empirical averages on test versus training subsets, and fundamentally differs from classical inductive Rademacher complexity which is designed for i.i.d. data. By directly symmetrizing over all equiprobable train–test partitions, PRC enables tight control of transductive empirical processes and underpins sharp, data-dependent risk bounds in this regime (Tolstikhin et al., 2015).

1. Formal Definition

Let $Z_N=\{z_1,\dots,z_N\}\subset\mathcal Z$, with $N=m+u$, be the fixed finite population. A learner receives a labeled training subset $Z_m\subset Z_N$ of size $m$, sampled uniformly without replacement, and must predict on the $u=N-m$ test points $Z_u=Z_N\setminus Z_m$. For any class of real-valued functions $F\subset\{\mathcal Z\to\mathbb R\}$, Permutational Rademacher Complexity is

$$\mathrm{PRC}_{m,u}(F,Z_N) := \mathbb{E}_{U\subset Z_N,\;|U|=m} \Bigg[ \sup_{f\in F} \Big( \tfrac{1}{u} \sum_{z\in Z_N\setminus U} f(z) - \tfrac{1}{m} \sum_{z\in U} f(z) \Big) \Bigg].$$

An equivalent formulation splits a fixed $Z_m$ into two parts of size $n$ and $m-n$:

$$Q_{m,n}(F,Z_m) := \mathbb{E}_{Z_n\subset Z_m} \Big[ \sup_{f\in F}(f(Z_k)-f(Z_n)) \Big],$$

where $Z_k=Z_m\setminus Z_n$. Conventionally, $n=\lfloor m/2 \rfloor$.

If $\mathcal{H}$ is a class of predictors and $\ell\colon Y\times Y\to[0,1]$ is a bounded loss,

$$F=L_\mathcal{H} := \{ \ell(h(\cdot),\cdot) : h\in\mathcal{H} \},$$

and for $h\in\mathcal{H}$,

$\Err_m(h) = \frac{1}{m}\sum_{z\in Z_m}\ell(h(z)), \quad \Err_u(h) = \frac{1}{u}\sum_{z\in Z_u}\ell(h(z)),$

yielding

$\mathrm{PRC}_{m,u}(L_\mathcal{H},Z_N)=\mathbb{E}\Big[\sup_{h\in\mathcal{H}}\big(\Err_u(h)-\Err_m(h)\big)\Big].$

2. Transductive Setting and Suitability

Transductive learning sets focus on prediction for a prescribed test set, given a labeled subset of a fixed population, with sampling performed without replacement. This scenario diverges fundamentally from the i.i.d. framework. Classical Rademacher complexity employs i.i.d.-based symmetrization, thereby failing to capture the dependency structure induced by finite-population splits.

PRC addresses this by symmetrizing over all equiprobable partitions, encoding the train–test split structure. This yields a measure that provides tight control over quantities of the form $\Err_u - \Err_m$ in the transductive regime and thereby facilitates sharper analysis of generalization in this context. Unlike Transductive Rademacher Complexity (TRC), PRC introduces no additional slack in the lower bound of the associated symmetrization inequalities and depends on test labels only via the losses observed on the training portion (Tolstikhin et al., 2015).

3. Symmetrization Inequality and Theoretical Guarantees

A central result is the sharp symmetrization inequality, which demonstrates that the expected supremum of the empirical process in the transductive scheme is tightly bounded in terms of PRC.

Symmetrization Theorem:

Assume $N=2m$ is even. For any $n\in\{1,\dots, m-1\}$,

$$\frac12\, \mathbb{E}_{Z_m}[Q_{m,m/2}(F,Z_m)] \le \mathbb{E}_{Z_m}\Big[\sup_{f\in F}(f(Z_u)-f(Z_m))\Big] \le \mathbb{E}_{Z_m}[Q_{m,n}(F,Z_m)]$$

with analogous results for the supremum of the absolute difference.

This result enables the use of PRC as a direct empirical process control tool in transductive settings, with no additive error in the case $m=u$.

4. Comparison with Rademacher and Transductive Rademacher Complexity

PRC generalizes and relates to classical and transductive Rademacher complexities. The traditional (conditional) Rademacher complexity on a sample $Z_m$ is

$$R_m(F,Z_m) = \mathbb{E}_{\epsilon_i=\pm1}\Big[\frac{2}{m} \sup_{f\in F}\sum_{i=1}^m \epsilon_i f(z_i)\Big].$$

Transductive Rademacher Complexity (TRC) is

$$R_{N}(F, Z_N, p) = (\tfrac{1}{m}+\tfrac{1}{u}) \; \mathbb{E}_{\sigma_i\in\{\pm1,0\}} \Big[\sup_{f\in F}\sum_{i=1}^N \sigma_i f(z_i)\Big],$$

where $\sigma_i=\pm 1$ with probability $p$, $0$ with probability $1-2p$.

The following relations hold:

Comparison to Rademacher:

For even $m$, any $Z_m$:

$$Q_{m,m/2}(F,Z_m) \le \left(1+\frac{2}{\sqrt{2\pi m}-2}\right)\, R_m(F,Z_m),$$

and if $|f(z)|\le B$ for all $f$,

$$|Q_{m,m/2}(F,Z_m)-R_m(F,Z_m)|\le \frac{2B}{\sqrt m}.$$

Comparison to TRC:

When $m=u$ and $N=2m$, $p=1/4$,

$R_N(F,Z_N) \le R_N(F,Z_N,1/4) \le 2 R_N(F,Z_N),$

and

$$\mathbb{E}_{Z_m}[Q_{m,m/2}(F,Z_m)] \le \left(2+\frac{4}{\sqrt{2\pi N}-2}\right) R_N(F,Z_N,1/4),$$

with a similar lower bound up to an $O(B/\sqrt N)$ term for bounded $f$.

These results indicate that PRC can be efficiently controlled via standard Rademacher-related complexity measures, while retaining features specifically adapted to the dependencies arising from finite-population splits.

5. Data-dependent Transductive Risk Bounds

Let $\mathcal{H}$ be a hypothesis class, $\ell\colon Y\times Y\to[0,1]$ a bounded loss, and $L_\mathcal{H}$ as above. When $m=u$ and $N=2m$, the following holds.

PRC-based Transductive Risk Bound:

For $n\in\{1,\ldots,m-1\}$, with probability at least $1-\delta$ over $Z_m$:

$\Err_u(h) \le \Err_m(h) + \mathbb{E}_{Z_n}[Q_{m,n}(L_{\mathcal{H}},Z_m)] + \sqrt{\frac{2N\ln(1/\delta)}{(N-1/2)^2}}$

for all $h\in\mathcal{H}$.

Replacing the expectation with a single sample PRC yields a fully empirical bound: with probability at least $1-\delta$,

$\Err_u(h) \le \Err_m(h) + Q_{m,n}(L_\mathcal{H},Z_m) + 2\sqrt{\frac{2N\ln(2/\delta)}{(N-1/2)^2}}$

The proof relies on a bounded-difference (McDiarmid-type) inequality for sampling without replacement and the symmetrization theorem, linking concentration of $g(Z_m)=\sup_{h}(\Err_u(h)-\Err_m(h))$ to PRC (Tolstikhin et al., 2015).

6. Context and Research Significance

The introduction of PRC by I. Tolstikhin, N. Zhivotovskiy, and G. Blanchard provides a rigorous framework for developing generalization bounds in transductive scenarios, where the standard i.i.d.-based tools are provably suboptimal. PRC achieves tighter control over the empirical process suprema and facilitates risk bounds that are both data-dependent and adaptive to the actual train–test split, without reliance on unknown test labels except through observed loss on training data.

Comparative results with classical and transductive Rademacher complexities provide quantifiable relationships, establishing PRC as a natural extension of these measures to finite, non-i.i.d. settings, and underscoring its suitability for properly quantifying hypothesis class capacity under the finite-population constraint (Tolstikhin et al., 2015).

• I. Tolstikhin, N. Zhivotovskiy, G. Blanchard, "Permutational Rademacher Complexity: a New Complexity Measure for Transductive Learning," (Tolstikhin et al., 2015).