Optimal Universal Rates in Binary Classification

• The paper establishes a tetrachotomy theorem that categorizes the optimal excess error rates into four distinct regimes, each tied to specific combinatorial properties of binary concept classes.
• It demonstrates that finite classes achieve exponential decay while infinite classes without infinite Littlestone trees attain near-exponential rates, and those with infinite Littlestone trees reach super-root or arbitrarily slow rates.
• The findings extend beyond traditional PAC and VC models, providing a unified framework for understanding agnostic binary classification and guiding the design of measurable learning rules.

Optimal universal rates for binary classification refer to the sharp, distribution-dependent asymptotics for the regression excess error achieved (and unimprovable) by measurable learning rules, for arbitrary binary concept classes $C \subseteq \{0,1\}^X$. The “universal” context removes the restrictive realizability assumption and demands that the optimal rate holds for each $P$ (allowing label noise), sharply distinguishing this theory from traditional PAC or VC-only frameworks. Recent advances establish a precise “tetrachotomy” theorem: for any class, the minimax optimal universal convergence rate of the excess error must fall into one of four distinct categories, each determined by combinatorial properties of $C$ (Hanneke et al., 28 Jan 2026).

1. Precise Definition of Universal Rates and Learning Setup

Let $C \subseteq \{0,1\}^X$ be a measurable concept class and $P$ a probability distribution over $X \times \{0,1\}$. For any sequence of measurable learning algorithms $$f_n : (X \times \{0,1\})^n \times X \rightarrow \{0,1\}$$, define the (agnostic) excess error at sample size $n$ by

$$E_n(C;P) := \mathbb{E}\left[\operatorname{er}_P(f_n(S_n, \cdot))\right] - \inf_{h \in C}\operatorname{er}_P(h),$$

where $\operatorname{er}_P(h) = P\{(x, y): h(x) \neq y\}$ and $S_n \sim P^n$.

$C$ is agnostically learnable at rate $R(n) \rightarrow 0$ if there exists a sequence $\{f_n\}$ such that for every $P$ there exist constants $c_P, C_P > 0$ (possibly depending on $C$, $P$), satisfying

$E_n(C; P) \leq C_P \, R(c_P n) \quad \forall n.$

Optimality at $R(n)$ further requires that for any strictly faster rate there exists a $P$ such that no measurable sequence of learning rules can achieve strictly smaller $E_n(C;P)$ for all large $n$ (Hanneke et al., 28 Jan 2026). Classes failing to satisfy this for any $R(n) \rightarrow 0$ require arbitrarily slow rates.

2. The Tetrachotomy Theorem

For any measurable concept class $C \subseteq \{0,1\}^X$, exactly one of the following four regimes holds for the optimal universal excess risk rate $R_n(C)$ (Hanneke et al., 28 Jan 2026):

Regime	Rate Type	Combinatorial Criterion
Exponential	$e^{-n}$	$|C| < \infty$ (finite class)
Near-Exponential	$e^{-o(n)}$	$C$ infinite, no infinite Littlestone tree
Super-root	$o(n^{-1/2})$	$C$ shatters infinite Littlestone tree, but not an infinite VCL tree
Arbitrarily Slow	$\to 0$ arbitrarily slowly	$C$ shatters infinite VCL tree

• “Near-exponential” means for all $ψ(n)=o(n)$, excess risk $\leq e^{-ψ(n)}$, but for any faster rate, not achievable.
• “Super-root” means for some $o(n^{-1/2})$ rate, but none strictly faster.
• “Arbitrarily slow” means for all $R(n)\rightarrow 0$, there exists $P$ s.t.\ no algorithm achieves $E_n(C;P) < c R(n)$ for all large $n$.

This tetrachotomy is exhaustive and mutually exclusive: any $C$ falls into exactly one case (Hanneke et al., 28 Jan 2026).

3. Combinatorial Structures Governing the Regimes

The regime for $C$ is determined by the existence of certain infinite combinatorial objects:

3.1 VC Dimension

$\mathrm{VC}(C) = \max\{|S|: C|_S = \{0,1\}^S\}$, with $|C|<\infty$ implying finite VC-dimension.

3.2 Infinite Littlestone Trees

A Littlestone tree of depth $d$ is a complete binary tree with nodes $x_u \in X$, indexed by $u \in \{0,1\}^{<d}$, such that for every root-to-depth-$k$ path with labels $y_1,\dots, y_k$, there is $h \in C$ with $h(x_{y_{<i}}) = y_i$. Existence of an infinite Littlestone tree ($d = \infty$) yields $C$ in super-root/arbitrarily slow regime; its absence yields (near-)exponential rates.

3.3 Infinite VCL Trees

A VCL (Vapnik–Chervonenkis Littlestone) tree generalizes shattering to paths in a tree-structured manner, encoding ever-more complex shattering along branches. Existence of an infinite VCL tree is the signature for arbitrarily slow rates: in such classes, no uniform rate (even $o(1)$) is possible (Hanneke et al., 28 Jan 2026).

4. Structural Proof Overview for Each Regime

4.1 Exponential ($|C| < \infty$)

Any learning rule using empirical risk minimization over $C$ receives exponential tail decay of excess risk via a union-bound and Hoeffding's inequality.

4.2 Near-Exponential (Infinite $C$; No Infinite Littlestone Tree)

Learning proceeds via sequential optimal algorithms that, by virtue of finite Littlestone dimension, achieve error decay $e^{-o(n)}$. Lower bounds are driven by slow-decaying distributions focused on “hard” points.

4.3 Super-Root (Infinite Littlestone Tree, No Infinite VCL Tree)

Admissible rates are $o(n^{-1/2})$ but cannot be improved uniformly. The upper bound follows from advanced transductive-ERM rules on “local” concept classes of bounded VC-dimension; lower bounds use coin-testing along the infinite tree with carefully decaying mass along branches.

4.4 Arbitrarily Slow (Infinite VCL Tree)

If $C$ shatters an infinite VCL tree, for any chosen rate $R(n)\rightarrow 0$ there is a realizable distribution on which error cannot decay faster—a fundamental obstruction to uniform convergence (Hanneke et al., 28 Jan 2026).

5. Canonical Examples for Each Regime

• Exponential: $C=\{\text{constant functions}\}$ or any finite class.
• Near-Exponential: Thresholds on $\mathbb{N}$; $C = \{1_{x \geq t} : t \in \mathbb{N}\}$. Infinite, but no infinite Littlestone tree.
• Super-root: Class of coordinate-wise nondecreasing functions on $\mathbb{N}^d$: infinite VC and Littlestone dimension, but no infinite VCL tree, so rates are super-root but not slower.
• Arbitrarily Slow: Parity functions or class of all finite supports on $\mathbb{N}$, both having infinite VCL trees (Hanneke et al., 28 Jan 2026).

6. Connections with Prior Realizable Theory and Multiclass Generalizations

This tetrachotomy strictly extends the previous trichotomy in the realizable case, which lacks the near-exponential and super-root distinctions, into the fully agnostic field (Hanneke et al., 28 Jan 2026). The multiclass scenario (for countable $\mathcal{Y}$) yields a related but distinct structure: presence or absence of infinite Littlestone or DSL (Daniely–Shalev–Shwartz–Littlestone) trees then separates exponential, near-linear, and arbitrarily slow regimes (Hanneke et al., 2023). The agnostic binary result uniquely introduces super-root and near-exponential scaling, filling the canonical hierarchy of optimal rates for learning curves.

7. Summary Table of the Tetrachotomy

Regime	Optimal Universal Rate	Combinatorial Structure	Example
Exponential	$e^{-n}$	$|C|<\infty$	Constant classifiers
Near-Exponential	$e^{-o(n)}$	No infinite Littlestone tree, $|C|=\infty$	Thresholds on $\mathbb{N}$
Super-root	$o(n^{-1/2})$	Infinite Littlestone, no infinite VCL tree	Monotone functions $\mathbb{N}^d$
Arbitrarily Slow	No uniform rate ($\to 0$ arbitrarily slowly)	Infinite VCL tree	Parity on $\mathbb{N}$, finite subsets

• "A Theory of Universal Agnostic Learning" (Hanneke et al., 28 Jan 2026)
• "Universal Rates for Multiclass Learning" (Hanneke et al., 2023)