Generalization Disagreement Equality (GDE)

• Generalization Disagreement Equality (GDE) is a property in deep learning ensembles where the average test error equals the expected prediction disagreement between independent model instances.
• It arises from class-aggregated calibration and feature interactions, enabling direct, label-free estimation of generalization error through disagreement rates.
• Empirical results across architectures and datasets confirm GDE’s robustness, highlighting its practical use in model evaluation and monitoring under distribution shifts.

The Generalization Disagreement Equality (GDE) is a phenomenon in modern deep learning ensembles, particularly evident in models trained via stochastic gradient descent (SGD). The GDE states that the average test error of an SGD-trained model matches, often with striking precision, the expected disagreement rate between two independently trained copies of the same model on unlabeled test data. This equivalence allows direct, label-free estimation of generalization error, closely links generalization to calibration, and enables new theoretical understanding of feature learning and model variability.

1. Formal Definitions and Mathematical Statement

Let $D$ denote the data-generating distribution on input-label pairs $(X,Y)$, where $X \in \mathcal{X}$ and $Y \in \{1, \ldots, K\}$; let $\mathcal{A}$ be a (possibly randomized) learning algorithm which, with random seed $s$, yields classifier $h_s: \mathcal{X} \to [K]$. The randomness in initialization, data order, or other algorithmic aspects induces a distribution $H$ over trained classifiers. Key quantities are:

• Test error for $h$:

$$\mathrm{err}_D(h) = \Pr_{(X,Y) \sim D}[h(X) \neq Y]$$

• Disagreement rate for two independently drawn classifiers $(X,Y)$0:

$(X,Y)$1

• Generalization Disagreement Equality (GDE): The equality is said to hold if

$(X,Y)$2

Thus, the population expected pairwise disagreement rate of two independently trained ensemble members equals the average test error of a single member (Jiang et al., 2021).

2. Theoretical Origin: Calibration, Ensembles, and the Emergence of GDE

The GDE is shown theoretically to arise from a property called class-aggregated calibration of the ensemble predictor. Define the ensemble’s “soft” predictor:

$(X,Y)$3

for $(X,Y)$4. The ensemble is class-aggregated calibrated if for every confidence level $(X,Y)$5,

$(X,Y)$6

This states: for datapoints where a class-confidence is $(X,Y)$7, the expected frequency of correct labeling matches $(X,Y)$8.

The main theoretical result establishes:

If the ensemble predictor is class-aggregated calibrated on $(X,Y)$9, then the GDE holds on $X \in \mathcal{X}$0.

The proof connects the expected accuracy and expected predicted confidence squared,

$X \in \mathcal{X}$1

where

$X \in \mathcal{X}$2

and the class-aggregated calibration ensures their expectations coincide. This equivalence implies the expected test error matches the expected disagreement rate (Jiang et al., 2021, Kirsch et al., 2022).

3. Empirical Manifestation and Protocol

Empirically, the protocol for observing GDE is as follows:

1. Train model instances: Fix architecture and training data; train two (or more) models with identical hyperparameters but different random seeds.
2. Compute disagreement: On a large unlabeled test set, compute the fraction of samples where the two models’ top-1 predictions differ.
3. Measure error: Independently, measure a single model’s test error using labeled data.
4. Compare: Across diverse datasets (CIFAR-10/100, SVHN), architectures (convolutional nets, ResNets), and hyperparameter sweeps, the test error and disagreement rates are observed to align nearly perfectly (typically $X \in \mathcal{X}$3), even under modest distribution shift.

This empirical equivalence is robust to network design, optimization details, and persists, to a degree, under mild dataset shift (Jiang et al., 2021, Kirsch et al., 2022).

4. Feature Learning, the Interaction Tensor, and Mechanistic Insights

A complementary perspective on GDE is provided by analyzing feature learning in deep networks via the interaction tensor $X \in \mathcal{X}$4, where $X \in \mathcal{X}$5 is the number of model seeds, $X \in \mathcal{X}$6 is the number of test points, and $X \in \mathcal{X}$7 enumerates discovered feature clusters (Jiang et al., 2023). Models trained from different seeds learn (partially) overlapping sets of features, and their predictions are determined by these features.

Closed-form expressions for expected test accuracy ($X \in \mathcal{X}$8) and agreement ($X \in \mathcal{X}$9) in a stylized two-tier (dominant vs. rare) feature model show that:

$Y \in \{1, \ldots, K\}$0

when specific combinatorial proportions among data, model capacity, and feature distributions are satisfied. In this construction, GDE holds without any explicit calibration assumption.

Variability induced by random seeds is essential: absent such randomness, all models share the same features and always agree.

5. Parameter Regimes, Calibration Error Bounds, and Limitations

Sufficient conditions for GDE:

In (Jiang et al., 2021), GDE holds exactly when the ensemble is class-aggregated calibrated. In (Jiang et al., 2023), GDE emerges from matching ratios between rare/dominant features in data and models. Under mild mismatch, GDE holds approximately; empirical violations occur under data transformations disrupting these ratios (e.g., partitioning by blue-channel intensity in CIFAR-10).

Calibration error bounds:

If calibration is not perfect, GDE becomes an approximate equality, with their difference bounded by the class-aggregated calibration error (CACE):

$Y \in \{1, \ldots, K\}$1

CACE, along with other calibration errors (e.g., ECE), empirically increases on subpopulations exhibiting higher disagreement, limiting the label-free accuracy of disagreement-based generalization estimates in these regimes (Kirsch et al., 2022).

Breakdown scenarios:

If the test distribution $Y \in \{1, \ldots, K\}$2 exhibits covariate or label shift, or if ensemble calibration degrades significantly (e.g., due to adversarial or highly structured interventions), the predictive equivalence can fail. The necessity to estimate CACE for new populations reintroduces the need for labeled data.

6. Practical Implications and Uses

Unlabeled accuracy estimation:

GDE provides a zero-label method to estimate test error: evaluate disagreement on unlabeled data. This tool is practical when labels are expensive, unavailable, or withheld.

Model evaluation under distribution shift:

GDE’s validity under mild domain adaptation suggests its use as a heuristic for monitoring generalization in transfer or semi-supervised contexts. However, rigorous application requires empirical calibration assessment or labeled sample evaluation.

Theoretical understanding:

GDE formalizes the link between generalization (test error) and calibration in modern neural networks, clarifying the ensemble properties that underlie generalization phenomena and informing studies in neural feature learning, model diversity, and ensemble theory.

7. Relation to Broader Literature and Open Directions

The GDE extends beyond the “double descent” and neural scaling phenomena as a striking empirical-statistical regularity in deep learning. The fact that GDE holds with identical training data (not requiring independent re-sampled datasets) strengthens earlier findings [see also Nakkiran & Bansal '20 as referenced in (Jiang et al., 2021)].

Current research identifies both the strengths and limitations of GDE:

• It is robust to seed-induced model variability, but fragile under certain data transformations.
• The combinatorial underpinnings in feature allocation, not just calibration, suffice for GDE in structured models.
• Open questions remain regarding GDE's behavior in regression, structured outputs, or highly non-i.i.d. data.

The property has catalyzed refined analyses of ensemble diversity, semi-supervised learning performance monitoring, and out-of-distribution error prediction.

---

Summary Table: GDE Empirical Protocol and Theoretical Conditions

Aspect	(Jiang et al., 2021)/(Kirsch et al., 2022)	(Jiang et al., 2023)
GDE Identity	$Y \in \{1, \ldots, K\}$3	$Y \in \{1, \ldots, K\}$4
Theoretical Sufficient Condition	Ensemble is class-aggregated calibrated	Feature pool and model capacity ratios matched
Empirical Manifestation	Observed across architectures, datasets	Observed under mild Pareto-type feature splits
Failure Regimes	Major domain shift, high disagreement, poor calibration	Destroyed feature ratio, adversarial data

References:

• "Assessing Generalization of SGD via Disagreement" (Jiang et al., 2021)
• "A Note on 'Assessing Generalization of SGD via Disagreement'" (Kirsch et al., 2022)
• "On the Joint Interaction of Models, Data, and Features" (Jiang et al., 2023)