Comparing asymptotic variances of adversarial versus non-adversarial R-NCE

Determine whether a general relationship (e.g., inequality, equivalence, or dominance) exists between the asymptotic variance of the R-NCE estimator under non-adversarial training, Vθ = −∇θ^2 L(θ⋆, ξ⋆)^{-1}, and the asymptotic variance under adversarial (Stackelberg) training, Vθ^a = −∇θ^2 L(θ⋆, ξ̄)^{-1}, when the proposal family F_n is not realizable; specifically, ascertain conditions under which adversarial training improves, worsens, or leaves unchanged the estimator’s asymptotic efficiency relative to non-adversarial training.

Background

The paper analyzes Ranking Noise Contrastive Estimation (R-NCE) for conditional energy-based models, establishing consistency and asymptotic normality for estimators trained with a learned (but non-adversarial) negative sampler. It derives the asymptotic variance Vθ for such estimators and shows near-efficiency when the proposal distribution matches the data.

The authors then study adversarial (Stackelberg) training of R-NCE and derive a corresponding asymptotic variance Vθ^a. They prove that when the proposal family is realizable—i.e., can represent the true conditional distribution—the adversarial saddle point yields no additional benefit: pθ⋆(·|x) = pξ⋆(·|x) = p(·|x), and the asymptotic behavior coincides.

However, when the proposal family is not realizable (the practically relevant case where the sampler is intentionally simpler than the EBM), the paper notes that the relationship between the converged proposal parameters under non-adversarial training and the Stackelberg adversary is non-trivial. Consequently, they cannot make a general comparison between Vθ and Vθ^a, leaving open whether adversarial training confers theoretical benefits in this regime.