Prediction-Powered Active Testing

Abstract: Active testing provides a label--efficient approach to risk estimation by adaptively selecting which test points should be labelled. However, existing estimators fail to exploit the informative predictions of powerful black--box models, even though such predictions are increasingly available in settings where labels remain expensive. To address this, we propose \textbf{Prediction--Powered Active Testing (PPAT)}, a novel label--efficient risk estimation framework that combines the unbiased LURE estimator \citep{farquhar2021statistical} with a prediction--powered control variate. Rather than using proxy predictions as biased pseudo--labels, PPAT uses them to residualise the loss, preserving unbiasedness while reducing variance. Beyond the estimator itself, PPAT also changes which points should be acquired: we derive oracle and practical surrogate--based acquisition rules tailored to reducing the variance of our estimator. Moreover, we establish asymptotic normality for PPAT, yielding asymptotically valid confidence intervals and thus a principled estimate of the uncertainty around our estimates. Across tabular regression and image--classification tasks, PPAT outperforms existing methods in risk estimation, while its confidence intervals attain the target coverage with substantially fewer labels and smaller widths.

• The paper presents PPAT, an unbiased risk estimator that incorporates a prediction-powered control variate to reduce variance using inexpensive proxy labels.
• It introduces a tailored acquisition strategy based on high-magnitude residualized losses for both regression and classification tasks.
• Empirical evaluations demonstrate that PPAT achieves lower median squared error and narrower confidence intervals than existing methods, ensuring label efficiency.

Prediction-Powered Active Testing: A Label-Efficient Framework for Model Risk Estimation

Motivation and Context

Estimating model risk, defined as the expected loss on a test pool, is essential for rigorous evaluation of AI systems. In practice, acquiring ground truth labels for large test sets remains costly, especially in domains requiring expert annotation. Active testing (AT) addresses this by adaptively querying labels, correcting for the bias induced by adaptive sampling. However, modern settings often provide inexpensive black-box predictions from pretrained or foundation models on the entire test pool. The challenge is to exploit these predictions to further improve label efficiency without incurring bias.

Existing approaches, including the Levelled Unbiased Risk Estimator (LURE) [farquhar2021statistical] and Active Surrogate Estimators (ASE) [kossen2022active], either treat proxy predictions as pseudo-labels (generally producing biased risk estimates) or interpolate losses with learned surrogates (compromising finite-sample unbiasedness and requiring strong surrogate generalization). This paper introduces Prediction-Powered Active Testing (PPAT) (2607.08347): an unbiased, importance-weighted estimator that leverages black-box predictions as a control variate to reduce the variance of risk estimates.

Technical Formulation

PPAT combines the unbiased LURE estimator with a prediction-powered control variate. Given a model $f$, test pool inputs $\mathbf{x}_i$, true unknown labels $y_i$, and a cheap proxy labeler $g$, PPAT computes proxy losses $$\widetilde{\ell}_i = \mathcal{L}(f(\mathbf{x}_i), g(\mathbf{x}_i))$$ and their mean $$\widetilde{R} = \frac{1}{N} \sum_{i = 1}^N \widetilde{\ell}_i$$. The PPAT estimator is:

$$R(\lambda) = \frac{1}{M} \sum_{m=1}^M V_m (\ell_{I_m} - \lambda (\widetilde{\ell}_{I_m} - \widetilde{R}))$$

where $V_m$ are the LURE importance weights, $\ell_{I_m}$ are queried losses, and $\lambda$ tunes the strength of the control variate. Choosing $\mathbf{x}_i$0 recovers LURE; higher $\mathbf{x}_i$1 exploits the proxy predictions to reduce variance. Unlike replacing true labels with proxy labels (as in ASE), PPAT preserves finite-sample unbiasedness, as the control variate is centered to have zero mean.

The authors derive oracle and surrogate-based acquisition rules tailored to the residualized objective induced by PPAT, maximizing the informativeness with respect to variance reduction. Additionally, they establish asymptotic normality for PPAT, leading to principled confidence intervals.

Variance Analysis and Selection of $\mathbf{x}_i$2

A central theoretical contribution is the variance decomposition:

$\mathbf{x}_i$3

where $\mathbf{x}_i$4 is the control variate estimator. Optimal $\mathbf{x}_i$5 (minimizing variance) is given by $\mathbf{x}_i$6. In practice, $\mathbf{x}_i$7 can be set heuristically (e.g., $\mathbf{x}_i$8 or $\mathbf{x}_i$9) depending on proxy informativeness, or estimated online via $y_i$0 which flattens the residuals, minimizing an upper bound on variance.

Empirical validation demonstrates that $y_i$1 achieves median squared error comparable to the best fixed $y_i$2 values.

Figure 1: Median squared error for PPAT on Keggdirected and CIFAR-10 across fixed $y_i$3 and $y_i$4, demonstrating optimality of $y_i$5.

Acquisition Strategy and Surrogate Proposal

The paper derives a myopic oracle proposal for sampling indices to minimize the variance of the PPAT estimator, favoring points with high-magnitude residualized losses. As true labels are unavailable, surrogate models predict conditional loss moments, and acquisition scores are based on root mean squared surrogate loss (rather than expected absolute value), ensuring acquisition is tailored to the variance objective.

A simplified form is provided for regression tasks (MSE loss), allowing practical computation, and analogous strategies are used for classification tasks.

Asymptotic Properties and Confidence Intervals

A significant result is the establishment of asymptotic normality for PPAT under mild conditions, with a plug-in estimator of the variance computable from acquired labels. This enables construction of confidence intervals with frequentist coverage guarantees. The uniform overlap assumption (implemented via proposal mixing) is critical to maintaining bounded importance weights.

Empirical Results

The methodology is evaluated on UCI regression datasets (Keggdirected, Keggundirected, Sml, Bike) and classification datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet), using strong proxies (TabPFN-2.5, CLIP zero-shot) and surrogates (Bayesian regression, Laplace BNNs).

Risk estimation performance: Across all datasets, PPAT achieves lower median squared error than LURE, ASE, and random sampling for multiple choices of $y_i$6.

Figure 2: Regression experiments comparing PPAT with LURE, ASE, and random sampling, demonstrating reduced median squared errors on UCI datasets.

Figure 3: Classification experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet, showing consistent improvements from PPAT in risk estimation accuracy.

PPAT with plug-in $y_i$7 converges to the optimal fixed-$y_i$8 variant, justifying online adaptation.

Component ablations: The distinct improvements from both the estimator and acquisition strategy are shown:

Figure 4: Influence of estimator choice; PPAT outperforms empirical and LURE risk estimation even when acquisition is tailored.

Figure 5: Influence of acquisition; PPAT's tailored strategy provides further gains over random or LURE acquisition with PPI estimator.

Coverage and interval widths: PPAT achieves confidence interval coverage at the target rate using fewer labels and produces substantially narrower intervals than baselines.

Figure 6: Empirical coverage rates for confidence intervals on UCI and classification datasets; PPAT attains nominal coverage quickly with narrower intervals.

Robustness and Additional Analyses

Extensive ablations confirm that PPAT's performance is robust to proxy and surrogate model choices (Figures 7, 8), and the method remains superior to directly using proxy predictions as labels or surrogate interpolation. Comparisons with active statistical inference frameworks show that PPAT also produces narrower confidence intervals in pool-based acquisition settings.

Theoretical and Practical Implications

PPAT integrates prediction-powered inference (PPI) and active testing, introducing a control variate mechanism that reduces estimator variance while retaining unbiasedness. This formal combination lays the groundwork for label-efficient evaluation of models, especially in settings where strong black-box proxies are available. The acquisition strategy, tailored to residualized losses, further advances pool-based risk estimation.

Practically, PPAT enables principled evaluation of AI systems in high-stakes and resource-constrained domains (medical, scientific, LLMs), producing reliable risk estimates with fewer labels and precise frequentist uncertainty quantification. Theoretical implications include the generalizability of control variate methods in adaptive sampling and the extension of prediction-powered inference beyond semi-supervised settings.

Future developments may include tighter non-asymptotic variance bounds under refined proposal conditions, the extension to settings with dynamically updated proxy models, and broader applications to model selection and benchmarking. The approach is compatible with in-context surrogates and can be integrated into hierarchical or stratified acquisition schemes.

Conclusion

Prediction-Powered Active Testing introduces a label-efficient, unbiased, and variance-reduced risk estimation framework by leveraging the predictions of powerful models as control variates within an active testing protocol. Empirical evaluations across regression and classification tasks show improved risk estimation accuracy, robustness to proxy and surrogate choices, and efficient construction of confidence intervals. The methodology advances adaptive data-efficient model evaluation and provides principled uncertainty quantification for AI systems (2607.08347).