Semi-Supervised Hypothesis Testing by Betting on Predictions

Abstract: We introduce a testing-by-betting framework that leverages predictions on unlabeled data to enhance the power of sequential hypothesis testing. Given limited samples from the joint distribution of $(X,Y)$, and additional unlabeled samples from the marginal of $X$, we ask how unlabeled data can be used to hypothesize about the distribution of $Y$, and the conditional distribution of $Y\mid X$. We introduce an e-statistic and use it to construct a sequential test. Under standard distributional assumptions -- label shift or concept shift -- we establish that the test is anytime valid. Furthermore, we show that for binary data, the e-statistic has non-trivial power. Crucially, our approach retains these properties even when the underlying predictions are inaccurate. Through simulations and applications to LLMs evaluation, we demonstrate power gains over baseline approaches, including prediction-powered inference. These gains persist even with relatively limited unlabeled data and when predictions have low accuracy due to weak correlation between $X$ and $Y$.

• The paper introduces a sequential hypothesis testing framework that combines predictions from abundant unlabeled data with betting strategies to achieve robust anytime type-I error control.
• It develops an imputed e-statistic leveraging soft-rank e-values and permutation methods to maintain validity under label and concept shifts, even with weak predictor accuracy.
• Empirical evaluations, including LLM assessments and simulations under various shifts, demonstrate a power advantage over traditional likelihood ratio tests.

Semi-Supervised Hypothesis Testing by Betting on Predictions

Introduction and Framework Overview

This work presents a sequential hypothesis testing methodology that leverages predictions from unlabeled covariate data to enhance the power of e-processes, especially in the context of semi-supervised settings where labeled samples are expensive, but unlabeled data are abundant. Classic hypothesis testing often relies purely on labeled data, pre-specifying sample sizes and ignoring the augmentation potential of predictive models trained on covariates. Motivated by tasks such as LLM evaluation and treatment efficacy monitoring, the proposed approach extends the “testing by betting” paradigm to the semi-supervised regime, constructing valid e-statistics by betting directly on predicted labels assigned to new, unlabeled data points.

The central methodological innovation is the introduction of an imputed e-statistic that operationalizes the use of predictions in e-processes. At each round, a predictive model, trained on recent labeled data, infers outcomes for a batch of fresh unlabeled samples. The test statistic is then constructed as a function of these predictions and their reference distribution under the null hypothesis, leveraging soft-rank e-values and permutation-based validity. Critically, this approach provides finite-sample, anytime type-I error control under both label shift (where $P_{X|Y}$ is fixed) and concept shift (where $P_X$ is fixed), with no reliance on the accuracy of the predictive model, thereby ensuring robustness even when $X$ and $Y$ are only weakly correlated.

Imputed E-Statistic Construction and Sequential Testing

The batch-mode imputed e-statistic for an observed labeled set $D$ and unlabeled covariates $\vec{\tilde{X}}$ takes the form

$$\breve{e}[D, \vec{\tilde{X}}] = \frac{(M+1) K(\tilde{\vec{Y}}^0)}{K(\tilde{\vec{Y}}^0) + \sum_{i=1}^M K(\tilde{\vec{Y}}^i)}$$

where $K$ is a positive, order-preserving function (often $K(\vec{y}) = \sum_{i=1}^N \exp(\gamma \tilde{y}_i)$), $\mathcal{A}$ is a learning algorithm for predictions, and the denominator aggregates pseudo-label scores from $P_X$0 exchangeable datasets sampled under the null.

Validity is established via exchangeability arguments and soft-rank e-values: under the null, the observed and simulated datasets are identically distributed, so the imputed e-statistic is a valid e-value. For sequential settings, the method constructs an imputed e-process by multiplying batchwise conditional e-statistics and uses Ville’s inequality to guarantee anytime control of type-I error.

Robustness to model mis-specification and plug-in estimation of the null are addressed via bounds on total variation between the true and estimated null distributions; excess type-I error is proportional to this distance but can be controlled in practical settings (see Theorem 1 and Appendix).

Power Analysis in Discrete Settings

A central focus is the power of the method when both $P_X$1 and $P_X$2 are binary. Under label shift, a threshold classifier that estimates $P_X$3 using the empirical distribution is shown to confer non-trivial power, regardless of prediction quality. The paper demonstrates, through stochastic ordering and soft-rank arguments, that even inaccurate predictors yield e-statistics whose expectation under the alternative exceeds unity. Similar results hold under concept shift, with the Bayes classifier generating probabilistic outputs.

Theoretical analysis is supported by theorems indicating that, for right-sided alternatives, the imputed e-process dominates or augments classical likelihood ratio e-processes, maintaining both validity and non-trivial power even with weak $P_X$4–$P_X$5 correlation and limited $P_X$6.

Empirical Evaluation: Simulations and Real-World Data

Label Shift Simulations

The authors extensively benchmark their method against baselines including likelihood ratio e-processes and prediction-powered inference (PPI) over a range of correlation strengths and unlabeled set sizes. Results consistently demonstrate that, in regimes of low correlation or scarce unlabeled data, the imputed e-process provides a tangible power advantage over both LRT and PPI, even when these baselines are optimally tuned. The combined e-process, aggregating all methods via universal portfolio selection, consistently benefits from the imputed component, especially in the most challenging statistical regimes.

Figure 1: Power as a function of increasing unlabeled sample size and correlation between $P_X$7 and $P_X$8 under label shift, illustrating superior power of the imputed e-process in scarce and low-correlation settings.

Concept Shift Simulations

In concept shift settings, where $P_X$9 is matched between null and observed distributions but $X$0 may shift, the imputed e-process shows similar empirical power gains, especially in low-correlation scenarios—and remains the sole test able to bet against the conditional null using information in $X$1.

Figure 2: Simulation results for concept shift, highlighting power improvements from integrating predictions on unlabeled data in both low- and high-correlation settings.

LLM Evaluation and In-Context Learning

Applying the method to LLM benchmark evaluation, the authors demonstrate its ability to detect minor improvements in model accuracy while minimizing annotation cost by incorporating LLM-generated annotations as $X$2. The gains persist when using data-scarce, hard-to-separate task settings (e.g., math QA), confirming theoretical robustness.

Figure 3: Power in sequential testing of LLM improvements, varying input type $X$3 and availability of unlabeled data.

For structured data, the approach is further extended to mixed continuous-multiclass inputs. Using state-of-the-art tabular foundation models (e.g., TabPFN), in-context training on census data yields significant power improvements for detecting changes in health insurance procurement, with the combined e-process outperforming standard PPI and LRT.

Figure 4: Improved power for detecting increased private health insurance procurement using in-context learning with mixed-type covariates.

Practical and Theoretical Implications

This methodology offers a principled, flexible enhancement to online and sequential testing—especially valuable in modern experimental pipelines characterized by scarcity of labeled outcomes and abundance of auxiliary covariate data. The framework provides a robust tool for hypothesis evaluation in operational and research settings where the cost of labeling is prohibitive, including ongoing model selection, A/B testing, and monitoring of shifting population subgroups.

From a theoretical standpoint, the analysis delineates clear identifiability and validity boundaries for leveraging predictions in e-processes. By requiring only weak assumptions about identifiability (label shift or concept shift), and being agnostic to predictor quality, the framework generalizes beyond previous prediction-powered approaches reliant on high-fidelity models and large $X$4. Furthermore, the formal robustness to plug-in null estimation provides practical guidance for deployment in data-limited contexts.

Negative results—in particular, the sharp reduction in power for non-monotonic alternatives unless $X$5 is appropriately designed—underscore the need for domain alignment between the alternative and the score function.

Future Directions

Open research questions include:

• Characterization of optimal predictors and score functions: What are the optimal choices of $X$6 and $X$7 for arbitrary data types and alternative hypotheses?
• Growth rate and expected stopping time: While non-trivial power is established, fine-grained analysis of time-to-rejection distributions and their dependence on $X$8, $X$9, and model class is a key direction.
• Extension to composite hypotheses and multivariate/multiclass regimes: The present binary focus generalizes under composite nulls and soft-rank constructions, as shown, but more comprehensive theory and applications to complex outputs are warranted.

Given the practical scalability (feasible even with foundation models and limited $Y$0), the framework is suitable for real-world deployment across both tabular and unstructured data domains.

Conclusion

The methodology of semi-supervised hypothesis testing by betting on predictions offers a rigorously justified, computationally viable approach for enhancing sequential testing with abundant but noisy covariate information. Empirical and theoretical analyses indicate that power gains are robust to weak model accuracy and limited unlabeled data, underpinned by strong validity guarantees even in plug-in null settings. This opens new possibilities for evidence accumulation and resource-efficient evaluation in evolving AI and statistical applications.