Targeted Test Selection (T-TS)

• Targeted Test Selection (T-TS) is a method for choosing specialized test subsets from large pools to efficiently detect regressions, close coverage gaps, or assess operational risks.
• T-TS techniques employ predictive modeling, submodular optimization, and statistical sampling to prioritize tests based on failure likelihood and resource constraints.
• Empirical studies show T-TS can reduce test executions by up to 30× while achieving fault detection rates above 95%, optimizing cost and performance in diverse scenarios.

Targeted Test Selection (T-TS) refers to a class of techniques for constructing, prioritizing, or curating specialized test subsets from large candidate pools under explicit operational constraints. Unlike comprehensive or uniform test execution, T-TS focuses resources on those tests most likely to deliver value regarding specific objectives: surfacing regressions, maximizing coverage growth, improving performance on critical distribution slices, detecting rare failures, or efficiently estimating deployment risks. Contemporary T-TS methods integrate advances from predictive modeling, optimization, program analysis, and statistical sampling, and are deployed in diverse contexts including continuous integration, simulation-based verification, language-model evaluation, and educational assessment.

1. Problem Formulations and Core Objectives

T-TS is motivated by the infeasibility of full-suite testing when test executions, labeling, or simulations are expensive or time-constrained. The central challenge is to construct, for each operational scenario (change, model update, target distribution, or requirement), a budgeted subset of tests or data points that maximizes specific utility criteria. Typical objectives include:

• Regression/fault detection efficiency: Maximizing the fraction of faults (test failures) surfaced, or minimizing the test budget required to catch regressions after code changes (Plyusnin et al., 12 Sep 2025, Machalica et al., 2018).
• Coverage progress acceleration: Selecting tests most likely to close remaining functional coverage gaps in simulation-based hardware or software verification (Masamba et al., 2022).
• Operational risk estimation: Estimating the true failure or error rate of a deployed system with minimal labeled queries by selecting representative or high-risk examples (Asgari et al., 25 Jun 2026, Kothawade et al., 2021).
• Targeted improvement for rare slices: Curating subsets that optimize performance on targeted slices of the data distribution, e.g., underrepresented classes or operational subpopulations (Kothawade et al., 2021).
• Educational/test suite curation: Producing concise test sets with high discriminative power for grading or assessment under resource limits (Marques et al., 2022).

The problem typically takes the form:

$$\underset{S \subseteq \mathcal{T},\, |S| \leq k}{\text{maximize}} \quad U(S)$$

where $\mathcal{T}$ is the candidate test set (potentially large), $k$ is the budget, and $U(S)$ encodes the operational utility (e.g., expected failures discovered, coverage increase, risk estimation accuracy).

2. Methodological Approaches

T-TS methodologies can be grouped according to the mechanisms and signals involved:

2.1 Predictive and Machine Learning-Based Selection

Machine learning-based T-TS methods model failure likelihood or test utility from historical data, code metadata, or behavioral traces. In continuous integration, high-profile examples include:

• Predictive Test Selection (PTS) at Facebook, training gradient-boosted classifiers to score the regression-finding value of each candidate test given a change, based on rich features encoding change history, prior test failures, dependency distances, and project context (Machalica et al., 2018).
• Bag-of-Words Commit Modeling: Modeling each commit as a sparse vector over changed files (features including frequency, authorship, and distance to test), then learning risk relationships without dependence on explicit code coverage maps. These approaches leverage XGBoost or similar classifiers and achieve industrial deployment scalability, with empirical evidence of reducing test executions by 5.9x while maintaining >95% fault detection (Plyusnin et al., 12 Sep 2025).
• Cross-feature augmentation: Features such as directory-structure distance, test historical flakiness, and change-author statistics are explicitly modeled.

2.2 Coverage-Directed and Supervised Selection

In hardware and simulation-driven verification, T-TS is operationalized as coverage-directed test selection. Here:

• Supervised classifiers (decision trees, gradient-boosted models, naive Bayes) are trained on earlier test stimuli and fine-grained functional coverage feedback to predict which tests will exercise currently uncovered (hole) coverage groups (Masamba et al., 2022).
• Only candidate test stimuli predicted to contribute to coverage closure are simulated. This reduces simulation resource consumption and decreases manual constraint authoring.

2.3 Submodular Optimization and Target-Aware Utility

Submodular mutual information (SMI) frameworks provide a principled T-TS formulation for scenarios where the aim is to optimize performance on a specified target distribution or slice:

$$\underset{\mathcal{A} \subseteq \mathcal{U},\, |\mathcal{A}| \leq k}{\text{maximize}}\, I_f(\mathcal{A}; \mathcal{T}) + \gamma\,g(\mathcal{A})$$

where $I_f$ is a submodular mutual information score between selected unlabeled points and the target set, and $g$ is an internal diversity reward. Greedy submodular maximization is used for tractable near-optimal selection, with empirical evidence showing >20% improvement in target-class accuracy over active-learning and baseline subset selection on image classification benchmarks (Kothawade et al., 2021).

2.4 Statistical and Diversity-Aware Sampling

In deep network test case selection, especially for expensive LLM evaluation:

• Feature-aware, diversity-oriented sample selection—using stratification or clustering in latent space—provides stronger estimation efficiency than classic uncertainty heuristics when the goal is estimation (e.g., KCenterHT, D$^2$Strat) (Asgari et al., 25 Jun 2026).
• Uncertainty-based sampling (e.g., highest-entropy, margin-based) prioritizes failure discovery but is less effective for unbiased deployment risk estimation. These distinctions hold robustly for LLMs-for-code, with context dependence on task and model.

2.5 Program Analysis and Granularity Control

Fine-grained T-TS, particularly for regression test selection, incorporates code and test dependence analysis:

• Assertion-based slicing: The smallest selectable unit is the assertion (not method/class). Slicing extracts the minimal code required for each assertion, mapping code changes to the exact checked behaviors impacted (Gu et al., 2024).
• Call graph and bytecode-level analysis: Tools such as CATTO use JIMPLE-level call graphs to map changes to test coverage without explicit coverage traces, supporting just-in-time T-TS directly in developer IDEs (d'Aragona et al., 2022).

2.6 Multi-objective Evolutionary Search

Multi-objective evolutionary algorithms (MOEAs), such as L2-NSGA, encode T-TS as a trade-off between cost (runtime) and quality (coverage, fault detection), with linkage learning to discover and preserve sets of correlated tests (Olsthoorn et al., 2021).

3. Evaluation Criteria and Metrics

T-TS techniques are evaluated/benchmarked using operational, model, and search quality criteria:

• Recall/Test Recall/Change Recall: Fraction of all (or faulty) tests/change points surfaced by the selected subset (Machalica et al., 2018).
• APFD/NAPFD: Average Percentage of Faults Detected and its normalized variant, considering order and fraction of failures found (Plyusnin et al., 12 Sep 2025).
• Coverage growth: Number of simulated (or real) tests to reach 95–99% closure (Masamba et al., 2022).
• RMSE (Root Mean Squared Error): For risk estimation, accuracy of the estimated error rate vs. ground truth (Asgari et al., 25 Jun 2026).
• ICE, IGD, HV: Area under the cost-effective front, Inverted Generational Distance, Hypervolume for evolutionary T-TS (Olsthoorn et al., 2021).
• Assertion/test selection ratio: Precision indicators for granularity-refined selection (Gu et al., 2024).
• Suite reduction/inclusiveness/fault-detection loss: For lightweight or just-in-time tools (d'Aragona et al., 2022).

4. Empirical Outcomes and Deployment Impact

Multiple studies report substantial operational benefits from T-TS paradigms:

Study	Context	Relative Suite Size	Recall	Cost/Time Reduction
PTS (Machalica et al., 2018)	Facebook CI	< 0.33	>0.95 (Test), >0.999 (Change)	2–3× infra cost/test exec
T-TS (Plyusnin et al., 12 Sep 2025)	Industrial CI	0.15 (15%)	0.97	5.9× execution, 5.6× pipe.
CDS (Masamba et al., 2022)	HW Sim	~0.82–0.86	n/a	~18% fewer simulations
Selertion (Gu et al., 2024)	Regression	0.158 (15.8%)	n/a	63% test-time reduction
L2-NSGA (Olsthoorn et al., 2021)	Regression	Varies	n/a	>19 pp. ICE improvement
TestSelector (Marques et al., 2022)	Education	Fixed (n=30)	n/a	2×–3× higher bug find

Empirically, T-TS reduces the number of tests executed by factors of 3–30×, with recall consistently above 90–95% in regression/fault-oriented profiles and 10–20% improvement in coverage or bug-discovery rates over random selection in education and simulation tasks.

5. Technical Limitations and Contextual Considerations

T-TS effectiveness and trade-offs depend on several context-specific and design factors:

• Label/ground-truth leakage: Training on nightly aggregates but deploying on per-commit may induce a domain shift (Plyusnin et al., 12 Sep 2025).
• Coverage-agnostic failures: Excluding coverage maps improves scalability but may miss dependency-induced failures.
• Feature/representation selection: Directory proximity is more robust than name similarity in industrial settings (Plyusnin et al., 12 Sep 2025).
• Flakiness handling: De-flaking by repeated retries or flag filtering is critical to avoid overfitting to noisy failure signals (Machalica et al., 2018, Plyusnin et al., 12 Sep 2025).
• Resource bottlenecks: In simulation, test generation is negligible; cost is concentrated in evaluation (Masamba et al., 2022).
• Task/model structure: In LLMs, representation-aware T-TS excels for estimation, uncertainty-aware for failure surfacing; the optimal method is task/model-context dependent (Asgari et al., 25 Jun 2026).
• Fine-grained granularity vs. analysis overhead: Finer selection (statement/assertion) produces greater reduction but higher static analysis requirements (Gu et al., 2024).
• Interpretability vs. predictive power: Tree-based models enable constraint extraction for test generation, whereas neural or ensemble models may outperform but are less interpretable (Masamba et al., 2022).

A general implication is that T-TS effectiveness improves when the granularity of selection (e.g., assertion, slice, test, input class) is well matched to the code/test structure and operational constraints.

6. Extensions and Specializations

T-TS has been specialized for advanced settings:

• Strategy-level test-time scaling in LLMs: Instead of randomly sampling reasoning paths, active budget allocation across extracted strategies mitigates selection bias and increases pass/accuracy rates in arithmetic reasoning (Wu et al., 22 Sep 2025).
• Type targeted testing: Refines T-TS to select tests instantiating formal input specifications, with refinement types translated to SMT-constrained input generation for systematic, exhaustive verification up to a bound (Seidel et al., 2014).
• Multi-objective search via linkage learning: Discovers co-occurrence patterns among high-value tests (or code units) to guide recombination in evolutionary subset selection, yielding better Pareto-optimal fronts in regression testing (Olsthoorn et al., 2021).
• Test set selection for text classification: Feature selection via per-category t-test statistics allows T-TS to identify discriminative terms for imbalanced multiclass problems, outperforming classic $\chi^2$ and IG in certain data regimes (Wang et al., 2013).
• Automated curriculum/assessment test selection: Instructor-facing frameworks integrate custom-defined coverage measures and efficient hitting-set optimization to construct maximal-bug-finding test suites in education (Marques et al., 2022).

7. Future Research and Open Problems

Key open areas for T-TS include formalizing trade-offs between cost, recall, and operational lag in dynamic repositories; unifying coverage-agnostic and coverage-aware strategies for cross-project generalizability; integrating interpretability and constraint extraction with deep or ensemble methods for test generation; developing robust uncertainty estimation under distributional drift; and supporting targeted improvement/monitoring under continually evolving data and adversarial inputs.

---

References: (Machalica et al., 2018, Masamba et al., 2022, Plyusnin et al., 12 Sep 2025, Gu et al., 2024, Kothawade et al., 2021, d'Aragona et al., 2022, Asgari et al., 25 Jun 2026, Wu et al., 22 Sep 2025, Olsthoorn et al., 2021, Seidel et al., 2014, Wang et al., 2013, Marques et al., 2022).