Efficient and Intuitive Two-Phase Validation Across Multiple Models via Principal Components

Abstract: Two-phase sampling offers a cost-effective way to validate error-prone measurements in observational databases or randomized trials. Inexpensive or easy-to-obtain information is collected for the entire study in Phase I. Then, a subset of patients undergoes cost-intensive validation to collect more accurate data in Phase II. Critically, any Phase I variables can be used to strategically select the Phase II subset, often enriched for a particular model of interest. However, when balancing primary and secondary analyses in the same study, competing models and priorities can result in poorly defined objectives for the most informative Phase II sampling criterion. We propose an intuitive, easy-to-use solution that balances and prioritizes explaining the largest amount of variability across all models of interest. Using principal components to succinctly summarize the inherent variability of the error-prone covariates for all models. Then, we sample patients with the most "extreme" principal components (i.e., the smallest or largest values) for validation. Through simulations and an application to data from the National Health and Nutrition Examination Survey (NHANES), we show that extreme tail sampling on the first principal component offers simultaneous efficiency gains across multiple models of interest relative to sampling for one specific model. Our proposed sampling strategy is implemented in the open-source R package, auditDesignR.