Hypothesis testing for two population means: parametric or non-parametric test?

Abstract: The parametric Welch $t$-test and the non-parametric Wilcoxon-Mann-Whitney test are the most commonly used two independent sample means tests. More recent testing approaches include the non-parametric, empirical likelihood and exponential empirical likelihood. However, the applicability of these non-parametric likelihood testing procedures is limited partially because of their tendency to inflate the type I error in small sized samples. In order to circumvent the type I error problem, we propose simple calibrations using the $t$ distribution and bootstrapping. The two non-parametric likelihood testing procedures, with and without those calibrations, are then compared against the Wilcoxon-Mann-Whitney test and the Welch $t$-test. The comparisons are implemented via extensive Monte Carlo simulations on the grounds of type I error and power in small/medium sized samples generated from various non-normal populations. The simulation studies clearly demonstrate that a) the $t$ calibration improves the type I error of the empirical likelihood, b) bootstrap calibration improves the type I error of both non-parametric likelihoods, c) the Welch $t$-test with or without bootstrap calibration attains the type I error and produces similar levels of power with the former testing procedures, and d) the Wilcoxon-Mann-Whitney test produces inflated type I error while the computation of an exact p-value is not feasible in the presence of ties with discrete data. Further, an application to real gene expression data illustrates the computational high cost and thus the impracticality of the non parametric likelihoods. Overall, the Welch t-test, which is highly computationally efficient and readily interpretable, is shown to be the best method when testing equality of two population means.