Black-box optimization on hyper-rectangle using Recursive Modified Pattern Search and application to ROC-based Classification Problem

Abstract: In statistics, it is common to encounter multi-modal and non-smooth likelihood (or objective function) maximization problems, where the parameters have known upper and lower bounds. This paper proposes a novel derivative-free global optimization technique that can be used to solve those problems even when the objective function is not known explicitly or its derivatives are difficult or expensive to obtain. The technique is based on the pattern search algorithm, which has been shown to be effective for black-box optimization problems. The proposed algorithm works by iteratively generating new solutions from the current solution. The new solutions are generated by making movements along the coordinate axes of the constrained sample space. Before making a jump from the current solution to a new solution, the objective function is evaluated at several neighborhood points around the current solution. The best solution point is then chosen based on the objective function values at those points. Parallel threading can be used to make the algorithm more scalable. The performance of the proposed method is evaluated by optimizing up to 5000-dimensional multi-modal benchmark functions. The proposed algorithm is shown to be up to 40 and 368 times faster than genetic algorithm (GA) and simulated annealing (SA), respectively. The proposed method is also used to estimate the optimal biomarker combination from Alzheimer's disease data by maximizing the empirical estimates of the area under the receiver operating characteristic curve (AUC), outperforming the contextual popular alternative, known as step-down algorithm.