An Open-Source Project for MapReduce Performance Self-Tuning

Abstract: Many Hadoop configuration parameters have significant influence in the performance of running MapReduce jobs on Hadoop. It is time-consuming and tedious for general users to manually tune the parameters for optimal MapReduce performance. Besides, most of existing self-tuning system have opaque implementation, making it difficult to use in practice. This study presents an open-source project that hosts the developing self-tuning system called Catla to address the issues. Catla integrates multiple direct search and derivative-free optimization-based techniques to facilitate tuning efficiency for users. An overview of the system and its usage are illustrated in this study. We also reported a simple example demonstrating the benefits of this ongoing project. Although this project is still developing and far from comprehensive, it is dedicated to contributing Hadoop ecosystem in terms of improving performance in big data analysis.