Network Load Analysis and Provisioning of MapReduce Applications

Abstract: In this paper, we study the dependency between configuration parameters and network load of fixed-size MapReduce applications in shuffle phase and then propose an analytical method to model this dependency. Our approach consists of three key phases: profiling, modeling, and prediction. In the first stage, an application is run several times with different sets of MapReduce configuration parameters (here number of mappers and number of reducers) to profile the network load of the application in the shuffle phase on a given cluster. Then, the relation between these parameters and the network load is modeled by multivariate linear regression. For evaluation, three applications (WordCount, Exim Mainlog parsing, and TeraSort) are utilized to evaluate our technique on a 4-node MapReduce private cluster.