PingAn: An Insurance Scheme for Job Acceleration in Geo-distributed Big Data Analytics System

Abstract: Geo-distributed data analysis in a cloud-edge system is emerging as a daily demand. Out of saving time in wide area data transfer, some tasks are dispersed to the edges. However, due to limited computing, overload interference and cluster-level unreachable troubles, efficient execution in the edges is hard, which obstructs the guarantee on the efficiency and reliability of jobs. Launching copies across clusters can be an insurance on a task's completion. Considering cluster heterogeneity and accompanying remote data fetch, cluster selection of copies affects execution quality, as different insuring plans drive different revenues. For providing On-Line-Real-Time analysis results, a system needs to insure the geo-distributed resource for the arriving jobs. Our challenge is to achieve the optimal revenue by dynamically weighing the gains due to insurance against the loss of occupying extra resource for insuring. To this end, we design PingAn, an online insurance algorithm promising $(1+\varepsilon)!!-!speed \ o(\frac{1}{\varepsilon^2+\varepsilon})!!-!competitive$ in sum of the job flowtimes via cross-cluster copying for tasks. PingAn shares resource among the anterior fraction of jobs with the least unprocessed datasize and the fraction is adjustable to fit the system load condition. After sharing, PingAn concretely insures for tasks following efficiency-first reliability-aware principle to optimize the revenue of copies on jobs' performance. Trace-driven simulations demonstrate that PingAn can reduce the average job flowtimes by at least $14\%$ than the state-of-the-art speculation mechanisms. We also build PingAn in Spark on Yarn System to verify its practicality and generality. Experiments show that PingAn can reduce the average job flowtimes by up to $40\%$ comparing to the default Spark execution.