DRONE: a Distributed Subgraph-Centric Framework for Processing Large Scale Power-law Graphs

Abstract: Nowadays, in the big data era, social networks, graph databases, knowledge graphs, electronic commerce etc. demand efficient and scalable capability to process an ever increasing volume of graph-structured data. To meet the challenge, two mainstream distributed programming models, vertex-centric (VC) and subgraph-centric (SC) were proposed. Compared to the VC model, the SC model converges faster with less communication overhead on well-partitioned graphs, and is easy to program due to the "think like a graph" philosophy. The edge-cut method is considered as a natural choice of subgraph-centric model for graph partitioning, and has been adopted by Giraph++, Blogel and GRAPE. However, the edge-cut method causes significant performance bottleneck for processing large scale power-law graphs. Thus, the SC model is less competitive in practice. In this paper, we present an innovative distributed graph computing framework, DRONE (Distributed gRaph cOmputiNg Engine). It combines the subgraph-centric model and the vertex-cut graph partitioning strategy. Experiments show that DRONE outperforms the state-of-art distributed graph computing engines on real-world graphs and synthetic power-law graphs. DRONE is capable of scaling up to process one-trillion-edge synthetic power-law graphs, which is orders of magnitude larger than previously reported by existing SC-based frameworks.