Parallelization in Extracting Fresh Information from Online Social Network

Abstract: Online Social Network (OSN) is one of the most hottest services in the past years. It preserves the life of users and provides great potential for journalists, sociologists and business analysts. Crawling data from social network is a basic step for social network information analysis and processing. As the network becomes huge and information on the network updates faster than web pages, crawling is more difficult because of the limitations of band-width, politeness etiquette and computation power. To extract fresh information from social network efficiently and effectively, this paper presents a novel crawling method and discusses parallelization architecture of social network. To discover the feature of social network, we gather data from real social network, analyze them and build a model to describe the discipline of users' behavior. With the modeled behavior, we propose methods to predict users' behavior. According to the prediction, we schedule our crawler more reasonably and extract more fresh information with parallelization technologies. Experimental results demonstrate that our strategies could obtain information from OSN efficiently and effectively.