SCGDet: Malware Detection using Semantic Features Based on Reachability Relation

Abstract: Recently, with the booming development of software industry, more and more malware variants are designed to perform malicious behaviors. The evolution of malware makes it difficult to detect using traditional signature-based methods. Moreover, malware detection has important effect on system security. In this paper, we present SCGDet, which is a novel malware detection method based on system call graph model (SCGM). We first develop a system call pruning method, which can exclude system calls that have little impact on malware detection. Then we propose the SCGM, which can capture the semantic features of run-time program by grouping the system calls based on the reachability relation. We aim to obtain the generic representation of malicious behaviors with similar system call patterns. We evaluate the performance of SCGDet using different machine learning algorithms on the dataset including 854 malware samples and 740 benign samples. Compared with the traditional n-gram method, the SCGDet has the smaller feature space, the higher detection accuracy and the lower false positives. Experimental results show that SCGDet can reduce the average FPR of 14.75% and improve the average Accuracy of 8.887%, and can obtain a TPR of 97.44%, an FPR of 1.96% and an Accuracy of 97.78% in the best case.