AWSOM-LP: An Effective Log Parsing Technique Using Pattern Recognition and Frequency Analysis

Abstract: Logs provide users with useful insights to help with a variety of development and operations tasks. The problem is that logs are often unstructured, making their analysis a complex task. This is mainly due to the lack of guidelines and best practices for logging, combined with a large number of logging libraries at the disposal of software developers. There exist studies that aim to parse automatically large logs. The main objective is to extract templates from samples of log data that are used to recognize future logs. In this paper, we propose AWSOM-LP, a powerful log parsing and abstraction tool, which is highly accurate, stable, and efficient. AWSOM-LP is built on the idea of applying pattern recognition and frequency analysis. First, log events are organized into patterns using a simple text processing method. Frequency analysis is then applied locally to instances of the same group to identify static and dynamic content of log events. When applied to 16 log datasets of the the LogPai project, AWSOM-LP achieves an average grouping accuracy of 93.5%, which outperforms the accuracy of five leading log parsing tools namely, Logram, Lenma, Drain, IPLoM and AEL. Additionally, AWSOM-LP can generate more than 80% of the final log templates from 10% to 50% of the entire log dataset and can parse up to a million log events in an average time of 5 minutes. AWSOM-LP is available online as an open source. It can be used by practitioners and researchers to parse effectively and efficiently large log files so as to support log analysis tasks.