Fine-grained Code Coverage Measurement in Automated Black-box Android Testing

Abstract: Today, there are millions of third-party Android applications. Some of these applications are buggy or even malicious. To identify such applications, novel frameworks for automated black-box testing and dynamic analysis are being developed by the Android community, including Google. Code coverage is one of the most common metrics for evaluating effectiveness of these frameworks. Furthermore, code coverage is used as a fitness function for guiding evolutionary and fuzzy testing techniques. However, there are no reliable tools for measuring fine-grained code coverage in black-box Android app testing. We present the Android Code coVerage Tool, ACVTool for short, that instruments Android apps and measures the code coverage in the black-box setting at the class, method and instruction granularities. ACVTool has successfully instrumented 96.9% of apps in our experiments. It introduces a negligible instrumentation time overhead, and its runtime overhead is acceptable for automated testing tools. We show in a large-scale experiment with Sapienz, a state-of-art testing tool, that the fine-grained instruction-level code coverage provided by ACVTool helps to uncover a larger amount of faults than coarser-grained code coverage metrics.