A Generic and Efficient Python Runtime Verification System and its Large-scale Evaluation

Abstract: Runtime verification (RV) now scales for testing thousands of open-source Java projects, helping find hundreds of bugs. The popular Python ecosystem could use such benefits. But, today's Python RV systems are limited to a domain or specification logic, or slow. We propose PyMOP, a generic, extensible, and efficient RV system for Python. PyMOP supports five logics, implements five existing monitoring algorithms, ships with 73 API specs of Python and widely-used libraries, supports three instrumentation strategies, and users can easily add more of these. On 290,133 unit tests in 1,463 GitHub projects, we find mainly that (i) the default monitoring algorithm for Java is often not the fastest for Python; (ii) PyMOP is up to 1,168.3x faster than two recent dynamic analysis systems; and (iii) 44 of 121 bugs that PyMOP helped find so far were fixed by developers. PyMOP's generality and efficiency position it well as an excellent platform for the next advances on RV for Python.