New Fault Domains for Conformance Testing of Finite State Machines

Abstract: A fault domain reflects a tester's assumptions about faults that may occur in an implementation and that need to be detected during testing. A fault domain that has been widely studied in the literature on black-box conformance testing is the class of finite state machines (FSMs) with at most $m$ states. Numerous strategies for generating test suites have been proposed that guarantee fault coverage for this class. These so-called $m$-complete test suites grow exponentially in $m-n$, where $n$ is the number of states of the specification, so one can only run them for small values of $m-n$. But the assumption that $m-n$ is small is not realistic in practice. In his seminal paper from 1964, Hennie raised the challenge to design checking experiments in which the number of states may increase appreciably. In order to solve this long-standing open problem, we propose (much larger) fault domains that capture the assumption that all states in an implementation can be reached by first performing a sequence from some set $A$ (typically a state cover for the specification), followed by $k$ arbitrary inputs, for some small $k$. The number of states of FSMs in these fault domains grows exponentially in $k$. We present a sufficient condition for $k$-$A$-completeness of test suites with respect to these fault domains. Our condition implies $k$-$A$-completeness of two prominent $m$-complete test suite generation strategies, the Wp and HSI methods. Thus these strategies are complete for much larger fault domains than those for which they were originally designed, and thereby solve Hennie's challenge. We show that three other prominent $m$-complete methods (H, SPY and SPYH) do not always generate $k$-$A$-complete test suites.