A survey on test practitioners' awareness of test smells

Abstract: Developing test code may be a time-consuming task that usually requires much effort and cost, especially when it is done manually. Besides, during this process, developers and testers are likely to adopt bad design choices, which may lead to the introduction of the so-called test smells in test code. Test smells are bad solutions to either implement or design test code. As the test code with test smells increases in size, these tests might become more complex, and as a consequence, much harder to understand and evolve correctly. Therefore, test smells may have a negative impact on the quality and maintenance of test code and may also harm the whole software testing activities. In this context, this study aims to understand whether test professionals non-intentionally insert test smells. We carried out an expert survey to analyze the usage frequency of a set of test smells. Sixty professionals from different companies participated in the survey. We selected 14 widely studied smells from the literature, which are also implemented in existing test smell detection tools. The yielded results indicate that experienced professionals introduce test smells during their daily programming tasks, even when they are using standardized practices from their companies, and not only for their personal assumptions. Another relevant evidence was that developers' professional experience can not be considered as a root-cause for the insertion of test smells in test code.

• The paper presents survey findings that conditional logic and general fixture practices are the primary contributors to test smells in code.
• The survey of 60 professionals indicates that test smells are prevalent irrespective of developers' experience levels.
• The study underscores the need for systematic training and automated measures to improve the readability and reliability of test code.

A Survey on Test Practitioners' Awareness of Test Smells

Introduction to Test Smells

The paper "A survey on test practitioners' awareness of test smells" (2003.05613) examines the introduction of test smells during the test code development process. Test smells represent suboptimal solutions in the implementation and design of test code. These smells can exacerbate the complexity of test suites, making them difficult to understand, evolve, and maintain. The presence of such smells can deteriorate the overall quality of software testing activities. Through a survey conducted among 60 professionals, the study aimed to elucidate whether test practitioners inadvertently introduce these test smells during their daily programming tasks.

Test Smells: Types and Impact

The study identified and evaluated 14 distinct test smells, frequently discussed in the literature and implemented in test smell detection tools. These include Assertion Roulette, Conditional Test Logic, Constructor Initialization, and others, each posing specific challenges to test code quality and maintenance. For instance, Assertion Roulette complicates fault localization due to the lack of descriptive assertions, while Conditional Test Logic undermines test determinism and consistency by incorporating conditional flows. The impact is significant: compromised test code can lead to unreliable software testing and maintenance activities.

Survey Design and Methodology

The survey employed a descriptive, case-control approach to analyze the habitual practices of developers which might lead to test smell introduction. Participants provided feedback on practices related to the various test smells, both from the test creation and execution perspectives. These practices were formulated without referencing the term "test smell," minimizing potential biases. The survey was structured into three blocks: participant profiles, test creation practices, and test execution practices. This design aimed at understanding participants' interactions with test smells while allowing for the analysis of consistency across different stages of professional experience.

Key Findings

Test Creation and Execution Practices

The survey revealed notable findings regarding the frequency of specific practices associated with test smell introduction. Conditional Test Logic and General Fixture were identified as the most frequently introduced smells during test creation. These findings suggest prevalent practices, potentially ingrained in both personal and company standards, which lead to introducing suboptimal patterns into test codes.

Figure 1: Test Smells frequencies on test creation and execution.

Professional Experience Correlation

Interestingly, the study did not find any significant correlation between professional experience and the tendency to introduce test smells. Whether participants were seasoned experts or novices, test smell introduction did not demonstrate a trend differencing by years of professional experience. This suggests that test smells, as an unintentional byproduct, might not be mitigated solely through experience but may require more systematic approaches or training.

Figure 2: Test Smells frequencies on test creation according to professional experience.

Practices Leading to Test Smells

The study identified that the use of conditional logic and general fixture files were the predominant contributors to test smell occurrences, representing common pitfalls across practitioners’ daily activities. However, other contributors, such as reliance on magic numbers or indirect testing, although less frequent, still persistently impact test reliability and comprehensibility.

Implications and Future Directions

The results imply a need for improved awareness and strategies to mitigate the introduction of test smells in routine testing activities. Practitioners could benefit from training focused on recognizing and addressing test smells, enhancing the readability, maintainability, and reliability of test codes. Future research may aim to develop automated solutions or guidelines that proactively detect and alert developers to potential smells during code development.

Conclusion

This study provides crucial insights into the common practices that lead to test smell introduction among practitioners. It underscores the persistence of test smells across varying levels of professional experience and highlights specific areas where practitioners frequently falter. These findings pave the way for targeted interventions and research aimed at mitigating test smells in coding practices, thereby enhancing the overall efficacy of software testing endeavors.