Mutation Testing for Improving Fault Detection Capability of Unit Tests: A Case Study

Abstract

Mutation testing helps assess and improve the quality of a test suite with respect to its fault detection capability. It generates mutants by making small syntactic changes in a program and checking whether tests can detect the changes. The more mutants are detected, the stronger the test suite is considered to be. While mutation testing is generally recognized to be superior to traditional code coverage criteria in terms of evaluating test suite quality, it has some limitations that prevent it from being widely adopted in the industry. These limitations are computational expensiveness and significant developer time spent evaluating mutants and adding tests to kill them. In this study, past software bugs of several Scala systems at Twilio have been used to determine whether mutation testing could have helped to prevent the bugs. In particular, it has been checked whether new tests added during the process of mutation testing could have detected the faults. The results have shown that mutation testing was beneficial only for 1 bug out of 17; the bug was a faulty algorithm with complex logic. It was discovered that mutation testing could bring other benefits, such as finding redundant test cases and improving code quality. However, there exist critical impediments for using mutation testing, such as the immaturity of the testing tool for Scala, high effort to integrate the tool into the development flow, and a large number of equivalent mutants. Overall, the minor benefits that mutation testing could give do not outweigh the effort required to use it at the company effectively.