Download: PDF, slides (PDF), Defects4J subject programs, Major mutation tool.
“Are mutants a valid substitute for real faults in software
testing?”
by
René Just,
Darioush Jalali,
Laura Inozemtseva,
Michael D. Ernst,
Reid Holmes,
and
Gordon Fraser.
In FSE 2014: Proceedings of the ACM SIGSOFT 22nd Symposium on the
Foundations of Software Engineering, (Hong Kong), Nov. 2014, pp.
654-665.
A previous version appeared as University of Washington Department
of Computer Science and Engineering technical report UW-CSE-14-02-02,
(Seattle, WA, USA), Mar. 2014.
A good test suite is one that detects real faults. Because the set of faults in a program is usually unknowable, this definition is not useful to practitioners who are creating test suites, nor to researchers who are creating and evaluating tools that generate test suites. In place of real faults, testing research often uses mutants, which are artificial faults — each one a simple syntactic variation — that are systematically seeded throughout the program under test. Mutation analysis is appealing because large numbers of mutants can be automatically-generated and used to compensate for low quantities or the absence of known real faults.
Unfortunately, there is little experimental evidence to support the use of mutants as a replacement for real faults. This paper investigates whether mutants are indeed a valid substitute for real faults, i.e., whether a test suite's ability to detect mutants is correlated with its ability to detect real faults that developers have fixed. Unlike prior studies, these investigations also explicitly consider the conflating effects of code coverage on the mutant detection rate.
Our experiments used 357 real faults in 5 open-source applications that comprise a total of 321,000 lines of code. Furthermore, our experiments used both developer-written and automatically-generated test suites. The results show a statistically significant correlation between mutant detection and real fault detection, independently of code coverage. The results also give concrete suggestions on how to improve mutation analysis and reveal some inherent limitations.
Download: PDF, slides (PDF), Defects4J subject programs, Major mutation tool.
BibTeX entry:
@inproceedings{JustJIEHF2014,
author = {Ren{\'e} Just and Darioush Jalali and Laura Inozemtseva and
Michael D. Ernst and Reid Holmes and Gordon Fraser},
title = {Are mutants a valid substitute for real faults in software
testing?},
booktitle = {FSE 2014: Proceedings of the {ACM} {SIGSOFT} 22nd
Symposium on the Foundations of Software Engineering},
pages = {654--665},
address = {Hong Kong},
month = nov,
year = {2014}
}
(This webpage was created with bibtex2web.)
Back to Michael Ernst's publications.