Download: PDF, slides (PDF), slides (Google Slides).
“Revisiting the relationship between fault detection, test adequacy criteria, and test set size” by Yiqun T. Chen, Rahul Gopinath, Anita Tadakamalla, Michael D. Ernst, Reid Holmes, Gordon Fraser, Paul Ammann, and René Just. In ASE 2020: Proceedings of the 35th Annual International Conference on Automated Software Engineering, (Melbourne, Australia), Sep. 2020, pp. 237-249.
The research community has long recognized a complex interrelationship between fault detection, test adequacy criteria, and test set size. However, there is substantial confusion about whether and how to experimentally control for test set size when assessing how well an adequacy criterion is correlated with fault detection and when comparing test adequacy criteria. Resolving the confusion, this paper makes the following contributions: (1) A review of contradictory analyses of the relationships between fault detection, test adequacy criteria, and test set size. Specifically, this paper addresses the supposed contradiction of prior work and explains why test set size is neither a confounding variable, as previously suggested, nor an independent variable that should be experimentally manipulated. (2) An explication and discussion of the experimental designs of prior work, together with a discussion of conceptual and statistical problems, as well as specific guidelines for future work. (3) A methodology for comparing test adequacy criteria on an equal basis, which accounts for test set size without directly manipulating it through unrealistic stratification. (4) An empirical evaluation that compares the effectiveness of coverage-based testing, mutation-based testing, and random testing. Additionally, this paper proposes probabilistic coupling, a methodology for assessing the representativeness of a set of test goals for a given fault and for approximating the fault-detection probability of adequate test sets.
Download: PDF, slides (PDF), slides (Google Slides).
BibTeX entry:
@inproceedings{ChenGTEHFAJ2020,
author = {Yiqun T. Chen and Rahul Gopinath and Anita Tadakamalla and
Michael D. Ernst and Reid Holmes and Gordon Fraser and Paul Ammann
and Ren{\'e} Just},
title = {Revisiting the relationship between fault detection, test
adequacy criteria, and test set size},
booktitle = {ASE 2020: Proceedings of the 35th Annual International
Conference on Automated Software Engineering},
pages = {237-249},
address = {Melbourne, Australia},
month = sep,
year = {2020}
}
(This webpage was created with bibtex2web.)
Back to Michael Ernst's publications.