Empirically revisiting the test independence assumption

Download: PDF, slides (PowerPoint).

“Empirically revisiting the test independence assumption” by Sai Zhang, Darioush Jalali, Jochen Wuttke, Kıvanç Muşlu, Wing Lam, Michael D. Ernst, and David Notkin. In ISSTA 2014, Proceedings of the 2014 International Symposium on Software Testing and Analysis, (San Jose, CA, USA), July 2014, pp. 385-396.
A previous version appeared as University of Washington Department of Computer Science and Engineering technical report UW-CSE-14-01-01, (Seattle, WA, USA), Jan. 2014.

Abstract

In a test suite, all the test cases should be independent: no test should affect any other test's result, and running the tests in any order should produce the same test results. Techniques such as test prioritization generally assume that the tests in a suite are independent. Test dependence is a little-studied phenomenon. This paper presents five results related to test dependence.

First, we characterize the test dependence that arises in practice. We studied 96 real-world dependent tests from 5 issue tracking systems. Our study shows that test dependence can be hard for programmers to identify. It also shows that test dependence can cause non-trivial consequences, such as masking program faults and leading to spurious bug reports.

Second, we formally define test dependence in terms of test suites as ordered sequences of tests along with explicit environments in which these tests are executed. We formulate the problem of detecting dependent tests and prove that a useful special case is NP-complete.

Third, guided by the study of real-world dependent tests, we propose and compare four algorithms to detect dependent tests in a test suite.

Fourth, we applied our dependent test detection algorithms to 4 real-world programs and found dependent tests in each human-written and automatically-generated test suite.

Fifth, we empirically assessed the impact of dependent tests on five test prioritization techniques. Dependent tests affect the output of all five techniques; that is, the reordered suite fails even though the original suite did not.

Download: PDF, slides (PowerPoint).

BibTeX entry:

@inproceedings{ZhangJWMLEN2014,
   author = {Sai Zhang and Darioush Jalali and Jochen Wuttke and
	K{\i}van{\c{c}} Mu{\c{s}}lu and Wing Lam and Michael D. Ernst and
	David Notkin},
   title = {Empirically revisiting the test independence assumption},
   booktitle = {ISSTA 2014, Proceedings of the 2014 International
	Symposium on Software Testing and Analysis},
   pages = {385--396},
   address = {San Jose, CA, USA},
   month = jul,
   year = {2014}
}

(This webpage was created with bibtex2web.)

Back to Michael Ernst's publications.