When tests collide: Evaluating and coping with the impact of test dependence

Download: PDF.

“When tests collide: Evaluating and coping with the impact of test dependence” by Wing Lam, Sai Zhang, and Michael D. Ernst. University of Washington Department of Computer Science and Engineering technical report UW-CSE-15-03-01, (Seattle, WA, USA), Mar. 2015.

Abstract

In a test suite, all the test cases should be independent: no test should affect any other test's result, and running the tests in any order should produce the same test results. The assumption of test independence is important so that tests behave consistently as designed. In addition, many downstream testing techniques, including test prioritization, test selection, and test parallelization, assume test independence. However, this critical assumption often does not hold in practice.

This paper empirically investigates the impact of test dependence on three downstream testing techniques (test prioritization, selection, and parallelization) and proposes a general approach to cope with such impact. It presents two sets of results.

First, we describe an empirical study to assess the impact of test dependence on 4 test prioritization, 6 test selection, and 2 test parallelization algorithms. Test dependence negatively affects the results of all these downstream testing algorithms. For example, an automatically-generated test suite for the XML-Security program contains 665 tests, and 111 of those tests yield a different test result (success vs. fail) if the suite is parallelized to run on 16 CPUs.

Second, we present an approach that enhances each test prioritization, selection, and parallelization algorithm to respect test dependence, so that each test in a suite yields the same result before and after applying the downstream testing technique. In an experimental evaluation, the enhanced testing algorithms worked as intended: the test results were consistent even in the presence of test dependence, and they did not substantially compromise the effectiveness of the original testing algorithms.

Download: PDF.

BibTeX entry:

@techreport{LamZE2015,
   author = {Wing Lam and Sai Zhang and Michael D. Ernst},
   title = {When tests collide: Evaluating and coping with the impact of
	test dependence},
   institution = {University of Washington Department of Computer Science
	and Engineering},
   number = {UW-CSE-15-03-01},
   address = {Seattle, WA, USA},
   month = mar,
   year = {2015}
}

(This webpage was created with bibtex2web.)

Back to Michael Ernst's publications.