Download: PDF, slides (PowerPoint), talk video (YouTube), implementation, data.
“Dependent-test-aware regression testing techniques” by Wing Lam, August Shi, Reed Oei, Sai Zhang, Michael D. Ernst, and Tao Xie. In ISSTA 2020, Proceedings of the 2020 International Symposium on Software Testing and Analysis, (Los Angeles, CA, USA), July 2020, pp. 298-311.
Developers check their changes using regression testing techniques. Unfortunately, regression testing techniques suffer from flaky tests, which can both pass and fail when run multiple times on the same version of code and tests. While many types of flaky tests exist, one prominent type is dependent tests, which are tests that pass when run in one order but fail when run in another order. Although dependent tests may cause flaky test failures, dependent tests can help developers run their tests faster. Since developers may still want dependent tests, we propose to make regression testing techniques dependent-test-aware to reduce flaky test failures.
To understand the necessity of dependent-test-aware regression testing techniques, we conduct the first study on the impact of dependent tests on three regression testing techniques: test prioritization, test selection, and test parallelization. In particular, we implement 4 test prioritization, 6 test selection, and 2 test parallelization algorithms, and we evaluate them on 11 Java modules with dependent tests. When we run the orders produced by the traditional, dependent-test-unaware regression testing algorithms, 90% of the human-written test suites with dependent tests have at least one flaky test failure, while 100% of the automatically-generated test suites have at least one flaky test failure.
We develop a general approach for enhancing regression testing algorithms to make them dependent-test-aware, and we apply our approach to enhance 12 algorithms. Compared to traditional, unenhanced regression testing algorithms, the enhanced algorithms use provided test dependencies to produce different orders or orders with extra tests. Our evaluation shows that, in comparison to the orders produced by the unenhanced algorithms, the orders produced by the enhanced algorithms (1) have overall 65% fewer flaky test failures due to dependent tests, and (2) may add extra tests but run only <1% slower on average. Our results suggest that enhancing regression testing algorithms to be dependent-test-aware can substantially reduce flaky test failures with only a minor slowdown to run the tests.
Download: PDF, slides (PowerPoint), talk video (YouTube), implementation, data.
BibTeX entry:
@inproceedings{LamSOZEX2020,
author = {Wing Lam and August Shi and Reed Oei and Sai Zhang and
Michael D. Ernst and Tao Xie},
title = {Dependent-test-aware regression testing techniques},
booktitle = {ISSTA 2020, Proceedings of the 2020 International
Symposium on Software Testing and Analysis},
pages = {298-311},
address = {Los Angeles, CA, USA},
month = jul,
year = {2020}
}
(This webpage was created with bibtex2web.)
Back to Michael Ernst's publications.