Download: slides (PDF), slides (PowerPoint).
“Reproducible tests? Non-duplicable results in testing and verification” by Michael D. Ernst. In ICST 2012: Fifth International Conference on Software Testing, Verification and Validation (ICST), (Montreal, Canada), Apr. 2012.
Reproducibility is a central tenet of testing. Randomization in test outputs could mask the signal that indicates correctness, so engineers work to ensure that test execution is consistent. Proofs, too, must be reproducible: a proof is of little value unless it can be independently verified.
Evaluation of tools and processes does not meet the standards that engineers expect in their software. Random testing is sometimes found to be superior to, sometimes inferior to, systematic testing. High test-coverage goals are adopted by one organization but abandoned by another. Test-first development strategies help one project but cripple another. Formal development methods (based on specification and verification) sometimes reduce costs but other times increase them, with varying correlation to quality. Programmers sing the praises of improved productivity when adopting languages with strong type systems — or languages without static typing. There are also rifts between techniques that are shown effective in research laboratories and those that are adopted in practice: research experiments are often not indicative of effectiveness in the field. These discordant observations hold back our field by sowing confusion among researchers and doubt among practitioners, and by preventing common ground within or between the communities. The divergences continue to occur despite our best intentions, and despite our increasing sophistication in tool-building, evaluation, realistic codebases, education, bridging communities, and the like.
This talk will illustrate the scope of the problem with examples of conflicting results and experiences in the testing, verification, and validation community. It will discuss reasons for non-reproducibility — some of which are standard and acknowledged, and others of which are more subtle and easily overlooked. It will discuss ways to avoid or mitigate the problems. This talk aims to help the audience to recognize non-reproducible results in their own work or that of others, and to avoid them whether in research or in practice.
Download: slides (PDF), slides (PowerPoint).
BibTeX entry:
@inproceedings{Ernst2012:ICST,
author = {Michael D. Ernst},
title = {Reproducible tests? Non-duplicable results in testing and
verification},
booktitle = {ICST 2012: Fifth International Conference on Software
Testing, Verification and Validation (ICST)},
pages = {xlvix},
address = {Montreal, Canada},
month = apr,
year = {2012}
}
(This webpage was created with bibtex2web.)
Back to Michael Ernst's publications.