An experimental evaluation of continuous testing during development

Download: PDF, slides (PDF), slides (PowerPoint), Eclipse plug-in.

“An experimental evaluation of continuous testing during development” by David Saff and Michael D. Ernst. In ISSTA 2004, Proceedings of the 2004 International Symposium on Software Testing and Analysis, (Boston, MA, USA), July 2004, pp. 76-85.

Abstract

Continuous testing uses excess cycles on a developer's workstation to continuously run regression tests in the background, providing rapid feedback about test failures as source code is edited. It is intended to reduce the time and energy required to keep code well-tested and prevent regression errors from persisting uncaught for long periods of time. This paper reports on a controlled human experiment to evaluate whether students using continuous testing are more successful in completing programming assignments. We also summarize users' subjective impressions and discuss why the results may generalize.

The experiment indicates that the tool has a statistically significant effect on success in completing a programming task, but no such effect on time worked. Participants using continuous testing were three times as likely to complete the task before the deadline. Participants using continuous compilation were twice as likely to complete the task, providing empirical support to a common feature in modern development environments. Most participants found continuous testing to be useful and believed that it helped them write better code faster, and 90% would recommend the tool to others. The participants did not find the tool distracting, and intuitively developed ways of incorporating the feedback into their workflow.

Download: PDF, slides (PDF), slides (PowerPoint), Eclipse plug-in.

BibTeX entry:

@inproceedings{SaffE2004:ISSTA,
   author = {David Saff and Michael D. Ernst},
   title = {An experimental evaluation of continuous testing during
	development},
   booktitle = {ISSTA 2004, Proceedings of the 2004 International
	Symposium on Software Testing and Analysis},
   pages = {76--85},
   address = {Boston, MA, USA},
   month = jul,
   year = {2004}
}

(This webpage was created with bibtex2web.)

Back to Michael Ernst's publications.