Download: PDF, slides (PDF), slides (Google Slides), data.
“Comparing developer-provided to user-provided tests for fault localization and automated program repair” by René Just, Chris Parnin, Ian Drosos, and Michael D. Ernst. In ISSTA 2018, Proceedings of the 2018 International Symposium on Software Testing and Analysis, (Amsterdam, Netherlands), July 2018, pp. 287-297.
To realistically evaluate a software testing or debugging technique, it must be run on defects and tests that are characteristic of those a developer would encounter in practice. For example, to determine the utility of a fault localization or automated program repair technique, it could be run on real defects from a bug tracking system, using real tests that are committed to the version control repository along with the fixes. Although such a methodology uses real tests, it may not use tests that are characteristic of the information a developer or tool would have in practice. The tests that a developer commits after fixing a defect may encode more information than was available to the developer when initially diagnosing the defect.
This paper compares, both quantitatively and qualitatively, the developer-provided tests committed along with fixes (as found in the version control repository) versus the user-provided tests extracted from bug reports (as found in the issue tracker). It provides evidence that developer-provided tests are more targeted toward the defect and encode more information than user-provided tests. For fault localization, developer-provided tests overestimate a technique's ability to rank a defective statement in the list of the top-n most suspicious statements. For automated program repair, developer-provided tests overestimate a technique's ability to (efficiently) generate correct patches — user-provided tests lead to fewer correct patches and increased repair time. This paper also provides suggestions for improving the design and evaluation of fault localization and automated program repair techniques.
Download: PDF, slides (PDF), slides (Google Slides), data.
BibTeX entry:
@inproceedings{JustPDE2018,
author = {Ren{\'e} Just and Chris Parnin and Ian Drosos and Michael D.
Ernst},
title = {Comparing developer-provided to user-provided tests for fault
localization and automated program repair},
booktitle = {ISSTA 2018, Proceedings of the 2018 International
Symposium on Software Testing and Analysis},
pages = {287-297},
address = {Amsterdam, Netherlands},
month = jul,
year = {2018}
}
(This webpage was created with bibtex2web.)
Back to Michael Ernst's publications.