Comparing developer-provided to user-provided tests for fault localization and automated program repair

“Comparing developer-provided to user-provided tests for fault localization and automated program repair” by René Just, Chris Parnin, Ian Drosos, and Michael D. Ernst. In ISSTA 2018, Proceedings of the 2018 International Symposium on Software Testing and Analysis, (Amsterdam, Netherlands), July 2018.

Abstract

To realistically evaluate a software testing or debugging technique, the technique must be run on defects and tests that are characteristic of those a developer would encounter in practice. In general, this means that the evaluation should use real defects and tests.

For example, to determine the utility of a fault localization technique, it could be run on real defects from a bug tracking system, using real tests that are committed to the version control repository along with the fix. Although such a methodology uses real tests, it may not use tests that are representative of the information a developer or tool would have in practice. The tests that a developer commits after fixing a defect may encode more information than was available to the developer when initially diagnosing the defect. This calls into question the results of past empirical studies on the effectiveness of fault localization and automated program repair that used developer-provided tests.

This paper compares, both quantitatively and qualitatively, the developer-provided tests committed along with fixes (as found in the version control repository) versus the user-provided tests extracted from bug reports (as found in the issue tracker).

Our results provide evidence that developer-provided tests encode more information than available in user-provided tests. For fault localization, developer-provided tests consistently overestimate the ability to rank a defective statement in the list of the top-n most suspicious statements. For automated program repair, developer-provided tests overstate the effectiveness, because user-provided tests generate far fewer correct patches and substantially increase the repair time. We also provide a novel benchmark that contains tests extracted from bug reports, and we make suggestions for improving the design and evaluation of fault localization and automated program repair techniques.

BibTeX entry:

@inproceedings{JustPDE2018,
   author = {Ren{\'e} Just and Chris Parnin and Ian Drosos and Michael D.
	Ernst},
   title = {Comparing developer-provided to user-provided tests for fault
	localization and automated program repair},
   booktitle = {ISSTA 2018, Proceedings of the 2018 International
	Symposium on Software Testing and Analysis},
   address = {Amsterdam, Netherlands},
   month = jul,
   year = {2018}
}

(This webpage was created with bibtex2web.)

Back to Michael Ernst's publications.