Download: PDF, slides (PDF), longer slide deck (PowerPoint), longer slide deck (PDF), technical report UW-CSE-16-08-03, data and scripts.
“Evaluating and improving fault localization”
by
Spencer Pearson,
José Campos,
René Just,
Gordon Fraser,
Rui Abreu,
Michael D. Ernst,
Deric Pang,
and Benjamin Keller.
In ICSE 2017, Proceedings of the 39th International Conference on
Software Engineering, (Buenos Aires, Argentina), May 2017, pp.
609-620.
An extended version appeared as
“Evaluating & improving fault localization techniques”
by
Spencer Pearson,
José Campos,
René Just,
Gordon Fraser,
Rui Abreu,
Michael D. Ernst,
Deric Pang,
and Benjamin Keller.
University of Washington Department of Computer Science and Engineering technical report UW-CSE-16-08-03, (Seattle, WA, USA), Sep. 2016. Revised February 2017.
Most fault localization techniques take as input a faulty program, and produce as output a ranked list of suspicious code locations at which the program may be defective. When researchers propose a new fault localization technique, they typically evaluate it on programs with known faults. The technique is scored based on where in its output list the defective code appears. This enables the comparison of multiple fault localization techniques to determine which one is better.
Previous research has evaluated fault localization techniques using artificial faults, generated either by mutation tools or manually. In other words, previous research has determined which fault localization techniques are best at finding artificial faults. However, it is not known which fault localization techniques are best at finding real faults. It is not obvious that the answer is the same, given previous work showing that artificial faults have both similarities to and differences from real faults.
We performed a replication study to evaluate 10 claims in the literature that compared fault localization techniques (from the spectrum-based and mutation-based families). We used 3242 artificial faults in 6 real-world programs. Our results support 7 of the previous claims as statistically significant, but only 3 as having non-negligible effect sizes. Then, we evaluated the same 10 claims, using 323 real faults from the 6 programs. Every previous result was refuted or was statistically and practically insignificant. Our experiments show that artificial faults are not useful for predicting which fault localization techniques perform best on real faults.
In light of these results, we identified a design space that includes many previously-studied fault localization techniques as well as hundreds of new techniques. We experimentally determined which factors in the design space are most important, using an overall set of 395 real faults. Then, we extended this design space with new techniques. Several of our novel techniques outperform all existing techniques, notably in terms of ranking defective code in the top-5 or top-10 reports.
Download: PDF, slides (PDF), longer slide deck (PowerPoint), longer slide deck (PDF), technical report UW-CSE-16-08-03, data and scripts.
BibTeX entry:
@inproceedings{PearsonCJFAEPK2017,
author = {Spencer Pearson and Jos{\'e} Campos and Ren{\'e} Just and
Gordon Fraser and Rui Abreu and Michael D. Ernst and Deric Pang
and Benjamin Keller},
title = {Evaluating and improving fault localization},
booktitle = {ICSE 2017, Proceedings of the 39th International
Conference on Software Engineering},
pages = {609-620},
address = {Buenos Aires, Argentina},
month = may,
year = {2017}
}
(This webpage was created with bibtex2web.)
Back to Michael Ernst's publications.