Evaluating & improving fault localization techniques

Download: technical report UW-CSE-16-08-03, data and scripts.

“Evaluating & improving fault localization techniques” by Spencer Pearson, José Campos, René Just, Gordon Fraser, Rui Abreu, Michael D. Ernst, Deric Pang, and Benjamin Keller. In ICSE'17, Proceedings of the 39th International Conference on Software Engineering, (Buenos Aires, Argentina), May 24-26, 2017.
An extended version appeared as University of Washington Department of Computer Science and Engineering technical report UW-CSE-16-08-03, (Seattle, WA, USA), Sep. 2016.

Abstract

A fault localization technique takes as input a faulty program, and it produces as output a ranked list of suspicious code locations at which the program may be defective. When researchers propose a new fault localization technique, they evaluate it on programs with known faults; they score the technique based on where in its output list the defective code appears. This enables comparison of multiple fault localization techniques to determine which one is better.

Previous research has evaluated fault localization techniques using artificial faults, generated either by mutation tools or manually. In other words, previous research has determined which fault localization techniques are best at finding artificial faults. However, it is not known which fault localization techniques are best at finding real faults. It is not obvious that the answer is the same, given previous work showing that artificial faults have both similarities to and differences from real faults.

We performed a replication study to evaluate 10 claims in the literature that compared fault localization techniques. We used 2273 artificial faults in 5 real-world programs. Our results refute 3 of the previous claims. Then, we evaluated the same 10 claims, using 297 real faults from the 5 programs. Every previous result was refuted or was statistically insignificant. In other words, our experiments show that artificial faults are not useful for predicting which fault localization techniques perform best on real faults.

In light of these results, we identified a design space that includes many previously-studied fault localization techniques as well as hundreds of new techniques. We experimentally determined which factors in the design space are most important. Then, we extended it with new techniques. Several of our novel techniques outperform all existing techniques, notably in terms of ranking defective code in the top-5 or top-10 reports.

Download: technical report UW-CSE-16-08-03, data and scripts.

BibTeX entry:

@inproceedings{PearsonCJFAEPK2017,
   author = {Spencer Pearson and Jos{\'e} Campos and Ren{\'e} Just and
	Gordon Fraser and Rui Abreu and Michael D. Ernst and Deric Pang
	and Benjamin Keller},
   title = {Evaluating \& improving fault localization techniques},
   booktitle = {ICSE'17, Proceedings of the 39th International Conference
	on Software Engineering},
   address = {Buenos Aires, Argentina},
   month = {May~24--26,},
   year = {2017}
}

(This webpage was created with bibtex2web.)

Back to Michael Ernst's publications.