Call Me Maybe: Using NLP to automatically generate unit test cases respecting temporal constraints

Download: PDF, slides (PDF), slides (Keynote), implementation & replication package.

“Call Me Maybe: Using NLP to automatically generate unit test cases respecting temporal constraints” by Arianna Blasi, Alessandra Gorla, Michael D. Ernst, and Mauro Pezzè. In ASE 2022: Proceedings of the 37th Annual International Conference on Automated Software Engineering, (Oakland Center, MI, USA), Oct. 2022, pp. 1-11.

Abstract

A class may need to obey temporal constraints in order to function correctly. For example, the correct usage protocol for an iterator is to always check whether there is a next element before asking for it; iterating over a collection when there are no items left leads to a NoSuchElementException. Automatic test case generation tools such as Randoop and EvoSuite do not have any notion of these temporal constraints. Generating test cases by randomly invoking methods on a new instance of the class under test may raise run time exceptions that do not necessarily expose software faults, but are rather a consequence of violations of temporal properties.

This paper presents CallMeMaybe, a novel technique that uses natural language processing to analyze Javadoc comments to identify temporal constraints. This information can guide a test case generator towards executing sequences of method calls that respect the temporal constraints. Our evaluation on 73 subjects from seven popular Java systems shows that CallMeMaybe achieves a precision of 83% and a recall of 70% when translating temporal constraints into Java expressions. For the two biggest subjects, the integration with Randoop flags 11,818 false alarms and enriches 12,024 correctly failing test cases due to violations of temporal constraints with clear explanation that can help software developers.

Download: PDF, slides (PDF), slides (Keynote), implementation & replication package.

BibTeX entry:

@inproceedings{BlasiGEP2022,
   author = {Arianna Blasi and Alessandra Gorla and Michael D. Ernst and
	Mauro Pezz{\`e}},
   title = {{Call Me Maybe}: Using {NLP} to automatically generate unit
	test cases respecting temporal constraints},
   booktitle = {ASE 2022: Proceedings of the 37th Annual International
	Conference on Automated Software Engineering},
   pages = {1-11},
   address = {Oakland Center, MI, USA},
   month = oct,
   year = {2022}
}

(This webpage was created with bibtex2web.)

Back to Michael Ernst's publications.