The high-level goals of this exercise are to (1) learn about mutation testing and (2) reason about test-goal utility.
Team up in groups of size 2.
Assign yourself to the correct (in-class-3-testing) group on Canvas. (You may work and submit alone, but you must still self-assign to a group on Canvas!)
Make sure Apache Ant, a Java 8 or 11 or 17 JDK, and Git are installed. (The set-up tips for Ant may be useful.) Use a Unix environment or Git bash on Windows for this exercise. The required software is already installed on attu.cs.washington.edu, if you prefer to do the exercise there.
Clone the following git repository and read its README.md file: https://bitbucket.org/rjust/mutation
Test your set up: compile and test the Triangle program.
Run mutation.sh (make sure that the script works as intended).
You may use an experimental IntelliJ plugin for this exercise. See the detailed instructions for more details.
Read the entire assignment and ask any clarifying questions that you might have.
Run mutation.sh and note the number of covered and detected mutants (see Question 1 below). Note that Major (the mutation tool) refers to “detected mutants” as “killed mutants”.
Add tests to testTable to satisfy mutation adequacy – that is, until your test suite detects all non-equivalent mutants:
Select a live mutant (for which you have not proven equivalence) for analysis.
Determine whether it is equivalent – that is, provide a test case or proof of equivalence (see Questions 2 and 3 below).
Run mutation.sh and continue with step a.
You may find the show_mutant.sh script useful for reasoning about a mutant.
Note that you will likely observe certain patterns (i.e., similar mutants requiring similar tests) because of the systematic mutation of the source code – adding multiple tests at once may speed up your testing process. Likewise, some mutants are easier to resolve than others – triaging the set of live mutants and selecting mutants out of order may speed up your testing process.
If you get stuck on a mutant, take notes, move on, and revisit unresolved mutants later.
Disable (comment out) the assertEquals statement on line 45 in the testTriangle method and run ant coverage and mutation.sh. Note the code coverage ratio(s) and mutation score (see Question 4 below).
How many mutants does the initial test suite (1) cover and (2) detect (result from step 1 in the instructions)?
How many mutants are equivalent (to the original program)? Provide a proof for each equivalent mutant.
Were any of the generated mutants unproductive? Briefly explain your answer.
What changes in code coverage ratio and mutation score did you observe after disabling the assertEquals statement in the testTriangle method? What are the implications for using the code coverage ratio as an adequacy criterion?
A plain-text file with your answers to the four questions above. Please list all group members.
Your mutation-adequate TriangleTest.java test suite.
Your <timestamp>.csv file, if you used the IntelliJ plugin.
One team member should upload the deliverables to Canvas, via the Canvas submission site for this course.
To avoid spurious results and confusion, run ant clean test after adding a new tests to ensure that it passes on the original program.
You do not need to provide a formal proof of equivalence, but you should demonstrate proper reasoning and provide a valid argument. You may group equivalent mutants in your answer if the reason for equivalence is the same for all mutants in a group (e.g., mutants 4711, 4712, 4713 are equivalent because they exist in dead code).