The high-level goals of this exercise are to (1) learn about mutation testing and (2) reason about test-goal utility.
Team up in groups of size 2.
Assign yourself to the correct (In-class3-Mutation) group on Canvas. (If you are in a Canvas group of size 1, you can still submit.)
Make sure Apache Ant, a Java 8 or 11 or 17 JDK, and Git are installed. (The set-up tips for Ant may be useful.) Use a Unix environment or Git bash on Windows for this exercise. The required software is already installed on attu.cs.washington.edu, if you prefer to do the exercise there.
Clone the following git repository and read its README.md file: https://bitbucket.org/rjust/mutation
Test your set up: compile and test the Triangle program.
Run mutation.sh (make sure that the script works as intended).
Read the entire assignment and ask any clarifying questions that you might have.
Incorporate your MCDC-adequate test suite from the prior in-class exercise into the testTable in TriangleTest.java.
Run mutation.sh and note the number of covered and detected mutants (see Question 1 below). Note that the mutation tool (Major) refers to “detected mutants” as “killed mutants”.
Add tests to testTable to satisfy mutation adequacy – that is, until your test suite detects all non-equivalent mutants:
Select a live mutant (for which you have not proven equivalence) for analysis.
Determine whether it is equivalent – that is, provide a test case or proof of equivalence (see Questions 2 and 3 below).
Run mutation.sh and continue with step a.
You may find the show_mutant.sh script useful for reasoning about a mutant.
Note that you will likely observe certain patterns (i.e., similar mutants requiring similar tests) because of the systematic mutation of the source code – adding multiple tests at once may speed up your testing process. Likewise, some mutants are easier to resolve than others – triaging the set of live mutants and selecting mutants out of order may speed up your testing process.
If you get stuck on a mutant, take notes, move on, and revisit unresolved mutants later.
Disable (comment out) the assertEquals statement in the testTriangle method and run ant coverage and mutation.sh. Note the code coverage ratio(s) and mutation score (see Question 4 below).
Extra credit: Re-enable the assertEquals statement in the testTriangle method and set exportKillMap to true in the build.xml file. Rerun mutation.sh, which now computes killMap.csv. Determine the set of dominator mutants (see Question 5 below).
How many mutants does your coverage-adequate test suite from instruction #2 above (1) cover and (2) detect?
How many mutants are equivalent (to the original program)? Provide a proof for each equivalent mutant.
Note that you do not need to provide a formal proof of equivalence, but you should demonstrate proper reasoning and provide a valid argument. You may group equivalent mutants in your answer if the reason for equivalence is the same for all mutants in a group (e.g., mutants 4711, 4712, 4713 are equivalent because they exist in dead code).
Were any of the generated mutants unproductive? Briefly explain your answer.
What changes in code coverage ratio and mutation score did you observe after disabling the assertEquals statement in the testTriangle method? What are the implications for using the code coverage ratio as an adequacy criterion?
Given the killMap.csv (mutant-test detection matrix), determine the set of dominator mutants in the DMSG (dynamic mutant subsumption graph) that can be derived from the mutant-test detection matrix. You may automate this analysis.
A plain-text file (or PDF) with your answers to the four questions above. Please list all group members.
Your mutation-adequate TriangleTest.java test suite.
One team member should upload the deliverables to Canvas, via the Canvas submission site for this course.
ant clean test to make sure that your added tests pass on the original program.