In-class exercise: Mutation testing

High-level goal

The high-level goals of this exercise are to (1) learn about mutation testing and (2) reason about test-goal utility.

Team up in groups of size 2.
Assign yourself to the correct (In-class-Mutation) group on Canvas. (If you are in a Canvas group of size 1, you can still submit.)
Important: Make sure a Java 8 or 11 or 17 or 21 JDK, and Git are installed. Use a Unix environment or Git bash on Windows for this exercise. The required software is already installed on attu.cs.washington.edu, if you prefer to do the exercise there.
Clone the following git repository and read its README.md file: https://bitbucket.org/rjust/mutation
Test your set up: compile and test the Triangle program.
Run mutation.sh (make sure that the script works as intended).

Read the entire assignment and ask any clarifying questions that you might have.
Run mutation.sh and note the number of covered and detected mutants (see Question 1 below). Note that Major (the mutation tool) refers to “detected mutants” as “killed mutants”.
Add tests to testTable to satisfy mutation adequacy – that is, until your test suite detects all non-equivalent mutants:
1. Select a live mutant (for which you have not proven equivalence) for analysis.
2. Determine whether it is equivalent – that is, provide a test case that detects it or a proof of equivalence (see Questions 2 and 3 below).
3. Rerun ./gradlew clean test to verify test correctness.
4. Run mutation.sh and continue with step a.
You may find the show_mutant.sh script useful for reasoning about a mutant.

Note that you will likely observe certain patterns (i.e., similar mutants requiring similar tests) because of the systematic mutation of the source code – adding multiple tests at once may speed up your testing process. Likewise, some mutants are easier to resolve than others – triaging the set of live mutants and selecting mutants out of order may speed up your testing process.

If you get stuck on a mutant, take notes, move on, and revisit unresolved mutants later.
Disable (comment out) the assertEquals statement on line 45 in the testTriangle method and run ./gradlew test and mutation.sh. Note the code coverage ratio(s) and mutation score (see Question 4 below).

How many mutants does the initial test suite (1) cover and (2) detect (result from step 1 in the instructions)?
How many mutants are equivalent (to the original program)? Provide a proof for each equivalent mutant.
Were any of the generated mutants unproductive? Briefly explain your answer.
What changes in code coverage ratio and mutation score did you observe after disabling the assertEquals statement in the testTriangle method? What are the implications for using the code coverage ratio as an adequacy criterion?
Extra credit: Set exportKillMap=true in mutation.xml, and rerun ./mutation.sh. Then, given the mutation_results/killMap.csv file (mutant-test detection matrix), determine the set of dominator mutants in the DMSG (dynamic mutant subsumption graph) that can be derived from the mutant-test detection matrix. You may automate this analysis.

A plain-text file with your answers to the four questions above. Please list all group members.
Your mutation-adequate TriangleTest.java test suite.

One team member should upload the deliverables to Canvas.

To avoid spurious results and confusion, run ./gradlew clean test after adding a new tests to ensure that it passes on the original program.
You do not need to provide a formal proof of equivalence, but you should demonstrate proper reasoning and provide a valid argument. You may group equivalent mutants in your answer if the reason for equivalence is the same for all mutants in a group (e.g., mutants 4711, 4712, 4713 are equivalent because they exist in dead code).