The high-level goal of this exercise is to learn about automated debugging and how to use the delta debugging approach to systematically isolate failure-inducing inputs.
Team up in groups of size 2.
Assign yourself to the correct (in-class-4-debugging) group on Canvas. (You may work and submit alone, but you must still self-assign to a group on Canvas!)
Make sure that you are using a Unix environment (e.g., attu.cs.washington.edu or OSX) or git bash for Windows for this exercise.
Clone the following Git repository: https://bitbucket.org/rjust/delta-debugging (Clone the repository into a directory whose absolute path does not contain any white spaces.)
Read the entire assignment and ask any clarifying questions that you might have.
In the top-level directory (delta-debugging), run:
perl mysort.pl passing.txt
perl mysort.pl failing.txt (does not terminate; use ctrl-c to interrupt)
A fellow developer hands you a legacy program (mysort.pl), written in Perl. The program works most of the time, but your fellow developer ran into a problem with it. In the spirit of following best practices, they provided instructions with a concrete test case that shows the problem (point 1 above).
Unfortunately, the provided test case isn’t quite minimal and your Perl skills are a little rusty. So, you decide to delta-debug (i.e., minimize) the test case first, before even thinking about the program itself. Luckily, you already have access to a delta-debugging implementation (delta-debugging/delta), so you “only” need to get it to work for this particular problem.
Use delta debugging to minimize failing.txt. A wrapper script dd-wrapper.sh for the delta command is provided, but you need to write a test oracle script that executes mysort.pl. Replace the <your test oracle script> in dd-wrapper.sh with your own script.
Hints:
How will your test script handle the fact that mysort may not terminate?
What test outcome constitutes success for delta debugging?
You can write your script in any way and any programming language.
The delta command executes your test oracle script in a subdirectory like tmp0/arena/. Take this into account when working with relative paths and calling mysort.pl from within your script.
You can find more information about the delta program under delta/doc/using_delta.txt.
If you can’t get your test oracle script to work within 15 minutes, ask the staff for help and move on.
Using the minimized test case, debug and fix the issue in mysort.pl. (See Questions 1-3 below)
Manually create two test cases (input lists) that trigger the problem in the original mysort.pl and that have the following properties (see Question 4 below) :
both test cases have four lines;
test case 1 represents the best case scenario – that his, requires the fewest delta-debugging steps.
test case 2 represents the worst case scenario – that is, requires the most delta-debugging steps.
Delta debugging implementations differ in the order in which they evaluate subsets and complements of the input. Validate your test cases for the provided delta-debugging implementation.
What is the root cause of the bug in mysort.pl?
Give a one sentence explanation to characterize all test cases (input lists) on which mysort.pl fails because of the root cause in Question 1.
Provide a fix to the bug as unified diff (use git diff).
Provide your two four-line test cases and:
briefly explain why these test cases are the best and worst case, respectively;
for each test case, list all inputs (subsets and complements) evaluated by the provided delta-debugging implementation (in order of evaluation);
for each test case and corresponding list of evaluated inputs, identify the steps that correspond to “increase granularity”, “reduce to subset”, and “reduce to complement”.
A plain-text file with your answers to the four questions above. Please list all group members.
The test oracle script you wrote.
One team member should upload the deliverables to Canvas, via the Canvas submission site for this course.