The high-level goal of this exercise is to learn about automated debugging and how to use the delta debugging approach to systematically isolate failure-inducing inputs.
Team up in groups of size 2.
Assign yourself to the correct (In-class4-Debugging) group on Canvas. (If you are in a Canvas group of size 1, you can still submit.)
Make sure that you are using a Unix environment (e.g., attu.cs.washington.edu or OSX) or git bash for Windows for this exercise.
Clone the following Git repository: https://bitbucket.org/rjust/delta-debugging (Clone the repo into a directory whose absolute path does not contain any white spaces.)
Read the entire assignment and ask any clarifying questions that you might have.
In the top-level directory (delta-debugging), run:
perl mysort.pl passing.txt
perl mysort.pl failing.txt (does not terminate; use ctrl-c to interrupt)
A fellow developer hands you a legacy program (mysort.pl), written in Perl. The program works most of the time, but your fellow developer ran into a problem with it. In the spirit of following best practices, they provided instructions with a concrete test case that shows the problem (point 1 above).
Unfortunately, the provided test case isn’t quite minimal and your Perl skills are a little rusty. So, you decide to delta-debug (i.e., minimize) the test case first, before even thinking about the program itself. Luckily, you already have access to a delta-debugging implementation (delta-debugging/delta), so you “only” need to get it to work for this particular problem.
Use delta debugging to minimize failing.txt. A wrapper script dd-wrapper.sh for the delta command is provided, but you need to write a test oracle script that executes mysort.pl. Replace the <your test oracle script> in dd-wrapper.sh with your own script. Note that you can write your script in any way and any programming language you want. Also note that the delta command executes your test oracle script in a subdirectory like tmp0/arena/. You may need to take this into account when working with relative paths and calling mysort.pl from within your script.
Hints:
How will your test script handle the fact that mysort may not terminate?
What test outcome constitutes success for delta debugging?
You can find more information about the delta program under delta/doc/using_delta.txt.
If you can’t get your test oracle script to work within 15 minutes, ask the staff for help and move on.
Using the minimized test case, debug and fix the issue in mysort.pl. (See Questions 1-3 below)
Manually create two test cases (see Question 4 below) that trigger the problem in the original mysort.pl and that have the following properties:
both test cases have four lines;
test case 1 represents the best case scenario – that his, requires the fewest delta-debugging steps.
test case 2 represents the worst case scenario – that is, requires the most delta-debugging steps.
Delta debugging implementations differ in the order in which they test chunks of the input. Validate your test cases for the provided implementation.
What is the root cause of the bug in mysort.pl?
Give a one sentence explanation to characterize all test cases (input lists) on which mysort.pl fails because of the reason in 1.
Provide a fix to the bug as unified diff (use git diff).
Provide your two four-line test cases and, for each test case, list all steps of the delta-debugging algorithm (provided implementation). Briefly explain why these test cases are the best and worst case, respectively.
The test oracle script you wrote.
A plain-text file with your answers to the four questions above. Please list all group members.
One team member should upload the deliverables to Canvas, via the Canvas submission site for this course.