In-class exercise: Invariants and partial oracles

Goal

The high-level goal of this two-part exercise is to learn about program invariants and partial test oracles (property-based testing and metamorphic testing).

Instructions

  1. Team up in groups of size 2, and self-assign to a group (in the In-class-Invariants groupset) on Canvas. (If you are in a Canvas group of size 1, you can still submit.)

  2. Follow the instructions and answer the corresponding questions of the two parts below.

  3. Upload the deliverables to Canvas. Only one team member needs to upload the deliverables to Canvas. Please list all group members in the deliverable(s).

Part 1: Invariants

Instructions

  • Familiarize yourself with the Daikon invariant detector

  • Run Daikon on a data structure example of your choice (other than the Stack example used in class). You may choose to run Daikon for any supported programming language (see installation instructions). (The example demoed in class used Daikon 5.9.0 and a Java 25 JDK.)

Questions

  1. Provide a link to the chosen data structure. For this data structure and the corresponding Daikon output, give one example for a true invariant and one example for a discovered invariant that only holds for the given test suite.

  2. Briefly explain which (if any) types of invariants discovered by Daikon are generally useful for testing and/or debugging (alternatively, briefly explain which types are not useful). There are no wrong answers: the provided answer is evaluated based on its justification not the expressed believe.

  3. For a software system that you are familiar with, briefly explain that system and state at least two types of invariants of interest (useful for testing and/or debugging). Briefly explain whether and how these invariants can be dynamically discovered.

Part 2: Testing with partial oracles

This part has two variants (A and B). You may complete just one of them or both.

Instructions

Variant A

  • Clone an open-source repository of your choice for a non-trivial software system that is amenable to metamorphic testing. The software system may be written in any programming language, but it must provide a build system for compilation and testing.

  • For the chosen software system, design two metamorphic relations (MRs):

    • The two MRs must hold for arbitrary, well-formed inputs.
    • For at least one MR, the output relation must not be the identity function.

  • Implement your two MRs as test-case generators, which take an initial test case as input and produce a follow-up test case as output.

  • Based on the designed MRs, automatically generate a few test cases and execute them. (For the initial test cases, you may rely on random input generation, any existing test cases, or manually created test cases.)

  • Extra credit: Assess the efficacy of the designed MRs using mutation analysis. That is, mutate the software system and compute the mutant-detection rate for the test cases generated by the MRs. (Mutation-analysis frameworks are available for many languages, including Java, C#, JavaScript, and Python.)

Variant B

  • Choose a non-trivial online software system (e.g., for classification of text, images, code, etc.).

  • For the chosen software system, design three metamorphic relations (MRs):

    • The three MRs must hold for arbitrary, well-formed inputs.
    • For at least two MRs, the output relation must not be the identity function.

  • Based on the designed MRs, generate a few test cases, execute the online system, and observe its output. (You may automate the test-case generation.)

Questions

  1. Provide a link to the chosen software system and briefly describe it.

  2. Justify for each of the designed MRs why it must hold for arbitrary, well-formed inputs. Include sufficient details and optionally test code to allow others to reproduce your testing approach.

  3. Briefly summarize your observations about testing with MRs and state at least two characteristics of effective MRs. You may reason about the specific MRs you designed or MRs in general.

Deliverables

  1. A plain-text (txt or md) file with your answers to the six questions above. Please list all group members.