• The big idea in the paper seems to be that of iteratively refining the model, and adding more predicates as needed. Given that the complexity grows very badly with the number of predicates, this seems to be a neat idea.
• The language for expressing safety properties seems complicated. Is the full power of an FSM really necessary? Perhaps safety properties could be expressed as statements of first-order predicate logic. Also, I thought it was unclear from the paper whether or not the use of function pointers would add complexity.
• The evaluation section of the paper is quite weak. It does not provide an example program in which SLAM reports a violation.
• It would be interesting to see SLAM applied to Java programs. The fact that SLAM can only handle a logical model of memory should pose no problem in Java, although polymorphism might (again, it’s not clear what SLAM does for function pointers).
• I think the main limitation of the paper is its poor analysis of the heap. SLAM gains some precision from the alias analysis of Das. However, given the ad hoc way that the NEWTON tool generates predicates, I find it hard to believe that SLAM would be able to prove properties about heap data structures. For example, if a set of locks is stored in the heap, SLAM's lock analysis probably becomes imprecise when it tries to disambiguate them.
• Iterative refinement really interested me. However, it's not clear from the paper how sophisticated the predicates can become. For example, if the heap is modeled as a memory store, and accesses are made using the typical select() and update() functions, would SLAM be able to generate arbitrarily complex abstractions for the heap? However, I imagine that if the abstraction becomes too complicated, then SLAM will have termination problems. Iterative refinement determines loop invariants, so if the predicates become complex, then SLAM would probably fail to terminate (as loop invariants are known to be hard to infer). In that case, it might be interesting to consider applying some sort of widening to the set of predicates that NEWTON generates.
• One possibility for decreasing the number of refinement iterations is to expand the initial predicate set. Given the right set of predicates, SLAM can terminate in one iteration. It is not clear what kind of heuristics should be used to pick the additional predicates. It would further validate SLAM by applying it to more code bases. The tool should also be released to outside of Microsoft to verify the results.
• If the program is buggy, SLAM returns one path that violates the property. This forces the developer to fix one bug at a time to get rid of one erroneous path. This debugging cycle may take a while if there are numerous independent paths that violate the property.
• An interesting line of future work could be to look for properties that cannot be easily checked using SLAM and then perhaps formulate other "automatically-tunable abstraction frameworks".
• It would be cool to apply this system to a more optimization-based application. They also need a better performance characterization for the tools (runtime and preciseness).
• Could the idea of iterative refinement help to reduce false positive rate in other analyses? (Look more carefully at flagged errors, doing more precise analysis, to minimize false positives.) Has this already been done?
• Would it be possible to get faster but less precise results using a more traditional, information-losing join function instead of the collecting semantics over predicate sets?
• The authors do not elaborate precisely why the algorithm may not terminate. It seems like an addition of a new predicate is what forces another iteration of the algorithm. Given a finite number of variables in the problem there should be a finite number of predicates over them, thus the algorithm should terminate in finite amount of time. I am obviously wrong but the non-termination issue should still be treated in more detail.
• While SLAM is great for verifying control-flow properties, I don't think it would do as well for properties involving extensive use of the heap. The paper shows that such a property is more difficult to verify. The one-level flow alias analysis they use is very imprecise for the heap, so for some heap-intensive property, it's essentially like using no alias analysis at all.
• I especially like how the theorem prover is used selectively to validate an already discovered path and to generate the boolean program. This plays to the theorem prover's strengths and avoids the difficulty of trying to do the whole verification in the theorem prover.
• C2BP is exponential in the number of predicates, but the paper claims that ignoring sets of predicates larger than three does not reduce accuracy, so C2BP is cubic in practice. It is not clear how they determined this, or if C2BP will automatically terminate after a cubic amount of work if appropriate.
• So what do I mean by iterations? Well, SLAM attempts to bridge the gap between precise, concrete interpreters (which of course will tell you exactly what you need to know, but can be arbitrarily slow) and very abstract checkers by starting off with a very coarse abstraction and then refining it as necessary. Each abstraction "session" will terminate, since the abstractions are finite, but there is no guarantee that you will be able to find a precise enough abstraction in a finite amount of time.
• Also novel was the idea of using VCGen techniques on single error traces instead of entire programs to find the interesting predicates to add to an abstraction.