Date: Wed, 10 May 2006 15:04:39 -0700
From: Prem Devanbu <devanbu@cs.ucdavis.edu>
Subject: review anti-patterns.
To: fse14-pc@cs.uoregon.edu

Colleagues,

    As you work on your reviews, please keep in mind the importance of
providing specific, constructive feedback to every submitter. Prima facie,
one might conclude that the reviewer's job is limited to selecting the best
papers for publication, and thus be tempted to address reviews only to
other members of the PC.  The reviews are read (perhaps most carefully) by
authors who have invested a significant amount of time and effort in their
submission, and who probably believe that it is worthy of publication.
They might, as such, reasonably expect not only a clearly justified
evaluation, but also specific and constructive advice on improving their
paper.


The ideal review:

---Evaluates the importance/relevance of the problem as listed by
the authors.  Authors of a paper may not agree with your conclusion,
    but it is vital that they understand your rationale.

---Lists the claims, and judges if they are applicable to the problem, and
    their expected importance/applicability/usefulness.

---Describes how the authors validate the claims, and evaluates the
validation.

---Gives specific suggestions to improve the writing

---Suggests specifically any relevant related work the authors should include.


Please consider these review anti-patterns and, if you find them in
your own reviewing, try the suggested modifications.

x--------x--------x--------x--------x--------x--------x--------x--------x


1) AntiPattern Class:  Rhetorical Reviews


1.a) Undesirable: "___ is an LPU "

Better:   " Adds just a small  improvement, viz., X,  to  previous
paper by Harold & Kumar [ICSE05]"


1.b) Undesirable: "  same old story " or " it's been said before"

Better: "It's not clear why this result improves upon previous
papers [citations]"


1.c) Undesirable: "  off-base "

Better: "The problem as described is not well-motivated, because...."


2) AntiPattern Class: Unsupported Subjective Opinion


2.a) Undesirable: " I don't think anyone will use this"

Better: " Unlikely to find practical application because <<specific reason>>"


2.b) Undesirable: " I don't believe the result "

Better:
"Claims should have been validated by <<specific experimental or
theoretical technique>> X ...., because the method used has flaw Y"
(OR) "Step 5 of proof does not follow from Step 4 because...."
(OR) "The measurement technique used in... is improper, because...."
(OR) "The example/case study used is not pertinent/cogent, because...."


2.c) Undesirable: "...On the wrong track..."  or

Better: "The problem is not well motivated, because...." OR "..the
approach is not applicable, because"


2.d) Undesirable "...this work seems very similar to Harold &
Kumar  [ICSE 2005]....."

Better: "Specific result X claimed by authors has already been done
by Harold & Kumar [ICSE 2005]".


2.e) Undesirable: "Paper has nothing original in it"

Better: "Specific result X claimed by authors has already been done by
Harold and Kumar [ICSE 2005]", and before that Smith & Wesson [ICSE 1980]


2.f) Undesirable: "Paper is unreadable"

Better:  "Here are some examples  of writing that make the paper
hard  to read....."

"Here are some organizational choices made by the author that make
the paper hard to read...."