Empirical Research Methods (CSE 599K/Winter 2022)

Valid experimental designs, sound data analyses, and reproducibility of empirical results are core tenets of the scientific method -- crucial not only for specific domains in computer science but rather any field that seeks empirical evidence.

Meetings

Staff

Communication

Syllabus

Course description

This course covers qualitative and quantitative research methods and focuses on properly designing experiments and observational studies, choosing appropriate statistical methods and models, and reasoning about the validity of experimental designs (in terms of internal, external, and construct validity). This course involves lectures, paper discussions, as well as a hands-on experience for data analysis and visualization with R.

Course format

Most class sessions will begin with a discussion and presentation of material, after (or during) which the floor will be open to rebuttals and discussion. We will all learn from the conversation, even the instructor. We will all have questions and confusions about the material, even the instructor. You are required to be not a passive listener to lectures but an active participant in the discussion. Classroom material is enhanced with assigned readings, in-class activities, and in-class exercises. Note that the first two class meetings are virtual on Zoom.

Paper readings

To help you prepare, you will write a one-paragraph commentary and answer readings questions for most papers, and submit your write up before the class meets to discuss the paper. You will post your commentary and answers to a Canvas discussion board for viewing by the instructor and by other students. The commentary should reflect your understanding and analysis of the issues raised by the paper, and should also help direct (both your and others') preparation for in-class discussion.

You may write the commentary in whatever style you prefer that meets the goals listed above. One good format for the commentary is to critique the paper, listing the following three points: its biggest contribution (and, briefly, why that result was not already obvious), its biggest mistake (in motivation, methodology, algorithm, data analysis, conclusions, or some other area), and the biggest question that it raises (or the most interesting and important follow-on work that it suggests). Another acceptable format is to summarize the paper, describing its thesis, approach, and conclusions, and stating why it is significant. The commentary should also list questions that you have about the paper, such as about technical points or connections to related work. For other ideas about how to critique a paper, see the following advice.

It's OK if you read the paper and there are issues you do not understand. Please ask questions about those issues -- both in your summary and in class -- and we will all gain by the discussion. It's best to explain why something makes no sense to you. For example, don't just say, "I didn't understand section 2", but state where there is a logical fallacy or a conclusion that does not follow or a term that is not defined. The instructor will use these questions to help shape the lectures.

You will have access to all the other students' write ups after posting your own. Please read them, which is a good way for you to get perspective. You can see what you missed (or what other people missed), and whether you agree with their take on the key ideas. It will help to make the class sessions more productive. We also encourage you to use the summaries to ask questions. If you have a question, it is likely that others have the same question but may be too shy (or vain, or insecure) to ask it; they will appreciate you raising the point.

Grading

Grades are based on a research project, in-class exercises, paper reviews, and participation: You can find the general course policies here.

Course material