In today's class, we will get a feel for the peer review process in research by examining a number of real peer reviews on actual papers. 

ICWSM CFP: https://www.icwsm.org/2022/index.html/ (Links to an external site.)

Instructions for Reviewers: https://2021.emnlp.org/blog/2021-05-17-instructions-for-reviewers (Links to an external site.)

The third thing we will do is to look at a simple set of reviews to get a feel for how reviews provide information to the authors.  We will use a paper and reviews from Professor Julian McAuley because he makes his reviews publicly available.

  1. Open Professor Julian McAuley's homepageLinks to an external site. (it will open in a new tab or window).
  2. Find the paper: Like hiking? You probably enjoy nature: Persona-grounded Dialog with Commonsense Expansions
  3. Click on the pdf link and read the abstract of the paper to get a sense of what the paper is about and its contribution
  4. Click on the reviews link, and read the reviews. In reading your reviews, make notes on the following:
    • On what points did the reviewers agree? On what points did they disagree?
    • In what ways did they point out the merits of the work?
    • What were the main issues raised in the review?
    • What suggestions for improvement did they give the author? Do these suggestions seem reasonable?


STOP FOR DISCUSSION.

Next, we will go back to Prof. McAuley's page to look at a different paper to get a sense of how different conferences/journals use different formats for reviews.

  1. Find the paper:  Image-based recommendations on styles and substitutes
  2. Click on the pdf link and read the abstract of the paper.
  3.  Click on the reviews link, and notice that these reviews look very different from the last set of reviews. There are in fact reviews from two conferences in this document. The top set of reviews are from a conference called SIGIR, where the paper was ultimately published. The bottom set of reviews are for the same paper (with some minor modifications) for a conference called CVPR, where the paper was ultimately withdrawn. Take notes on the following:
    • What are the required sections in the reviews for each conference? How do they compare? What are the advantages or disadvantages of each?
    • What are "Rebuttal requests?" (See CVPR reviews.) Do the reviewers agree on these?
    • In general, how does the feedback differ from one conference to the other?

STOP FOR DISCUSSION