1   Paper Review

The goal for this deliverable is to practice the skills needed to serve as an academic peer reviewer.

What is peer review?

Academic peer review is the process by which research artifacts being considered for publication are evaluated by independent experts in the relevant field to determine the artifacts’ suitability for publication. Typically, work considered for publication will be formatted as a written paper describing an experimental setup, the observed results and the researchers’ motivations for carrying out the described experiments. Peer review of scholarly manuscripts plays a crucial role in academia and in advancing our body of knowledge. When you review a scholarly work, you are providing a service for the public, the field of researchers in your discipline, and the scholars whose work you are reviewing. You are serving the public and the field by providing an objective source of quality control and credibility to the work being considered for publication. At the same time, you’re serving the scholars by providing input and perspective that can help cultivate new ideas and shape the work’s relevance.

It is important to remember the distinction between peer review and the scientific method. Science uses the testing of falsifiable hypotheses by reproducable experimentation to establish new knowledge. Peer review uses independent experts to evaluate the standards, quality and reputability of claimed scientific results. Notably, the standards of peer review are themselves judged and questioned by scientific communities. Fields contributing to research in deep learning for robot perception (i.e. robotics, artificial intelligence, machine learning, computer vision, etc.) have been experimenting with an open review process (Soergel et al., 2013) intended to improve the quality, speed, and accountability of reviews.

The structure of a review in robotics research

Quality reviews will be formatted in two distinct sections: a review scoring and a review narrative. The review scoring section is an assessment of the manuscript based on its suitability for publication in the venue to which it was submitted (e.g. conference, journal, workshop, etc.). Notably as part of the scoring, the reviewer may be expected to score their own confidence in their review based on perceived expertise with respect to the paper under consideration. The review narrative is written feedback that is provided to authors to use for strengthening their work. Students in DeepRob will only be expected to submit a review narrative as part of the final project (not a review scoring).

A review narrative will generally be expected to address the following main points: a paper summary, a review summary, and a set of individual points of feedback. See below for the expectated format in DeepRob.

In general, the tone of reviews should be positive about the work and constructive about the potential for contributions. The best reviews provide a clear rationale for how to move the paper forward. Especially useful is a list of items that need to be addressed for the paper to be acceptable for publication or to improve an already accepted paper.

In general, please do not use “I,” “you,” “the authors,” etc. in your reviews. Reviews should be depersonalized as much as possible. The review should focuse on the work and not the individuals involved with the work (neither authors nor reviewers).

Expected Review Format

Student teams in DeepRob should complete a review narrative for their assigned paper as part of the final project. The review narrative should include the following structure: a paper summary, a review summary, and specific points of feedback. Students are expected to typeset their reviews using LaTeX in IEEE conference style. A LaTeX review template is provided for your convenience. You may write your review collaboratively using online LaTeX tools, such as Overleaf.

Paper Summary

This section of the review should give a summary of the paper in 1-2 paragraphs (4-10 sentences). The purpose of the summary is to show that you, as a reviewer, understood the paper and to provide its best possible interpretation before going into critiques. This section is helpful to authors, in part, because it helps them understand what ideas a reader is most likely to remember from the paper. When summarizing a paper, you should:

  • Summarize the main contributions of the paper in one sentence.
  • Identify the core problem being addressed by the paper.
  • Describe the key idea of the paper and how it connects to the core problem.
  • Summarize the implementation and methods used to evaluate the paper’s key ideas.
  • Identify the paper’s conclusion from its findings.

Review Summary

The second paragraph of your review should provide an overall assessment. The first sentence of this paragraph should provide the overall conclusion of the review. This is followed by individual sentences that assess the paper with respect to its clarity, technical and experimental soundness, intellectual novelty, and relevance to the field. The high-level points of feedback to improve the paper should also be included.

Specific Points of Feedback

After the first two paragraphs, you should then provide a list that addresses specific points of feedback. This list can be as long as needed to address all of your points of feedback. Please be sure to be both critical and helpful. You should identify flaws and shortcomings in the work and provide suggestions for their improvement. These points can be of varying length, depending on the amount of description needed. For example, points about technical problems tend to be about a paragraph in length. In contrast, typographical and grammatical errors tend to be expressed briefly in one line.

Useful Resources

Review Deadline

Students in DeepRob should submit their paper review at least one week prior to their scheduled paper presentation. Students should submit their review as a PDF file via email to the course instructor.