Skip to main content Link Search Menu Expand Document (external link)
DeepRob

Final Project

Overview

The objective of this project is to build skills needed to understand and lead research projects in deep learning for robot perception. In this project you will read state of the art deep learning papers, reproduce results from an existing paper and finally propose then develop an extension to the existing results.

The goals for this project are as follows:

  • Practice reading deep learning papers in your areas of interest.
  • Analyze the published motivations, results, and methodologies in deep learning papers from the perspective of an academic reviewer.
  • Present your analysis of an existing deep learning paper during the seminar portion of the course (starting week 8).
  • Practice reproducing the results of published work. Build an understanding of what challenges are common when reproducing published results.
  • Propose an algorithmic extension to state of the art results, which could result in a workshop or conference submission.
  • Document the results of your work in a scholarly format so that they can be shared with peers and move the field forward.

Project Deliverables

  1. Paper Review
  2. Paper Presentation
  3. Paper Reproduction
  4. Algorithmic Extension
  5. Written Report

Grading

Overall, the final project is worth 24% of the course grade. The breakdwon by each deliverable is shown below:

  1. Paper Review: 3%
  2. Paper Presentation: 3%
  3. Paper Reproduction: 6%
  4. Algorithmic Extension: 6%
  5. Written Report: 6%

1   Paper Review

The goal for this deliverable is to practice the skills needed to serve as an academic peer reviewer.

What is peer review?

Academic peer review is the process by which research artifacts being considered for publication are evaluated by independent experts in the relevant field to determine the artifacts’ suitability for publication. Typically, work considered for publication will be formatted as a written paper describing an experimental setup, the observed results and the researchers’ motivations for carrying out the described experiments. Peer review of scholarly manuscripts plays a crucial role in academia and in advancing our body of knowledge. When you review a scholarly work, you are providing a service for the public, the field of researchers in your discipline, and the scholars whose work you are reviewing. You are serving the public and the field by providing an objective source of quality control and credibility to the work being considered for publication. At the same time, you’re serving the scholars by providing input and perspective that can help cultivate new ideas and shape the work’s relevance.

It is important to remember the distinction between peer review and the scientific method. Science uses the testing of falsifiable hypotheses by reproducable experimentation to establish new knowledge. Peer review uses independent experts to evaluate the standards, quality and reputability of claimed scientific results. Notably, the standards of peer review are themselves judged and questioned by scientific communities. Fields contributing to research in deep learning for robot perception (i.e. robotics, artificial intelligence, machine learning, computer vision, etc.) have been experimenting with an open review process (Soergel et al., 2013) intended to improve the quality, speed, and accountability of reviews.

The structure of a review in robotics research

Quality reviews will be formatted in two distinct sections: a review scoring and a review narrative. The review scoring section is an assessment of the manuscript based on its suitability for publication in the venue to which it was submitted (e.g. conference, journal, workshop, etc.). Notably as part of the scoring, the reviewer may be expected to score their own confidence in their review based on perceived expertise with respect to the paper under consideration. The review narrative is written feedback that is provided to authors to use for strengthening their work. Students in DeepRob will only be expected to submit a review narrative as part of the final project (not a review scoring).

A review narrative will generally be expected to address the following main points: a paper summary, a review summary, and a set of individual points of feedback. See below for the expectated format in DeepRob.

In general, the tone of reviews should be positive about the work and constructive about the potential for contributions. The best reviews provide a clear rationale for how to move the paper forward. Especially useful is a list of items that need to be addressed for the paper to be acceptable for publication or to improve an already accepted paper.

In general, please do not use “I,” “you,” “the authors,” etc. in your reviews. Reviews should be depersonalized as much as possible. The review should focuse on the work and not the individuals involved with the work (neither authors nor reviewers).

Expected Review Format

Student teams in DeepRob should complete a review narrative for their assigned paper as part of the final project. The review narrative should include the following structure: a paper summary, a review summary, and specific points of feedback. Students are expected to typeset their reviews using LaTeX in IEEE conference style. A LaTeX review template is provided for your convenience. You may write your review collaboratively using online LaTeX tools, such as Overleaf.

Paper Summary

This section of the review should give a summary of the paper in 1-2 paragraphs (4-10 sentences). The purpose of the summary is to show that you, as a reviewer, understood the paper and to provide its best possible interpretation before going into critiques. This section is helpful to authors, in part, because it helps them understand what ideas a reader is most likely to remember from the paper. When summarizing a paper, you should:

  • Summarize the main contributions of the paper in one sentence.
  • Identify the core problem being addressed by the paper.
  • Describe the key idea of the paper and how it connects to the core problem.
  • Summarize the implementation and methods used to evaluate the paper’s key ideas.
  • Identify the paper’s conclusion from its findings.

Review Summary

The second paragraph of your review should provide an overall assessment. The first sentence of this paragraph should provide the overall conclusion of the review. This is followed by individual sentences that assess the paper with respect to its clarity, technical and experimental soundness, intellectual novelty, and relevance to the field. The high-level points of feedback to improve the paper should also be included.

Specific Points of Feedback

After the first two paragraphs, you should then provide a list that addresses specific points of feedback. This list can be as long as needed to address all of your points of feedback. Please be sure to be both critical and helpful. You should identify flaws and shortcomings in the work and provide suggestions for their improvement. These points can be of varying length, depending on the amount of description needed. For example, points about technical problems tend to be about a paragraph in length. In contrast, typographical and grammatical errors tend to be expressed briefly in one line.

Review Deadline

Students in DeepRob should submit their paper review at least one week prior to their scheduled paper presentation. Students should submit their review as a PDF file via email to the course instructor.

Useful Resources

2   Paper Presentation

The goals for this deliverable are to practice preparing and delivering a concise research presentation and to build an appreciation for the importance of clarity when presenting research.

The importance of presenting research

Coming soon.

The structure of a robotics research presentation

Quality research presentations will capture an audience’s attention, motivate them to take an interest in the challenge at hand, demonstrate what knowledge has been generated to solve the challenge, and encourage the audience to extend the presented ideas towards new challenges. For maximum affect, research presentations should be correct, clear, concise, and broadly understandable. Given the challenge of achieving all this, we suggest developing your presentations using the following section structure:

1   Hot Start

The first task of a presenter is to command the attention and interest of their audience. Use a hot start to pique the audience’s interest in your talk. There will always be distrations that a presenter must compete with to keep the audience’s focus, even after a successful hot start.

2   Value Proposition

A clear value proposition will motivate the audience to keep paying attention beyond the hot start. This section of the presentation establishes what benefits can be realized by solving an existing challenge or technical problem. The proposition can be framed in language such as, “if we can solve <this challenge>, we’ll be able to realize <these benefits>.”

3   Approach

A motivated audience will then want to know how the challenge being presented can be solved. This section of the talk should provide background on what factors have made the challenge difficult in the past and the key insights the presenters have discovered to alleviate these factors.

4   Resolution

Given a proposed approach, the presenter should demonstrate how the their insights can be applied to the challenge in question to arrive at a solution. This section should show how the value proposition has been more fully realized towards the desired benefits.

5   Call to Action

At this point in the presentation, a natural question from audience members will be, “how can we build on the insights from this work to better realize the desired benefits?” Hence, this section is the presenter’s opportunity to answer the question before it is even asked.

An Example

As an example of a robotics research presentation, consider Anthony’s presentation from the ICRA 2022 Robotic Perception and Mapping workshop. The slides from this presentation can be viewed online and a recording of the corresponding talk is included below:

Expected Presentation Format

Student teams in DeepRob should prepare a 10-minute slide-based oral presentation for their assigned paper as part of the final project. The presentation should include the following structure: background on the problem being addressed, the value proposition, approach and methods, key results, conclusions and directions for future work. Students are expected to use the provided DeepRob Keynote Theme or the provided DeepRob PowerPoint Template for styling your slides.

Presentation Deadline

Students in DeepRob should submit their presentation slides formatted as a PDF at least 3 days prior to their scheduled paper presentation. Students should submit a copy of their slides as a PDF file via email to the course instructor.

Useful Resources

  • How to Give a Great Research Talk: Advice from Simon Peyton Jones, Engineering Fellow at Epic Games.

  • A great resource where you can find recorded research presentations are the recent robotics conferences. For example, the recorded oral presentations from ICRA 2022 and CoRL 2022 can be found online.

  • Oral Presentation Advice: Advice from Professor Mark D. Hill at the University of Wisconsin.

3   Paper Reproduction

The goal for this deliverable is to practice the skills needed to reproduce the results of existing research. For this deliverable, your team should choose a paper related to deep learning and robot perception that you find interesting and reproduce at least one quantitative or qualitative result published in the original paper. You are not required to reproduce the paper your team presented on.

Paper Reproduction Deadline

The Paper Reproduction is due Friday, April 21st by 11:59PM EST. Students in DeepRob should submit their paper reproduction along with the other final project deliverables (algorithmic extension and written report) as a ZIP file via email to the course instructor. Your submission should be documented and organized such that the course staff can replicate your reported results.

Useful Resources

  • The Seminar Papers: The collection of seminar papers was chosen to give broad coverage within deep learning for robot perception. The papers from this list can serve as great subjects for your final project.

  • Papers With Code: A repository tracking public codebases associated with published papers and datasets in machine learning.

4   Algorithmic Extension

The goal for this deliverable is to gain experience crafting a small research project. In this process your team will propose a technical extension to an existing deep learning algorithm, plan how to analyze your proposed extension, and execute on the plan by implementing and evaluating the ideas you propose. The expectation is that your extension will build on the paper reproduction deliverable by using your reproduction as a starting point and baseline for the extension.

Algorithmic Extension Deadline

The algorithmic extension is due Friday, April 21st by 11:59PM EST. Students in DeepRob should submit their algorithmic extension along with the other final project deliverables (paper reproduction and written report) as a ZIP file via email to the course instructor. Your submission should be documented and organized such that the course staff can replicate your extension and reported results.

5   Written Report

The goal for this deliverable is to share the findings of your research in the form of a written paper with peers in the field.

Expected Report Format

Student teams in DeepRob should complete a written report documenting their paper reproduction and algorithmic extension as part of the final project. The project report should include the following sections: a paper abstract, an introduction, a related work section, a section describing your algorithmic extension, your experimental setup and results, and finally a conclusion. Final reports are expected to be 4-6 pages, not including references. Students are expected to typeset their reviews using LaTeX in IEEE conference style. A LaTeX report template is provided for your convenience. You may write your review collaboratively using online LaTeX tools, such as Overleaf.

Report Deadline

The written report is due Friday, April 21st by 11:59PM EST. Students in DeepRob should submit their written report along with the other final project deliverables (paper reproduction and algorithmic extension) as a ZIP file via email to the course instructor.

Useful Resources

Institutional Teaching Collaborative