Deep Learning Research Papers for Robot Perception

A collection of deep learning research papers with coverage in perception and associated robotic tasks. Within each research area outlined below, the course staff has identified a core and extended set of research papers. The core set of papers will form the basis of our seminar-style lectures starting in week 8. The extended set provides additional coverage of even more exciting work being done within each area.

---

Table of contents

1. RGB-D Architectures
2. Point Cloud Processing
3. Object Pose, Geometry, SDF, Implicit surfaces
4. Dense Descriptors, Category-level Representations
5. Recurrent Networks and Object Tracking
6. Visual Odometry and Localization
7. Semantic Scene Graphs and Explicit Representations
8. Neural Radiance Fields and Implicit Representations
9. Datasets
10. Self-Supervised Learning
11. Grasp Pose Detection
12. Tactile Perception for Grasping and Manipulation
13. Pre-training for Robot Manipulation
14. Perception Beyond Vision
15. More Frontiers
    1. Interpreting Deep Learning Models
    2. Fairness and Ethics
    3. Certifiable Perception
    4. Articulated Objects
    5. Deformable Objects
    6. Transparent Objects
    7. Dynamic Scenes
    8. Beyond 2D Convolutions
    9. Reinforcement Learning
    10. Generative Modeling

---

RGB-D Architectures

Core List

1. PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes, Xiang et al., 2018

2. A Unified Framework for Multi-View Multi-Class Object Pose Estimation, Li et al., 2018

3. PVN3D: A Deep Point-Wise 3D Keypoints Voting Network for 6DoF Pose Estimation, He et al., 2020

4. Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation, Li et al., 2021

Extended List

• 3D ShapeNets: A Deep Representation for Volumetric Shapes, Wu et al., 2015

• VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition, Maturana et al., 2015

• Multi-view Convolutional Neural Networks for 3D Shape Recognition, Su et al., 2015

• Volumetric and Multi-View CNNs for Object Classification on 3D Data, Qi et al., 2016

• Robust 6D Object Pose Estimation with Stochastic Congruent Sets, Mitash et al., 2018

• What’s Behind the Couch? Directed Ray Distance Functions (DRDF) for 3D Scene Reconstruction, Kulkarni et al., 2022

Point Cloud Processing

Core List

1. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, Qi et al., 2017

2. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space, Qi et al., 2017

3. PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation, Xu et al., 2018

4. DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion, Wang et al., 2019

Extended List

• Just Go with the Flow: Self-Supervised Scene Flow Estimation, Mittal et al., 2019

• PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows, Yang et al., 2019

• 3D Object Detection with Pointformer, Pan et al., 2021

• Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories, Harley et al., 2022

Object Pose, Geometry, SDF, Implicit surfaces

Core List

1. SUM: Sequential scene understanding and manipulation, Sui et al., 2017

2. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation, Park et al., 2019

3. Implicit surface representations as layers in neural networks, Michalkiewicz et al., 2019

4. iSDF: Real-Time Neural Signed Distance Fields for Robot Perception, Oriz et al., 2022

Extended List

• Local Deep Implicit Functions for 3D Shape, Genova et al., 2020

• Implicit geometric regularization for learning shapes, Gropp et al., 2020

• TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulation, Pan et al., 2022

• Improving Object Pose Estimation by Fusion With a Multimodal Prior – Utilizing Uncertainty-Based CNN Pipelines for Robotics, Richter-Klug et al., 2022

Dense Descriptors, Category-level Representations

Core List

1. Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation, Florence et al., 2018

2. Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation, Wang et al., 2019

3. kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation, Manuelli et al., 2019

4. Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Image, Lin et al., 2022

Extended List

• Visual Descriptor Learning from Monocular Video, Deekshith et al., 2020

• SurfEmb: Dense and Continuous Correspondence Distributions for Object Pose Estimation with Learnt Surface Embeddings, Haugaard et al., 2021

Recurrent Networks and Object Tracking

Core List

1. DeepIM: Deep Iterative Matching for 6D Pose Estimation, Li et al., 2018

2. PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking, Deng et al., 2019

3. 6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints, Wang et al., 2020

4. XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model, Cheng and Schwing, 2022

Extended List

• Long Short-Term Memory, Hochreiter et al., 1997

• The Unreasonable Effectiveness of Recurrent Neural Networks, Karpathy, 2015

• TrackFormer: Multi-Object Tracking with Transformers, Meinhardt et al., 2022

• RNNPose: Recurrent 6-DoF Object Pose Refinement with Robust Correspondence Field Estimation and Pose Optimization, Xu et al., 2022

Visual Odometry and Localization

Core List

1. Backprop KF: Learning Discriminative Deterministic State Estimators, Haarnoja et al., 2016

2. Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors, Jonschkowski et al., 2018

3. Multimodal Sensor Fusion with Differentiable Filters, Lee et al., 2020

4. Differentiable SLAM-net: Learning Particle SLAM for Visual Navigation, Karkus et al., 2021

Extended List

• Factor Graphs and GTSAM, Dellaert et al., 2012

• SuperPoint: Self-Supervised Interest Point Detection and Description, DeTone et al., 2017

• Particle Filter Recurrent Neural Networks, Ma et al., 2019

• Differentiable Algorithm Networks for Composable Robot Learning, Karkus et al., 2019

• SuperGlue: Learning Feature Matching with Graph Neural Networks, Sarlin et al., 2019

• Chasing Ghosts: Instruction Following as Bayesian State Tracking, Anderson et al., 2019

• Differentiable Factor Graph Optimization for Learning Smoothers, Yi et al., 2021

• How to train your differentiable filter, Kloss et al., 2021

• Differentiable Nonparametric Belief Propagation, Opipari et al., 2021

• A Robot Web for Distributed Many-Device Localisation, Murai et al., 2022

Semantic Scene Graphs and Explicit Representations

Core List

1. Image Retrieval using Scene Graphs, Johnson et al., 2015

2. Semantic Robot Programming for Goal-Directed Manipulation in Cluttered Scenes, Zeng et al., 2018

3. Semantic Linking Maps for Active Visual Object Search, Zeng et al., 2020

4. Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization, Hughes et al., 2022

Extended List

• RoboSherlock: Unstructured information processing for robot perception, Beetz et al., 2015

• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations, Krishna et al., 2016

• Image Generation from Scene Graphs, Johnson et al., 2018

• 3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera, Armeni et al., 2020

• Differentiable Scene Graphs, Raboh et al., 2020

• ConceptFusion: Open-set Multimodal 3D Mapping, Jatavallabhula et al., 2023

Neural Radiance Fields and Implicit Representations

Core List

1. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, Mildenhall et al., 2020

2. iMAP: Implicit Mapping and Positioning in Real-Time, Sucar et al., 2021

3. NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields, Rosinol et al., 2022

4. NARF22: Neural Articulated Radiance Fields for Configuration-Aware Rendering, Lewis et al., 2022

5. Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation, Shen et al., 2023

Extended List

• NeRF Explosion 2020, Dellaert, 2020

• Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations, Sitzmann et al., 2019

• Local Implicit Grid Representations for 3D Scenes, Jiang et al., 2020

• Convolutional occupancy networks, Peng et al., 2020

• Object-Centric Neural Scene Rendering, Guo et al., 2020

• INeRF: Inverting Neural Radiance Fields for Pose Estimation, Yen-Chen et al., 2021

• ILabel: Interactive Neural Scene Labelling, Zhi et al., 2021

• Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation, Simeonov et al., 2021

• BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering, Xiangli et al., 2021

• Block-NeRF: Scalable Large Scene Neural View Synthesis, Tancik et al., 2022

• NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields, Yen-Chen et al., 2022

• Language Embedded Radiance Fields, Kerr et al., 2023

Datasets

Core List

1. Deep Learning for Robots: Learning from Large-Scale Interaction, Levine et al., 2016

2. Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning, Makoviychuk et al., 2021

3. Grounding Predicates through Actions, Migimatsu and Bohg, 2022

4. All You Need is LUV: Unsupervised Collection of Labeled Images using Invisible UV Fluorescent Indicators, Thananjeyan et al., 2022

Extended List

Collecting data with robots

• TossingBot: Learning to Throw Arbitrary Objects, Zeng et al., 2019

RGB-D Datasets

• (NYU Depth v2) Indoor Segmentation and Support Inference from RGBD Images, Silberman et al., 2012

• SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite, Song et al., 2015

• YCB-Video Dataset, Xiang et al., 2018

• BOP: Benchmark for 6D Object Pose Estimation, Hodaň et al., 2019

• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes, Dai et al., 2019

• ProgressLabeller: Visual Data Stream Annotation for Training Object-Centric 3D Perception, Chen et al., 2022

• TO-Scene: A Large-scale Dataset for Understanding 3D Tabletop Scenes, Xu et al., 2022

Semantic Datasets

• Understanding Human Hands in Contact at Internet Scale, Shan et al., 2020

• Habitat-Matterport 3D Semantics Dataset, Yadav et al., 2022

Object Model Datasets

• ShapeNet: An Information-Rich 3D Model Repository, Chang et al., 2015

• PartNet-Mobility Dataset

Simulators

• MuJoCo: A physics engine for model-based control, Todorov et al., 2015

• Pybullet, a python module for physics simulation for games, robotics and machine learning, Coumans et al., 2015

• NVIDIA Isaac Sim

• CARLA: An Open Urban Driving Simulator, Dosovitskiy et al., 2017

• SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation, Lin et al., 2020

• ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills, Gu et al., 2023

Self-Supervised Learning

Core List

1. Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks, Lee et al., 2019

2. VICRegL: Self-Supervised Learning of Local Visual Features, Bardes et al., 2022

3. Fully Self-Supervised Class Awareness in Dense Object Descriptors, Hadjivelichkov and Kanoulas, 2022

4. Self-Supervised Geometric Correspondence for Category-Level 6D Object Pose Estimation in the Wild, Zhang et al., 2022

Extended List

• Emerging Properties in Self-Supervised Vision Transformers, Caron et al., 2021

• DINOv2: Learning Robust Visual Features without Supervision, Oquab et al., 2023

Grasp Pose Detection

Core List

1. Real-Time Grasp Detection Using Convolutional Neural Networks, Redmon and Angelova, 2015

2. Using Geometry to Detect Grasps in 3D Point Clouds, ten Pas and Platt, 2015

3. Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics, Mahler et al., 2017

4. Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes, Sundermeyer et al., 2021

5. Sample Efficient Grasp Learning Using Equivariant Models, Zhu et al., 2022

Extended List

• Deep Learning for Detecting Robotic Grasps, Lenz et al., 2013

• High precision grasp pose detection in dense clutter, Gualtieri et al., 2016

• GlassLoc: Plenoptic Grasp Pose Detection in Transparent Clutter, Zhou et al., 2019

• MetaGraspNet_v0: A Large-Scale Benchmark Dataset for Vision-driven Robotic Grasping via Physics-based Metaverse Synthesis, Chen et al., 2021

• Grasp Learning: Models, Methods, and Performance, Platt, 2022

Tactile Perception for Grasping and Manipulation

Core List

1. More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch, Calandra et al., 2018

2. Tactile Object Pose Estimation from the First Touch with Geometric Contact Rendering, Bauza et al., 2020

3. Visuotactile Affordances for Cloth Manipulation with Local Control, Sunil et al., 2022

4. ShapeMap 3-D: Efficient shape mapping through dense touch and vision, Suresh et al., 2022

Extended List

• The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?, Calandra et al., 2017

• GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force, Yuan et al., 2017

• Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation, Alspach et al., 2019

• A Review of Tactile Information: Perception and Action Through Touch, Li et al., 2020

• TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors, Wang et al., 2020

• Active Extrinsic Contact Sensing: Application to General Peg-in-Hole Insertion, Kim et al., 2021

• Active Visuo-Haptic Object Shape Completion, Rustler et al., 2022

• Learning Self-Supervised Representations from Vision and Touch for Active Sliding Perception of Deformable Surfaces, Kerr and Huang et al., 2022

• See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation, Li et al., 2022

• Learning to Grasp the Ungraspable with Emergent Extrinsic Dexterity, Zhou and Held, 2022

Pre-training for Robot Manipulation

Core List

1. SORNet: Spatial Object-Centric Representations for Sequential Manipulation, Yuan et al., 2021

2. CLIPort: What and Where Pathways for Robotic Manipulation, Shridhar et al., 2021

3. Real-World Robot Learning with Masked Visual Pre-training, Radosavovic et al., 2022

4. R3M: A Universal Visual Representation for Robot Manipulation, Nair et al., 2022

5. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances, Ahn et al., 2022

6. RT-1: Robotics Transformer for Real-World Control at Scale, Brohan et al., 2022

Extended List

• Attention and Augmented Recurrent Neural Networks, Olah & Carter, 2016

• Attention is All You Need, Vaswani et al., 2017

• Feature-wise transformations, Dumoulin et al., 2018

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, Dosovitskiy et al., 2020

• Transporter Networks: Rearranging the Visual World for Robotic Manipulation, Zeng et al., 2020

• CLIP: Learning Transferable Visual Models From Natural Language Supervision, Radford et al., 2021

• Masked Autoencoders Are Scalable Vision Learners, He et al., 2021

• Interactive Language: Talking to Robots in Real Time, Lynch et al., 2022

• Transformers are Adaptable Task Planners, Jain et al., 2022

Perception Beyond Vision

Specialized Sensors

• Pigeons (Columba livia) as Trainable Observers of Pathology and Radiology Breast Cancer Images, Levenson et al., 2015

• Automatic color correction for 3D reconstruction of underwater scenes, Skinner et al., 2017

• GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force, Yuan et al., 2017

• Classification of Household Materials via Spectroscopy, Erickson et al., 2018

• Through-Wall Human Pose Estimation Using Radio Signals, Zhao et al., 2018

• A bio-hybrid odor-guided autonomous palm-sized air vehicle, Anderson et al., 2020

• Event-based, Direct Camera Tracking from a Photometric 3D Map using Nonlinear Optimization, Bryner et al., 2019

• SoundSpaces: Audio-Visual Navigation in 3D Environments, Chen et al., 2019

• Neural Implicit Surface Reconstruction using Imaging Sonar, Qadri et al., 2022

More Frontiers

Interpreting Deep Learning Models

• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps, Simonyan et al., 2013

• Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization, Selvaraju et al., 2016

• The Building Blocks of Interpretability, Olah et al., 2018

• Multimodal Neurons in Artificial Neural Networks, Goh et al., 2021

Fairness and Ethics

• Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification, Buolamwini and Gebru, 2018

• Saving Face: Investigating the Ethical Concerns of Facial Recognition Auditing, Raji et al., 2020

Certifiable Perception

• Certifiably Optimal Outlier-Robust Geometric Perception: Semidefinite Relaxations and Scalable Global Optimization, Yang and Carlone, 2021

• Certifiable 3D Object Pose Estimation: Foundations, Learning Models, and Self-Training, Talak et al., 2022

Articulated Objects

• Autonomous Tool Construction Using Part Shape and Attachment Prediction, Nair et al., 2019

• Parts-Based Articulated Object Localization in Clutter Using Belief Propagation, Pavlasek et al., 2020

• Category-Level Articulated Object Pose Estimation, Li et al., 2020

• Differentiable Nonparametric Belief Propagation, Opipari et al., 2021

• Category-Independent Articulated Object Tracking with Factor Graphs, Heppert et al., 2022

• Kineverse: A Symbolic Articulation Model Framework for Model-Agnostic Mobile Manipulation, Röfer et al., 2022

Deformable Objects

• DensePose: Dense Human Pose Estimation In The Wild, Xiao et al., 2018

• FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy, Weng et al., 2021

• DextAIRity: Deformable Manipulation Can be a Breeze, Xu et al., 2022

• Self-supervised Transparent Liquid Segmentation for Robotic Pouring, Narasimhan et al., 2022

• Visio-tactile Implicit Representations of Deformable Objects, Wi et al., 2022

Transparent Objects

• LIT: Light-field Inference of Transparency for Refractive Object Localization, Zhou et al., 2019

• Multi-modal Transfer Learning for Grasping Transparent and Specular Objects, Weng et al., 2020

• Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects, Ichnowski et al., 2021

• ClearPose: Large-scale Transparent Object Dataset and Benchmark, Chen et al., 2022

• TransNet: Category-Level Transparent Object Pose Estimation, Zhang et al., 2022

Dynamic Scenes

• D-NeRF: Neural Radiance Fields for Dynamic Scenes, Pumarola et al., 2020

• 3D Neural Scene Representations for Visuomotor Control, Li et al., 2021

• HexPlane: A Fast Representation for Dynamic Scenes, Cao and Johnson, 2023

Beyond 2D Convolutions

• Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks, Tolstaya et al., 2019

• A Gentle Introduction to Graph Neural Networks, Sanchez-Lengeling et al., 2021

Reinforcement Learning

• Deep Reinforcement Learning from Human Preferences, Christiano et al., 2017

• Understanding RL Vision, Hilton et al., 2020

Generative Modeling

• WaterGAN: Unsupervised Generative Network to Enable Real-time Color Correction of Monocular Underwater Images, Li et al., 2017

• Differentiable Particle Filters through Conditional Normalizing Flow, Chen et al., 2021

• Planning with Diffusion for Flexible Behavior Synthesis, Janner et al., 2022

• Anything-3D: Towards Single-view Anything Reconstruction in the Wild, Shen et al., 2023