Deep Learning Tools and Frameworks

A collection of tools and projects that support deep learning applications to robotic tasks. Within each category below, the course staff provides a sample of tools that may be helpful for implementing course and research projects with deep learning.

Table of contents

  1. Learning Frameworks
  2. Library Ecosystem
  3. Data Annotation
  4. Simulation Environments
  5. Visualization
  6. Model Formats
  7. Misc

Learning Frameworks

Deep Learning frameworks speed-up model development and testing by providing users with optimized implementations of low-level functions (automatic differentiation, gradient descent, matrix operations, etc.) that can be used as building-blocks for robot applications.

Open-source python-based deep learning frameworks have fostered a community and set of tools responsible for many of the core research and engineering developments powering deep learning. Here are some of the most popular python-based frameworks:


While Python-based learning frameworks typically offer C-APIs, they have been built-on and inspired-by additional open-source projects, including:


The more recent language, Julia, which you may be familiar with through the ROB curriculum, also offers deep learning support:

Library Ecosystem

A vast number of specialized software libraries have been built upon the popular open-source learning frameworks to provide data structures and algorithms that are customized to specific domains and use-cases. Often, these provide optimized implementations that are useful for a specific group of developers working within a specific sub-area of robot learning while also facilitating interactions and collaborations within the community. Contributing to these open-source projects is valuable and encouraged. Here are a sample that may be related to your interests:

  • PyTorch3D: a specialized library for integrating 3D data (points, geometries, renderers, etc.) with deep learning in PyTorch

  • nerfstudio: for implementing, training, visualizing, and exporting your Neural Radiance Fields

  • Ray: a library for scalable reinforcement learning

  • GPyTorch: for implementing Gaussian Processes in with GPU-acceleration in PyTorch

Data Annotation

Much of deep learning (but not all) relies on human-annotated data for training or evaluation. The data and annotations are typically expensive to obtain, which motivates the development and use of tools to make data annotation more efficient and cheaper. Here are a small sample of tools that exist to support your labeling efforts:

Simulation Environments

Robots and sensors can be prohibitively expensive, so simulation environments have been created to enable large-scale training (i.e. reinforcement learning) and data collection within robotics. Here are a few to get you started:

Visualization

Understanding model architectures and outputs is crucial to validate and develop useful deep learning-based systems. Visualization tools can help support our understanding, especially for perception tasks, by depicting model predictions. Knowing which tools are useful for a given modality and how to use those tools is valuable skill for deep learning practicioners. Here are some visualization tools that are broadly useful:

  • TensorBoard: to understand model architectures and summary statistics measured during training

  • Open3D: for visualizing 3D data modalities

  • Plotly: for general purpose analysis and plotting

  • RViz: for integrating visualization with ROS topics

Model Formats

Open-source model formats describe learning-based architectures and models in order to enable sharing and deploying the models independent of the software-hardware used during model training and development. These are especially useful for deploying the models, where minimizing latencies caused by the general-purpose learning frameworks can be crucial.

Misc

  • distill.pub: for organizing your publications in an elegant and shareable format

  • ConvNetJS: A minimal Javascript library for neural network training and inference within web-browsers