Visual Learning and Reasoning for Robotic Manipulation

Full-day workshop at RSS 2020

Oregon State University at Corvallis, Oregon, USA

July 13, 2020, Pacific Time (PT)

Welcome! Please attend our virtual workshop via this link.

This workshop includes three live events:
  • Invited Talks (25 min talk + 5 min Q&A)
  • Spotlight Talks (4 × 5 min pre-recorded videos + 10 min Q&A)
  • Panel Discussion (60 min)


Time (PT) Invited Speaker Title
9:15 - 9:30 -
Opening Remarks
| Video |
9:30 - 10:00

Kostas Daniilidis
University of Pennsylvania
The Curious Explorer
| Video |
10:00 - 10:30

Thomas Funkhouser
Google Research
Spatial Action Maps
| Video |
10:30 - 11:00 Spotlight Talks
Latent Space Roadmap for Visual Action Planning
Martina Lippi (University of Salerno); Petra Poklukar (KTH); Michael Welle (KTH); Anastasiia Varava (KTH); Hang Yin (KTH); Alessandro Marino (University of Cassino and Southern Lazio); Danica Kragic (KTH)
| PDF | Video |

Revisiting Grasp Map Representation with a Focus on Orientation in Grasp Synthesis
Nikolaos Gkanatsios (DeepLab); Georgia Chalvatzaki (TU Darmstadt); Petros Maragos (National Technical University of Athens); Jan Peters (TU Darmstadt + Max Planck Institute for Intelligent Systems)
| PDF | Video |

Simple Sensor Intentions for Exploration
Tim Hertweck (DeepMind); Martin Riedmiller (DeepMind); Michael Bloesch (Google); Jost Tobias Springenberg (DeepMind); Noah Siegel (DeepMind); Markus Wulfmeier (DeepMind); Roland Hafner (Google DeepMind); Nicolas Heess (DeepMind)
| PDF | Video |

Goal-Aware Prediction: Learning to Model What Matters
Suraj Nair (Stanford University); Silvio Savarese (Stanford University); Chelsea Finn (Stanford University)
| PDF | Supp | Video |

| Video |
11:00 - 11:30

Sonia Chernova
Georgia Tech / FAIR
Semantic Grasping through Wide and Deep Learning
| Video |
11:30 - 12:00

Russ Tedrake
MIT / Toyota Research Institute
Toward Category-Level Manipulation
| Video |
12:00 - 1:30 - Lunch Break
1:30 - 2:00

Pieter Abbeel
UC Berkeley
Can Deep Reinforcement Learning from Pixels Be Made as Efficient as from States?
| Video |
2:00 - 2:30

Fei-Fei Li
Stanford University
Octopus, Kittens & Babies: From Seeing to Doing
| Video |
2:30 - 3:00 Spotlight Talks
Cloth Region Segmentation for Robust Grasp Selection
Thomas Weng (Carnegie Mellon University); Jianing Qian (Carnegie Mellon University); Brian Okorn (Carnegie Mellon University); Luxin Zhang (Carnegie Mellon University); David Held (Carnegie Mellon University)
| PDF | Supp | Video |

Action for Better Prediction
Bernadette K Bucher (University of Pennsylvania); Karl Schmeckpeper (University of Pennsylvania); Nikolai Matni (University of Pennsylvania); Kostas Daniilidis (University of Pennsylvania)
| PDF | Video |

Learning Visual Servo Policies via Planner Cloning
Ulrich Viereck (Northeastern University); Kate Saenko (Boston University); Robert Platt (Northeastern University)
| PDF | Video |

Learning to Plan with Point Cloud Affordances for General-Purpose Dexterous Manipulation
Anthony Simeonov (MIT); Yilun Du (MIT); Beomjoon Kim (MIT); Francois Hogan (MIT); Alberto Rodriguez (MIT); Pulkit Agrawal (MIT)
| PDF | Video |

| Video |
3:00 - 3:30 Spotlight Talks
Efficient Adaptation for End-to-End Vision-Based Robotic Manipulation
Ryan C Julian (University of Southern California); Benjamin Swanson (Google); Gaurav Sukhatme (University of Southern California); Sergey Levine (Google); Chelsea Finn (Google Brain); Karol Hausman (Google Brain)
| PDF | Video |

Self-Supervised Goal-Conditioned Pick and Place
Coline M Devin (UC Berkeley); Payam Rowghanian (Osaro); Chris Vigorito (Osaro); Will Richards (Osaro); Khashayar Rohanimanesh (Osaro)
| PDF | Video |

Seeing Through Your Skin: A Novel Visuo-Tactile Sensor for Robotic Manipulation
Francois R Hogan (Samsung Electronics), Sahand Rezaei-Shoshtari (Samsung Electronics), Michael Jenkin (Samsung Electronics), Yogesh Girdhar (Samsung Electronics), David Meger (Samsung Electronics), Gregory Dudek (Samsung Electronics)
| PDF | Video |

kPAM 2.0: Feedback Control for Category-Level Robotic Manipulation
Wei Gao (MIT); Russ Tedrake (MIT)
| PDF | Video |

| Video |
3:30 - 4:00

Dieter Fox
University of Washington / NVIDIA
Manipulating Known and Unknown Objects
| Video |
4:00 - 5:00

Greg Dudek (Moderator)
McGill University / Samsung

Ken Goldberg
UC Berkeley
Invited Speakers
Panel Discussion
| Video |


Visual perception is essential for achieving robot autonomy in the real world. To perform complex manipulation tasks in unknown environments, a robot needs to actively acquire knowledge through physical interactions and conduct sophisticated reasoning of the observed objects. This invites a series of research challenges in developing computational tools to close the perception-action loop. Given the recent advances in computer vision and deep learning, we look for new potential solutions for performing real-world robotic manipulation in an effective and computationally efficient manner.

We focus on the two parallel themes in this workshop:

Call for Papers

We're inviting submissions! If you're interested in (remotely) presenting a spotlight talk, please submit a short paper (or extended abstract) to CMT. We suggest extended abstracts of 2 pages in the RSS format. A maximum of 4 pages will be considered. References will not count towards the page limit. The review process is double-blind.

Please note that we have a Best Paper Award. The winner will receive a prize sponsored by Samsung.

Important Dates:


Best Paper Award:
kPAM 2.0: Feedback Control for Category-Level Robotic Manipulation.
Wei Gao (MIT); Russ Tedrake (MIT)


Kuan Fang
Stanford University

David Held

Yuke Zhu
UT Austin / NVIDIA

Dinesh Jayaraman
Univ. of Pennsylvania

Animesh Garg
Univ. of Toronto / NVIDIA

Lin Sun

Yu Xiang

Greg Dudek
McGill / Samsung



For further information, please contact us at rss20vlrrm [AT] gmail [DOT] com