How do you teach a robot to do something like watering a plant or ironing a shirt?<\/p>\n
RIGVid: Robots Imitating Generated Videos<\/p>\n
Computer science professor Svetlana Lazebnik<\/a> and Ph.D. student Shivansh Patel<\/a> from The Grainger College of Engineering Siebel School of Computing and Data Science at the University of Illinois Urbana-Champaign say that \u201ctraditionally, we must perform laborious, task-specific demonstrations to collect data to teach robots. While recent works try to avoid this by imitating from videos, they need to perfectly match the videos to the robot\u2019s environment, and the video collection process itself remains laborious.\u201d<\/p>\n Looking at the rapid progress of state-of-the-art video generators like\u00a0OpenAI Sora and Kling AI,\u00a0they asked: \u201cWhat if, instead of collecting real videos, we could just generate the demonstration video on the fly? The idea was to generate a single video tailored exactly to the robot’s starting environment and the specific task.\u201d Accordingly, they created a method entitled RIGVid: Robots Imitating Generated Videos<\/a>.\u00a0Collaborating on this project with\u00a0Lazebnik and Patel are Illinois’ Master of Computer Science student Shraddhaa Mohan<\/a>, Department of Electrical & Computer Engineering\u00a0Ph.D. student Hanlin Mai<\/a>, alumnus\u00a0Unnat Jain<\/a> (’22, Ph.D., Computer Science), now at\u00a0UC Irvine, and Columbia University CS professor Yunzhu Li<\/a>.\u00a0<\/p>\n \n Lazebnik and Patel explain that, as opposed to the time-consuming process of data collection and manual recording of videos that match a robot\u2019s setup, the\u00a0team \u201csimply gives a text command, like \u2018pour water on the plant,\u2019 and an image of the current scene. The video generation model generates the demonstration video. One of the advantages of our method is that it doesn\u2019t actually involve any training \u2013 all the \u2018training\u2019 is already encapsulated in the pre-trained video generation model. After the desired video demonstration is generated, all we have to do is extract the object trajectory from the video and imitate it with the robot.\u201d<\/p>\n They use a monocular depth estimator to predict a corresponding depth map for every frame of the generated video. The ground-truth depth map from the initial RGB-D image of the real scene is used to “ground” the generated motion in real-world units. So, the robot ultimately relies on both the visual content from the generated video and the corresponding estimated depth data to extract the full six degrees of freedom (6DoF) trajectory of the object.<\/p>\n RIGVid was shown to outperform alternative robot imitation methods such as Stanford\u2019s ReKep, which uses a Vision-Language Model to predict a sequence of keypoints for the object to follow. The rich detail of a generated video played a crucial role as RIGVid achieved an 85% success rate across tasks, while the keypoint-based method succeeded only 50% of the time.<\/p>\n RIGVid\u2019s approach for extracting an object\u2019s motion from a generated video, based on model-based six DoF object pose tracking, also outperformed alternative methods for trajectory extraction using optical flow and sparse keypoints. The improvement was apparent on challenging tasks, like sweeping dirt or placing a thin spatula into a pan, where the other methods struggled with object occlusion.<\/p>\n Even better, Lazebnik and Patel note that \u201cRIGVid is embodiment-agnostic, meaning it’s not tied to a single robot. We successfully transferred to a different robot platform and even extended it to bimanual (two-armed) tasks, where it placed a pair of shoes into a box simultaneously.\u201d<\/p>\n See the RIGVid robot in action<\/a>.<\/p>\n Grainger Engineering Affiliations<\/p>\nSvetlana Lazebnik<\/p>\n
Shivansh Patel<\/p>\n