Graph-Structed Visual Imitation (original) (raw)

Graph-Structed Visual Imitation

Visual entity graph (VEG) representation of human demonstration and robot imitation for two timesteps of a typical imitation. Corresponding nodes in the human demonstration (top) and robot imitation (bottom) share the same color. Our graphs are hierarchical. Edges exist between object and robot/human hand nodes and point feature nodes, and are added/deleted dynamically over time based on the motion saliency (as shown in the figure with solid/dashed lines).

Abstract

We cast visual imitation as a visual correspondence problem. Our robotic agent is rewarded when its actions result in better matching of relative spatial configurations for corresponding visual entities detected in its workspace and teacher's demonstration. We build upon recent advances in Computer Vision,such as human finger keypoint detectors, object detectors trained on-the-fly with synthetic augmentations, and point detectors supervised by viewpoint changes and learn multiple visual entity detectors for each demonstration without human annotations or robot interactions. We empirically show the proposed factorized visual representations of entities and their spatial arrangements drive successful imitation of a variety of manipulation skills within minutes, using a single demonstration and without any environment instrumentation. It is robust to background clutter and can effectively generalize across environment variations between demonstrator and imitator, greatly outperforming unstructured non-factorized full-frame CNN encodings of previous works.

We use human hand keypoint detectors, multi-viewpoint feature learning, and synthetic image generation for on-the-fly object detector training from only a few object mask examples. Using a manually designed mapping between the human hand and the robot, the visual detectors can effectively bridge the visual gap between demonstrator and imitator environment, are robust to background clutter, and generalize across different object instances.

Example of the pouring imitation task. Top row shows demonstration, second row shows imitation, and third row shows generalization to a novel object instance.

Supplementary Videos

Pushing Task: we train the robot to push the yellow hexagon to purple ring along different trajectories, and show generalization to pushing a different object

Stacking Task: we train the robot to place the yellow hexagon on the purple ring, and show generalization to a different object and different background

Pouring Task: we train the robot to pour liquid from the orange can to mug, and show generalization of this task to a different can using pixel features

Below is a video of all demonstrated tasks and their successful imitation.

Paper

Code