RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation (original) (raw)

Yufei Wang*,1,Zhou Xian*,1,Feng Chen*, 2,Tsun-Hsuan Wang3,Yian Wang4,
Katerina Fragkiadaki1,Zackory Erickson1,David Held1,Chuang Gan4,5

1CMU, 2Tsinghua IIIS, 3MIT CSAIL, 4UMass Amherst, 5MIT-IBM AI Lab

*Equal Contribution

Abstract

We present RoboGen, a generative robotic agent that automatically learns diverse robotic skills at scale via generative simulation. RoboGen leverages the latest advancements in foundation and generative models. Instead of directly using or adapting these models to produce policies or low-level actions, we advocate for a generative scheme, which uses these models to automatically generate diversified tasks, scenes, and training supervisions, thereby scaling up robotic skill learning with minimal human supervision. Our approach equips a robotic agent with a self-guided propose-generate-learn cycle: the agent first proposes interesting tasks and skills to develop, and then generates corresponding simulation environments by populating pertinent objects and assets with proper spatial configurations. Afterwards, the agent decomposes the proposed high-level task into sub-tasks, selects the optimal learning approach (reinforcement learning, motion planning, or trajectory optimization), generates required training supervision, and then learns policies to acquire the proposed skill. Our work attempts to extract the extensive and versatile knowledge embedded in large-scale models and transfer them to the field of robotics. Our fully generative pipeline can be queried repeatedly, producing an endless stream of skill demonstrations associated with diverse tasks and environments.

RoboGen is simulated and rendered with Genesis, a multi-material multi-solver generative simulation engine for general-purpose robot learning.

RoboGen Pipeline

RoboGen is a fully automated pipeline for endless and diverse skill acquisition. RoboGen pipeline consists of 4 stages: A) task proposal, B) scene generation, C) training supervision generation, and D) skill learning with generated information.

Long-horizon task generation and learning

RoboGen is able to propose high-level tasks, generate corresponding environments, decompose the high-level goal into low-level subtasks, and then learn the sub-skills sequentially.

RoboGen generated tasks, scenes, training supervisions, and learned skills

Please select an image below to view the results.

Select an image above to show the RoboGen results:

RoboGen response shown within code block.

Gallery

Change lamp light direction

Pull lever to start coffee brewing

Store the toy into the storage

Stand upright and walk forward using only hind legs

Bend noodle into a U-shape

Retrieve the gold bar from safe

Close the drawer of the table

Heat up a bowl of soup in microwave

Kick the soccer ball to the left

Spin clockwise without right hind foot touching the ground

Retrieve the toy car from box

Store the apple in refrigerator

Set the clock back by 5 minutes

Lie face down and cawl forward

Load dish into dishwasher

Stay balanced and move forward

Arrange three different cans in a row

Unload the milk from cart

Open the door of trash can

Take out a bottle of water from fridge

Press and rotate dispenser lid

Rotate globe horizontally

Store a toy inside the box

Put an toy into the storage

Direct and press dispenser

Kick the soccer ball to the left

Open washing machine door

Open fridges freezer door

Jump higher than 5 meters

Spin left without using right hind leg

Run backward fast using only front and left hind legs

Pull out the top drawer of the cabinet

Press the button to turn on the printer

Wrap the dumpling wrapper

Start dishwasher by pressing the start button

Opening both refrigerator doors

Shape the dough into a baguette

Slide down display screen

Put a fruit into a fruit bowl

Jump and kick the basketball

Turning on coffee machine using the knob

Turn rightward continuously

Kick the soccer ball to the right

Stand upright on two hind legs

Do a backflip and kick the soccer ball to the right

Put filling onto the wrapper

Turn on lamp by pressing the toggle button

Move clock ahead for daylight saving

BibTeX

@misc{wang2023robogen,
      title={RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation}, 
      author={Yufei Wang and Zhou Xian and Feng Chen and Tsun-Hsuan Wang and Yian Wang and Katerina Fragkiadaki and Zackory Erickson and David Held and Chuang Gan},
      year={2023},
      eprint={2311.01455},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
}