Lerrel Pinto (original) (raw)

Lerrel Pinto

I am an Assistant Professor of Computer Science at NYU Courant and part of the CILVR group. Before that, I was at UC Berkeley for a postdoc, at CMU Robotics Institute for a PhD, and at IIT Guwahati for undergrad. My short bio is here.

Research: I run the General-purpose Robotics and AI Lab (GRAIL) with the goal of getting robots to generalize and adapt in the messy world we live in. Our research focuses broadly on robot learning and decision making, with an emphasis on large-scale learning (both data and models), representation learning for sensory data, developing algorithms to model actions and behavior, reinforcement learning for adapting to new scenarios, and building open-sourced affordable robots. A talk on our robotics efforts is here. If you are interested in joining our lab, please read this.

Recent News

Courses Taught at NYU

Selected Research and Publications

Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments

Haritheja Etukuru, Norihito Naka, Zijin Hu, Seungjae Lee, Julian Mehu, Aaron Edsinger, Chris Paxton, Soumith Chintala, , Nur Muhammad Mahi Shafiullah

In ICRA 2025

AnySkin: Plug-and-play Skin Sensing for Robotic Touch

Raunaq Bhirangi, Venkatesh Pattabiraman, Enes Erciyes, Yifeng Cao, Tess Hellebrekers, Lerrel Pinto

In ICRA 2025

Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards

Irmak Güzey, Yinlong Dai, Georgy Savva, Raunaq Bhirangi, Lerrel Pinto

In ICRA 2025

DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation

Peiqi Liu, Zhanqiu Guo, Mohit Warke, Soumith Chintala, Chris Paxton, Nur Muhammad Mahi Shafiullah, Lerrel Pinto

In ICRA 2025

P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policies

Mara Levy, Siddhant Haldar, Lerrel Pinto, Abhinav Shirivastava

In ICRA 2025

Training Language Models on Synthetic Edit Sequences Improves Code Synthesis

Ulyana Piterbarg, Lerrel Pinto, Rob Fergus

In ICLR 2025

DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control

Zichen Jeff Cui, Hengkai Pan, Aadhithya Iyer, Siddhant Haldar, Lerrel Pinto

In NeurIPS 2024

BAKU: An Efficient Transformer for Multi-Task Policy Learning

Siddhant Haldar, Zhuoran Peng, Lerrel Pinto

In NeurIPS 2024

OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulation

Aadhithya Iyer, Zhuoran Peng, Yinlong Dai, Irmak Güzey, Siddhant Haldar, Soumith Chintala, Lerrel Pinto

In CoRL 2024

OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics

Peiqi Liu, Yaswanth Orru, Jay Vakil, Chris Paxton, Nur Muhammad Mahi Shafiullah, Lerrel Pinto

In RSS 2024

VQ-BeT: Behavior Generation with Latent Actions

Seungjae Lee, Yibin Wang, Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, Lerrel Pinto

In ICML 2024_(Spotlight Paper, Outstanding Paper Award at MFM-EAI at ICML 2024)_

Adaptive Sampling of k-space in Magnetic Resonance for Rapid Pathology Prediction

Chen-Yu Yen, Raghav Singhal, Umang Sharma, Rajesh Ranganath, Sumit Chopra, Lerrel Pinto

In ICML 2024

Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling

Raunaq Bhirangi, Chenyu Wang, Venkatesh Pattabiraman, Carmel Majidi, Abhinav Gupta, Tess Hellebrekers, Lerrel Pinto

In ICML 2024_(Best Paper Award at NGSM at ICML 2024)_

diff History for Neural Language Agents

Ulyana Piterbarg, Lerrel Pinto, Rob Fergus

In ICML 2024

On Bringing Robots Home

Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, Lerrel Pinto

In ICRA EXPO 2024_(Best Demo Award Finalist)_

See to Touch: Learning Tactile Dexterity through Visual Incentives

Irmak Güzey, Yinlong Dai, Ben Evans, Soumith Chintala, Lerrel Pinto

In ICRA 2024

NetHack is Hard to Hack

Ulyana Piterbarg, Lerrel Pinto, Rob Fergus

In NeurIPS 2023

Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play

Irmak Güzey, Ben Evans, Soumith Chintala, Lerrel Pinto

In CoRL 2023

That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulation

Abitha Thankaraj, Lerrel Pinto

In CoRL 2023

Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations

Siddhant Haldar, Jyothish Pari, Anant Rai, Lerrel Pinto

In RSS 2023_(Best Student Paper Award)_

CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory

Mahi Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, Arthur Szlam

In RSS 2023_(Outstanding Paper Award at the LangRob workshop)_

From play to policy: Conditional behavior generation from uncurated robot data

Jeff Cui, Yibin Wang, Mahi Shafiullah, Lerrel Pinto

In ICLR 2023_(Notable Top-5% Paper / Oral Presentation)_

Holo-Dex: Teaching Dexterity with Immersive Mixed Reality

Sridhar Pandian Arunachalam, Irmak Güzey, Soumith Chintala, Lerrel Pinto

In ICRA 2023

Behavior Transformers: Cloning k modes with one stone

Mahi Shafiullah, Jeff Cui, Ariuntuya Altanzaya, Lerrel Pinto

In NeurIPS 2022_(Nominated for Outstanding Paper Award)_

Watch and Match: Supercharging Imitation with Regularized Optimal Transport

Siddhant Haldar, Vaibhav Mathur, Denis Yarats, Lerrel Pinto

In CoRL 2022_(Best Paper Award Finalist)_

Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation

Sridhar Pandian Arunachalam, Sneha Silwal, Ben Evans, Lerrel Pinto

In ICRA 2023

Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning

Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, Lerrel Pinto

Preprint

The Surprising Effectiveness of Representation Learning for Visual Imitation

Jyothish Pari, Nur Muhammad Shafiullah, Sridhar Pandian Arunachalam, Lerrel Pinto

In RSS 2022

Context is Everything: Implicit Identification for Dynamics Adaptation

Ben Evans, Abitha Thankaraj, Lerrel Pinto

In ICRA 2022

Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning

Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto

In ICLR 2022

Reinforcement Learning with Prototypical Representations

Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto

In ICML 2021

Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency

Qiang Zhang, Tete Xiao, Alexei A. Efros, Lerrel Pinto, Xiaolong Wang

In ICLR 2021_(Oral Presentation)_

Self-Supervised Policy Adaptation during Deployment

Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenya, Pieter Abbeel, Alexei A. Efros, Lerrel Pinto, Xiaolong Wang

In ICLR 2021_(Spotlight Presentation)_

Task-Agnostic Morphology Evolution

Donald J. Hejna III, Pieter Abbeel, Lerrel Pinto

In ICLR 2021

Visual Imitation Made Easy

Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, Lerrel Pinto

In CoRL 2020

Learning Predictive Representations for Deformable Objects Using Contrastive Estimation

Wilson Yan, Ashwin Vangipuram,, Pieter Abbeel, Lerrel Pinto

In CoRL 2020

Robust Policies via Mid-Level Visual Representations

Bryan Chen, Alexander Sax, Francis E. Lewis, Silvio Savarese, Jitendra Malik, Amir Zamir, Lerrel Pinto

In CoRL 2020

Automatic Curriculum Learning through Value Disagreement

Yunzhi Zhang, Pieter Abbeel, Lerrel Pinto

In NeurIPS 2020

Generalized Hindsight for Reinforcement Learning

Alexander Li, Lerrel Pinto, Pieter Abbeel

In NeurIPS 2020

Reinforcement Learning with Augmented Data

Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, Aravind Srinivas

In NeurIPS 2020_(Spotlight Presentation)_

Learning to Manipulate Deformable Objects without Demonstrations

Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, Pieter Abbeel

In RSS 2020

Swoosh! Rattle! Thump! - Actions that Sound

Dhiraj Gandhi, Abhinav Gupta, Lerrel Pinto

In RSS 2020

Hierarchically Decoupled Imitation for Morphological Transfer

Donald J. Hejna III, Pieter Abbeel, Lerrel Pinto

In ICML 2020

PyRobot: An Open-source Robotics Framework for Research and Benchmarking

Adithya Murali, Tao Chen, Kalyan Vasudev Alwala, Dhiraj Gandhi, Lerrel Pinto, Saurabh Gupta, Abhinav Gupta

Open source platform (https://www.pyrobot.org/)

Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation

Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, Abhinav Gupta

In CoRL 2018

Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias

Abhinav Gupta, Adithya Murali, Dhiraj Gandhi, Lerrel Pinto

In NeurIPS 2018

Asymmetric Actor Critic for Image-Based Robot Learning

Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, Pieter Abbeel

In RSS 2018

CASSL: Curriculum Accelerated Self-Supervised Learning

Adithyavairavan Murali, Lerrel Pinto, Dhiraj Gandhi, Abhinav Gupta

In ICRA 2018

Learning to Fly by Crashing

Dhiraj Gandhi, Lerrel Pinto, Abhinav Gupta

In IROS 2017

Robust Adversarial Reinforcement Learning

Lerrel Pinto, James Davidson, Rahul Sukthankar, Abhinav Gupta

In ICML 2017

Learning to Push by Grasping: Using multiple tasks for effective learning

Lerrel Pinto, Abhinav Gupta

In ICRA 2017

Supervision via Competition: Robot Adversaries for Learning Tasks

Lerrel Pinto, James Davidson, Abhinav Gupta

In ICRA 2017

The Curious Robot: Learning Visual Representations via Physical Interactions

Lerrel Pinto, Dhiraj Gandhi, Yuanfeng Han, Yong-Lae Park, Abhinav Gupta

In ECCV 2016_(Spotlight Presentation)_

Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours

Lerrel Pinto, Abhinav Gupta

In ICRA 2016_(Best Student Paper Award)_