GitHub - yixxuan-li/OmniClone (original) (raw)

OmniClone is a whole-body humanoid teleoperation system designed for affordable, reproducible, and versatile robot control across diverse operator body shapes and task scenarios.

[Paper] [Project Page]

Overview

OmniClone takes a systematic perspective to develop an affordable, robust, and versatile whole-body teleoperation system. The system provides:

• Bias mitigation — identification and correction of distortions arising from retargeting and hardware fluctuations
• Transformer-based whole-body control — robust, affordable humanoid control with integrated optimizations
• Multi-operator support — stable operation across operator heights ranging from 147 cm to 194 cm
• High-fidelity dexterous manipulation — long-horizon task execution across diverse domestic environments
• Autonomy policy training — expert trajectory generation enabling downstream robot learning
• Zero-shot motion execution — diverse motion generation without task-specific training

Operation Modes

Table of Contents

• Release Status
• Requirements
• Repository Layout
• Environment Setup
• Optional Onboard Setup
• Network Setup
• Run OmniClone
• Documentation and Related Projects
• License
• Citation
• Acknowledgements

Release Status

Requirements

Software Platform (Host PC)

Core Hardware

Optional Peripherals

Repository Layout

Environment Setup

These steps assume Ubuntu, an NVIDIA GPU, and CUDA 12.4 are already available.

Run all commands from the repository root unless noted otherwise.

1. Create the Conda Environment

conda create -n omniclone python=3.10 -y conda activate omniclone

Fix potential Mujoco / OpenGL rendering issues

conda install -c conda-forge libstdcxx-ng -y

2. Install Local Packages

GMR

cd GMR pip install -e . cd ..

For dataset and body-model preparation details, see GMR/README.md.

Sim2Everything

cd Sim2Everything pip install -e . cd ..

If Sim2Everything/ is not already present in your workspace, clone it from https://github.com/Yutang-Lin/Sim2Everything first.

3. Install Common Python Dependencies

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

pip install opencv-python pillow scipy numpy matplotlib tqdm pip install unitree-sdk2py dex-retargeting pip install cyclonedds pyserial pyngrok pyyaml rich pyqt5 pyqtgraph

4. Install Feature-Specific Components

Nokov MoCap SDK

The Nokov wheel is bundled in sdks/:

pip install sdks/nokovpy-3.0.1-py3-none-any.whl

Pinocchio (Optional)

conda install pinocchio -c conda-forge

Pico XR SDK

Headset client:

• Install the Pico client from the XRoboToolkit Unity client release page.

PC service:

sudo dpkg -i XRoboToolkit_PC_Service_1.0.0_ubuntu_22.04_amd64.deb

Start xrobotoolkit-pc-service before launching teleoperation.

Python bindings:

git clone https://github.com/YanjieZe/XRoboToolkit-PC-Service-Pybind.git cd XRoboToolkit-PC-Service-Pybind

mkdir -p tmp cd tmp git clone https://github.com/XR-Robotics/XRoboToolkit-PC-Service.git cd XRoboToolkit-PC-Service/RoboticsService/PXREARobotSDK bash build.sh cd ../../../..

mkdir -p lib include cp tmp/XRoboToolkit-PC-Service/RoboticsService/PXREARobotSDK/PXREARobotSDK.h include/ cp -r tmp/XRoboToolkit-PC-Service/RoboticsService/PXREARobotSDK/nlohmann include/nlohmann/ cp tmp/XRoboToolkit-PC-Service/RoboticsService/PXREARobotSDK/build/libPXREARobotSDK.so lib/

conda install -c conda-forge pybind11 pip uninstall -y xrobotoolkit_sdk python setup.py install

Runtime packages:

pip install xrobotoolkit-sdk xrobotoolkit-teleop

Optional Onboard Setup

This section only applies to the full robot configuration with a 2-DoF head, ego-view camera, and Inspire hands.

If those services are maintained outside this repository, make sure they are already running before you launch server_head.py.

Network Setup

All machines should be on the same local network.

Quick checklist:

• Keep the host PC, robot, and headset / MoCap server on the same subnet.
• Run server.py or server_head.py on one clearly designated host.
• Point --client_ip to that host, or use 127.0.0.1 if everything runs on the same machine.

Disable the firewall on the Ubuntu host if needed:

Network Topology

┌─────────────────────────────────────────────────────────────────────┐
│                           Local Network                             │
│                                                                     │
│  ┌──────────────┐    Wi-Fi / UDP    ┌─────────────────────────────┐ │
│  │ Pico XR      │ ───────────────── │ Host PC                     │ │
│  │ Headset      │   body tracking   │ motion retargeting          │ │
│  └──────────────┘                   │ policy inference            │ │
│                                     │ low-level control           │ │
│  ┌──────────────┐    SDK / LAN      │                             │ │
│  │ Nokov MoCap  │ ───────────────── │ `pico_to_robot.py`          │ │
│  │ Server       │                   │ `nokov_to_robot.py`         │ │
│  └──────────────┘                   │ `server.py`                 | │
│                                     │ `server_head.py`            │ │
│                                     │ `lowcmd_publisher.py`       │ │
│                                     └────────────┬────────────────┘ │
│                                                  │ DDS / ROS 2      │
│                                     ┌────────────▼────────────────┐ │
│                                     │ Unitree G1 Robot            │ │
│                                     │ optional D455 / hands / head│ │
│                                     └─────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘

Run OmniClone

All commands below assume the repository root as the working directory.

Safety Checklist

Before enabling policy execution:

• Make sure the robot is physically supported and has free space around it.
• Source the required ROS 2 environment before running lowcmd_publisher.py.
• Verify that the correct policy checkpoint exists in models/.
• Confirm that your motion-input source is streaming normally.

Safety-critical: After starting server.py and before pressing L1 to begin policy inference, press R2 once to normalize heading alignment. Skipping this step may cause a sudden turn at startup.

A. Standard Teleoperation (No Head / No Hands)

This is the default real-time setup. Open the following processes in separate terminals on the host PC.

Host terminal 1: low-level motor command publisher

python lowcmd_publisher.py

Host terminals 2-3: choose one motion-input backend

Nokov MoCap:

python nokov_to_robot.py
--robot unitree_g1
--mocap_ip 192.168.123.101
--client_ip 127.0.0.1
--client_port 7002

policy server

python server.py
--server_ip 0.0.0.0
--server_port 7002

Pico XR:

python pico_to_robot.py
--robot unitree_g1
--client_ip 127.0.0.1
--client_port 7002

policy server

python server.py
--server_ip 0.0.0.0
--server_port 7002

server.py will move the G1 toward its default pose before policy execution begins.

Safety-critical: After starting server.py and before pressing L1 to begin policy inference, press R2 once to normalize heading alignment. Skipping this step may cause a sudden turn at startup.

B. Full Teleoperation + Episode Recording (Pico Only)

Use this mode when you need:

• Pico-based whole-body teleoperation
• optional hand control
• ego-view image collection
• episode recording for downstream VLA workflows

Recommended launch order:

Onboard terminal 1: Realsense image stream

cd XRoboToolkit-Orin-Video-Sender ./open_image_sender.sh

Onboard terminal 2: Inspire hand control server

cd inspire_hand_ws/inspire_hand_sdk/example python Headless_driver_double.py

Host terminal 1: low-level motor publisher

python lowcmd_publisher.py

Host terminal 2: Pico retargeting bridge

python pico_to_robot_whand.py
--robot unitree_g1
--use_hand
--client_ip 127.0.0.1
--client_port 7002

Host terminal 3: head-enabled server

python server_head.py
--server_ip 0.0.0.0
--server_port 7002

Gamepad Controls

server_head.py will move the G1 toward its default pose before policy execution begins.

Safety-critical: After starting server_head.py and before pressing L1 to begin policy inference, press R2 once to normalize heading alignment. Skipping this step may cause a sudden turn at startup.

Pico Controller Hand Controls

These bindings refer to the Pico controller, not the gamepad controls listed above.

Related Projects

Related projects:

• Sim2Everything: Sim-to-real transfer utilities for humanoid robots
• GMR: Real-time human-to-humanoid motion retargeting on CPU
• Gen2Humanoid: Text-to-motion pipeline for humanoid robots
• Clone: A related humanoid teleoperation system
• COLA: Proprioception-only learning for human-humanoid collaborative carrying
• LessMimic: Long-horizon humanoid-scene interaction with unified distance fields

License

This project is released under the MIT License.

Citation

@article{omniclone2026, title = {OmniClone: Engineering a Robust, All-Rounder Whole-Body Humanoid Teleoperation System}, author = {Yixuan Li and Le Ma and Yutang Lin and Yushi Du and Mengya Liu and Kaizhe Hu and Jieming Cui and Yixin Zhu and Wei Liang and Baoxiong Jia and Siyuan Huang}, journal = {arXiv preprint}, year = {2026} }

Acknowledgements

We extend our sincere gratitude to Peiyang Li, Zimeng Yuan, Zhen Chen, Chengcheng Zhang, Yang Zhang, Nian Liu, Zhidan Liu, Zihui Liu and Mulin Sui, for their invaluable help in filming this demo.