GitHub - zerchen/hort: HORT: Monocular Hand-held Objects Reconstruction with Transformers (original) (raw)
This is the implementation of HORT, an state-of-the-art hand-held object reconstruction algorithm:
Installation 👷
git clone https://github.com/zerchen/hort.git
cd hort
The code has been tested with PyTorch 2.4.1 and CUDA 12.1. It is suggested to use an anaconda encironment to install the the required dependencies:
conda create --name hort python=3.12 conda activate hort
conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 pytorch-cuda=12.1 -c pytorch -c nvidia
Install requirements
pip install -r requirements.txt pip install -U git+https://github.com/luca-medeiros/lang-segment-anything.git conda install pytorch3d-0.7.8-py312_cu121_pyt241.tar.bz2 # https://anaconda.org/pytorch3d/pytorch3d/files?page=2 cd /home/zerchen/workspace/code/hort_init/hort/models/tgs/models/snowflake/pointnet2_ops_lib && python setup.py install
Download the pretrained models using:
wget https://huggingface.co/spaces/rolpotamias/WiLoR/resolve/main/pretrained_models/detector.pt -P ./pretrained_models/ wget https://huggingface.co/spaces/rolpotamias/WiLoR/resolve/main/pretrained_models/wilor_final.ckpt -P ./pretrained_models/ wget https://huggingface.co/zerchen/hort_models/resolve/main/hort_final.pth.tar -P ./pretrained_models/
It is also required to download MANO model from MANO website. Create an account by clicking Sign Up and download the models (mano_v*_*.zip). Unzip and place the right hand model MANO_RIGHT.pkl
under the mano_data/mano/
folder. Note that MANO model falls under the MANO license.
Demo 🎮
python demo.py --img_folder demo_img python vis_ho.py -e out_demo/test1.json # visualize the result in open3d
Start a local gradio demo 🤗
You can start a local demo for inference by running:
Acknowledgements
Parts of the code are based on WiLoR, SnowflakeNet and Lang-SAM.
License 📚
HORT is licensed under MIT License. This repository also depends on WiLoR, Ultralytics library and MANO Model, which are fall under their own licenses.
Citation 📝
If you find HORT useful for your research, please consider citing our paper:
@article{chen2025hort, title={{HORT}: Monocular Hand-held Objects Reconstruction with Transformers}, author={Chen, Zerui and Potamias, Rolandos Alexandros and Chen, Shizhe and Schmid, Cordelia}, journal={arXiv preprint arXiv:2503.21313}, year={2025} }