GitHub - DaLi-Jack/SSR-code: Official implementation of 3DV24 paper "Single-view 3D Scene Reconstruction with High-fidelity Shape and Texture" (original) (raw)

Paper PDF Paper arXiv Project Page

Yixin Chen*,Junfeng Ni*,Nan Jiang,Yaowei Zhang,Yixin Zhu,Siyuan Huang

News

Abstract

We propose a novel framework for simultaneous high-fidelity recovery of object shapes and textures from single-view images. Our approach utilizes SSR, Single-view neural implicit Shape and Radiance field representations, leveraging explicit 3D shape supervision and volume rendering of color, depth, and surface normal images. To overcome shape-appearance ambiguity under partial observations, we introduce a two-stage learning curriculum that incorporates both 3D and 2D supervisions. A distinctive feature of our framework is its ability to generate fine-grained textured meshes while seamlessly integrating rendering capabilities into the single-view 3D reconstruction model. Beyond individual objects, our approach facilitates composing object-level representations into flexible scene representations, thereby enabling applications such as holistic scene understanding and 3D scene editing.

Setup

conda create -n ssr python=3.8 conda activate ssr conda install pytorch==1.10.0 torchvision==0.11.0 cudatoolkit=11.3 -c pytorch -c conda-forge pip install -r requirements.txt

Data

Please download the preprocessed data and unzip in the data folder. The resulting folder structure should be:

└── SSR-code
  └── data
    ├── FRONT3D
    ├── FRONT3D-demo
    ├── Pix3D
    ├── SUNRGBD

Since the full FRONT3D dataset is large, we pick some demo data from the test set, namely FRONT3D-demo, to demonstrate the results of the method.

Checkpoints

Please download our pre-trained model and unzip in the output folder, the resulting folder structure should be:

└── SSR-code
  └── output
    └── front3d_ckpt
      ├── model_latest.pth
    └── pix3d_ckpt
      ├── model_latest.pth

Training

NOTE: set show_rendering=False

We use 4xA100-80GB gpus to train our model, if you want to reproduce our model,

please set gpu_ids='0,1,2,3', batch_size.train=96, batch_size.val=16 for 3D-FRONT dataset.

and for Pix3D dataset, batch size can be bigger, e.g. batch_size.train=128, batch_size.val=64.

for 3D-FRONT dataset

python train.py --config configs/train_front3d.yaml

for Pix3D dataset

python train.py --config configs/train_pix3d.yaml

Inference

Export mesh with texture

NOTE: set show_rendering=False, eval.export_mesh=True, eval.export_color_mesh=True

for 3D-FRONT dataset

python inference.py --config configs/train_front3d.yaml

for Pix3D dataset

python inference.py --config configs/train_pix3d.yaml

for SUNRGB-D dataset

python inference_sunrgbd.py --config configs/train_sunrgbd.yaml

Novel view synthesis

NOTE: set show_rendering=True

python inference_rot_angle.py --config configs/train_front3d.yaml

Scene fusion

NOTE: set show_rendering=True, eval.fusion_scene=True

remember to change data.batch_size.test to the number of objects!

please carefully compare the differences between train_front3d.yaml and train_front3d_fusion.yaml

python inference_rot_angle.py --config configs/train_front3d_fusion.yaml

Evaluation

For evaluation, gaps is required to conduct ICP alignment. Run the following commands to install gaps:

cd external bash build_gaps.sh

run the following commands for evaluation:

NOTE: set eval.export_mesh=True, eval.export_color_mesh=False

for 3D-FRONT dataset

bash eval/evaluate.sh configs/train_front3d.yaml

for Pix3D dataset

bash eval/evaluate.sh configs/train_pix3d.yaml

Citation

If you find our project useful, please cite:

@inproceedings{chen2023ssr, title={Single-view 3D Scene Reconstruction with High-fidelity Shape and Texture}, author={Chen, Yixin and Ni, Junfeng and Jiang, Nan and Zhang, Yaowei and Zhu, Yixin and Huang, Siyuan}, booktitle=ThreeDV, year={2024} }

Acknowledgements

Some codes are borrowed from InstPIFu, PixelNeRF and MonoSDF. We thank all the authors for their great work.