Learning Pyramidal Feature Hierarchy for 3D Reconstruction (original) (raw)

Learning to Reconstruct High-Quality 3D Shapes with Cascaded Fully Convolutional Networks

Computer Vision – ECCV 2018, 2018

We present a data-driven approach to reconstructing highresolution and detailed volumetric representations of 3D shapes. Although well studied, algorithms for volumetric fusion from multi-view depth scans are still prone to scanning noise and occlusions, making it hard to obtain high-fidelity 3D reconstructions. In this paper, inspired by recent advances in efficient 3D deep learning techniques, we introduce a novel cascaded 3D convolutional network architecture, which learns to reconstruct implicit surface representations from noisy and incomplete depth maps in a progressive, coarse-to-fine manner. To this end, we also develop an algorithm for end-to-end training of the proposed cascaded structure. Qualitative and quantitative experimental results on both simulated and real-world datasets demonstrate that the presented approach outperforms existing state-of-the-art work in terms of quality and fidelity of reconstructed models.

A 3D template-based point generation network for 3D reconstruction from single images

Applied Soft Computing, 2021

Learning-based approaches in 3D reconstruction problem have attracted researchers, due to the excellent performance of this approach in image segmentation and image classification. The increasing attention to the learning-based approach for the 3D reconstruction application is also due to the availability of 3D datasets shared publicly, such as ShapeNet and ModelNet datasets. Several deep learning approaches use voxel representation-based approaches. However, voxel-based methods suffer from inefficiency and inability to create higher dimensional 3D results. Another representation is by using point cloud representation, an unstructured 3D points in the object's surface. However, learning such irregular structures is a challenging task due to the unordered properties of such representations. This paper proposes a new framework for 3D reconstruction of 2D images that introduces a 3D template-based point generation network. The 3D template-based point generation network infers a 3D template and generates 3D point clouds representing the reconstructed 3D object, based on an input image. The proposed network introduces two inputs, the encoded 2D image and the encoded 3D point template produced by an image classification module and a 3D template generation module. Experiments on the ShapeNet dataset show better performance than existing methods in terms of the Chamfer distance between the 3D ground-truth data and the 3D reconstructed data.

NeuralNetwork Based 3D Surface Reconstruction

Computing Research Repository, 2009

This paper proposes a novel neural-network-based adaptive hybrid-reflectance three-dimensional (3-D) surface reconstruction model. The neural network combines the diffuse and specular components into a hybrid model. The proposed model considers the characteristics of each point and the variant albedo to prevent the reconstructed surface from being distorted. The neural network inputs are the pixel values of the two-dimensional images to be reconstructed. The normal vectors of the surface can then be obtained from the output of the neural network after supervised learning, where the illuminant direction does not have to be known in advance. Finally, the obtained normal vectors can be applied to integration method when reconstructing 3-D objects. Facial images were used for training in the proposed approach

IRJET-Reconstruction of 3D Objects using Neural Network

IRJET, 2020

Motivated from the recent developments of mechanisms that make use of shapes to achieve 3D reconstructions, we present a system built on neural networks. Our network acquires skill to map target images to their respective 3D reconstructions with the help of artificial data. Multi-view images of target is fed to this network as input and after completion of the process the network returns target's reconstructed view which is the form of 3D.This network does not need any notations on the target image for training or testing which makes it different from other recent works. This network is an add-on to the typical LSTM networks that can easily fit in the images of target in a proper way.

3D Object Reconstruction from Imperfect Depth Data Using Extended YOLOv3 Network

Sensors, 2020

State-of-the-art intelligent versatile applications provoke the usage of full 3D, depth-based streams, especially in the scenarios of intelligent remote control and communications, where virtual and augmented reality will soon become outdated and are forecasted to be replaced by point cloud streams providing explorable 3D environments of communication and industrial data. One of the most novel approaches employed in modern object reconstruction methods is to use a priori knowledge of the objects that are being reconstructed. Our approach is different as we strive to reconstruct a 3D object within much more difficult scenarios of limited data availability. Data stream is often limited by insufficient depth camera coverage and, as a result, the objects are occluded and data is lost. Our proposed hybrid artificial neural network modifications have improved the reconstruction results by 8.53% which allows us for much more precise filling of occluded object sides and reduction of noise d...

3D object reconstruction and representation using neural networks

… of the 2nd international conference on …, 2004

3D object reconstruction is frequent used in various fields such as product design, engineering, medical and artistic applications. Numerous reconstruction techniques and software were introduced and developed. However, the purpose of this paper is to fully integrate an adaptive artificial neural network (ANN) based method in reconstructing and representing 3D objects. This study explores the ability of neural networks in learning through experience when reconstructing an object by estimating it's z-coordinate. Neural networks' capability in representing most classes of 3D objects used in computer graphics is also proven. Simple affined transformation is applied on different objects using this approach and compared with the real objects. The results show that neural network is a promising approach for reconstruction and representation of 3D objects.

Next-best-view regression using a 3D convolutional neural network

Machine Vision and Applications, 2021

Automated three-dimensional (3D) object reconstruction is the task of building a geometric representation of a physical object by means of sensing its surface. Even though new single view reconstruction techniques can predict the surface, they lead to incomplete models, specially, for non commons objects such as antique objects or art sculptures. Therefore, to achieve the task's goals, it is essential to automatically determine the locations where the sensor will be placed so that the surface will be completely observed. This problem is known as the next-best-view problem. In this paper, we propose a data-driven approach to address the problem. The proposed approach trains a 3D convolutional neural network (3D CNN) with previous reconstructions in order to regress the position of the next-best-view. To the best of our knowledge, this is one of the first works that directly infers the next-best-This work was partially supported by CONACYT-cátedra 1507 project.

High-Quality Textured 3D Shape Reconstruction with Cascaded Fully Convolutional Networks

IEEE Transactions on Visualization and Computer Graphics, 2019

We present a learning-based approach to reconstructing high-resolution three-dimensional (3D) shapes with detailed geometry and high-fidelity textures. Albeit extensively studied, algorithms for 3D reconstruction from multi-view depth-and-color (RGB-D) scans are still prone to measurement noise and occlusions; limited scanning or capturing angles also often lead to incomplete reconstructions. Propelled by recent advances in 3D deep learning techniques, in this paper, we introduce a novel computation-and memory-efficient cascaded 3D convolutional network architecture, which learns to reconstruct implicit surface representations as well as the corresponding color information from noisy and imperfect RGB-D maps. The proposed 3D neural network performs reconstruction in a progressive and coarse-to-fine manner, achieving unprecedented output resolution and fidelity. Meanwhile, an algorithm for end-to-end training of the proposed cascaded structure is developed. We further introduce Human10, a newly created dataset containing both detailed and textured full-body reconstructions as well as corresponding raw RGB-D scans of 10 subjects. Qualitative and quantitative experimental results on both synthetic and real-world datasets demonstrate that the presented approach outperforms existing state-of-the-art work regarding visual quality and accuracy of reconstructed models.

Learning Unsupervised Hierarchical Part Decomposition of 3D Objects From a Single RGB Image

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020

Humans perceive the 3D world as a set of distinct objects that are characterized by various low-level (geometry, reflectance) and high-level (connectivity, adjacency, symmetry) properties. Recent methods based on convolutional neural networks (CNNs) demonstrated impressive progress in 3D reconstruction, even when using a single 2D image as input. However, the majority of these methods focuses on recovering the local 3D geometry of an object without considering its part-based decomposition or relations between parts. We address this challenging problem by proposing a novel formulation that allows to jointly recover the geometry of a 3D object as a set of primitives as well as their latent hierarchical structure without part-level supervision. Our model recovers the higher level structural decomposition of various objects in the form of a binary tree of primitives, where simple parts are represented with fewer primitives and more complex parts are modeled with more components. Our experiments on the ShapeNet and D-FAUST datasets demonstrate that considering the organization of parts indeed facilitates reasoning about 3D geometry.

A neural network for recovering 3D shape from erroneous and few depth maps of shaded images

In this paper, we present a new neural network (NN) for three-dimensional (3D) shape reconstruction. This NN provides an analytic mapping of an initial 3D polyhedral model into its projection depth images. Through this analytic mapping, the NN can analytically refine vertices position of the model using error back-propagation learning. This learning is based on shape-from-shading (SFS) depth maps taken from multiple views. The depth maps are obtained by Tsai–Shah SFS algorithm. They are considered as partial 3D shapes of the object to be reconstructed. The task is to reconstruct an accurate and complete representation of a given object relying only on a limited number of views and erroneous SFS depth maps. Through hierarchical reconstruction and annealing reinforcement strategies, our reconstruction system gives more exact and stable results. In addition, it corrects and smoothly fuses the erroneous SFS depth maps. The implementation of this neural network algorithm used in this paper is available at http://kumazawa-www.cs.titech.ac.jp/\~fanany/MV-SPRNN/mv-sprnn.html.