Active 3-D Shape Cosegmentation With Graph Convolutional Networks (original) (raw)

3D shape segmentation via shape fully convolutional networks

Computers & Graphics

We design a novel fully convolutional network architecture for shapes, denoted by Shape Fully Convolutional Networks (SFCN). 3D shapes are represented as graph structures in the SFCN architecture, based on novel graph convolution and pooling operations, which are similar to convolution and pooling operations used on images. Meanwhile, to build our SFCN architecture in the original image segmentation fully convolutional network (FCN) architecture, we also design and implement a generating operation with bridging function. This ensures that the convolution and pooling operation we have designed can be successfully applied in the original FCN architecture. In this paper, we also present a new shape segmentation approach based on SFCN. Furthermore, we allow more general and challenging input, such as mixed datasets of different categories of shapes which can prove the ability of our generalisation. In our approach, SFCNs are trained triangles-to-triangles by using three low-level geometric features as input. Finally, the feature voting-based multi-label graph cuts is adopted to optimise the segmentation results obtained by SFCN prediction. The experiment results show that our method can effectively learn and predict mixed shape datasets of either similar or different characteristics, and achieve excellent segmentation results.

Image Co-segmentation using Graph Convolution Neural Network

Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing, 2018

Image co-segmentation is jointly segmenting two or more images sharing common foreground objects. In this paper, we propose a novel graph convolution neural network (graph CNN) based end-to-end model for performing co-segmentation. At the beginning, each input image is over-segmented into a set of superpixels. Next, a weighted graph is formed using the over-segmented images exploiting spatial adjacency and both intra-image and inter-image feature similarities among the image superpixels (nodes). Subsequently, the proposed network, consisting of graph convolution layers followed by node classification layers, classifies each superpixel either into the common foreground or its complement. During training, along with the cosegmentation network, an additional network is introduced to exploit the corresponding semantic labels, and the two networks share the same weights in graph convolution layers. The whole model is learned in an end-to-end fashion using a novel cost function comprised of a superpixel wise binary cross entropy and a multi-label cross entropy. We also use empirical class probabilities in the loss function to deal with class imbalance. Experimental results reflect that the proposed technique is very competitive with the state-of-the-art methods on two challenging datasets, Internet and Pascal-VOC.

Learning Local Neighboring Structure for Robust 3D Shape Representation

Proceedings of the AAAI Conference on Artificial Intelligence

Mesh is a powerful data structure for 3D shapes. Representation learning for 3D meshes is important in many computer vision and graphics applications. The recent success of convolutional neural networks (CNNs) for structured data (e.g., images) suggests the value of adapting insight from CNN for 3D shapes. However, 3D shape data are irregular since each node's neighbors are unordered. Various graph neural networks for 3D shapes have been developed with isotropic filters or predefined local coordinate systems to overcome the node inconsistency on graphs. However, isotropic filters or predefined local coordinate systems limit the representation power. In this paper, we propose a local structure-aware anisotropic convolutional operation (LSA-Conv) that learns adaptive weighting matrices for each node according to the local neighboring structure and performs shared anisotropic filters. In fact, the learnable weighting matrix is similar to the attention matrix in random synthesizer -...

Efficiently consistent affinity propagation for 3D shapes co-segmentation

The Visual Computer, 2018

Unsupervised co-segmentation for a set of 3D shapes is a challenging problem as no prior information is provided. The accuracy of the current approaches is necessarily restricted by the accuracy of the unsupervised face classification, which is used to provide an initialization for the following optimization to improve the consistency between adjacent faces. However, it is exceedingly difficult to obtain a satisfactory initialization pre-segmentation owing to variation in topology and geometry of 3D shapes. In this study, we consider the unsupervised 3D shape co-segmentation as an exemplar-based clustering problem, aimed at simultaneously discovering optimal exemplars and obtaining co-segmentation results. Therefore, we introduce a novel exemplar-based clustering method based on affinity propagation for 3D shape co-segmentation, which can automatically identify representative exemplars and patterns in 3D shapes considering the high-order statistics, yielding consistent and accurate co-segmentation results. Experiments using various datasets, especially large sets with 200 or more shapes that would be challenging to manually segment, demonstrate that our method exhibits a better performance compared to state-ofthe-art methods.

Efficient Single-view 3D Co-segmentation using Shape Similarity and Spatial Part Relations

The practical use of the latest methods for supervised 3D shape co-segmentation is limited by the requirement of diverse training data and a watertight mesh representation. Driven by practical considerations, we assume only one reference shape to be available and the query shape to be provided as a partially visible point cloud. We propose a novel co-segmentation approach that constructs a part-based object representation comprised of shape appearance models of individual parts and isometric spatial relations between the parts. The partial query shape is pre-segmented using planar cuts, and the segments accompanied by the learned representation induce a compact Conditional Random Field (CRF). CRF inference is performed efficiently by A * -search with global optimality guarantees. A comparative evaluation with two baselines on partial views generated from the Labelled Princeton Segmentation Benchmark and point clouds recorded with an RGB-D sensor demonstrate superiority of the proposed approach both in accuracy and efficiency.

Image-driven unsupervised 3D model co-segmentation

The Visual Computer, 2019

Segmentation of 3D models is a fundamental task in computer graphics and vision. Geometric methods usually lead to non-semantic and fragmentary segmentations. Learning techniques require a large amount of labeled training data. In this paper we explore the feasibility of 3D model segmentation by taking advantage of the huge number of easy-to-obtain 2D realistic images available on the Internet. The regional color exhibited in images provides information that is valuable for segmentation. To transfer the segmentations, we first filter out inappropriate images with several criteria. The views of these images are estimated by our proposed texture-invariant view estimation Siamese Network. The training samples are generated by rendering-based synthesis without laborious labeling. Subsequently, we transfer and merge the segmentations produced by each individual image by applying registration and a graph-based aggregation strategy. The final result is obtained by combining all segmentations within the 3D model set. Our qualitative and quantitative experimental results on several model categories validate effectiveness of our proposed method.

Geometric deep learning on graphs and manifolds using mixture model CNNs

Deep learning has achieved a remarkable performance breakthrough in several fields, most notably in speech recognition, natural language processing, and computer vision. In particular, convolutional neural network (CNN) ar-chitectures currently produce state-of-the-art performance on a variety of image analysis tasks such as object detection and recognition. Most of deep learning research has so far focused on dealing with 1D, 2D, or 3D Euclidean-structured data such as acoustic signals, images, or videos. Recently, there has been an increasing interest in geometric deep learning, attempting to generalize deep learning methods to non-Euclidean structured data such as graphs and manifolds, with a variety of applications from the domains of network analysis, computational social science, or computer graphics. In this paper, we propose a unified framework allowing to generalize CNN architectures to non-Euclidean domains (graphs and manifolds) and learn local, stationary, and compositional task-specific features. We show that various non-Euclidean CNN methods previously proposed in the literature can be considered as particular instances of our framework. We test the proposed method on standard tasks from the realms of image-, graph-and 3D shape analysis and show that it consistently outper-forms previous approaches.

Shape Learning with Function-Described Graphs

Lecture Notes in Computer Science

A new method for shape learning is presented in this paper. This method incorporates abilities from both statistical and structural pattern recognition approaches to shape analysis. It borrows from statistical pattern recognition the capability of modelling sets of point coordinates, and from structural pattern recognition the ability of dealing with highly irregular patterns, such as those generated by points missingness. To that end we use a novel adaptation of Procrustes analysis, designed by us to align sets of points with missing elements. We use this information to generate sets of attributed graphs (AGs). From each set of AGs we synthesize a function-described graph (FDG), which is a type of compact representation that has the capability of probabilistic modelling of both structural and attribute information. Multivariate normal probability density estimation is used in FDGs instead of the originally used histograms. Comparative results of classification performance are presented of structural vs. attributes + structural information.

Active co-analysis of a set of shapes

ACM Transactions on Graphics, 2012

Unsupervised co-analysis of a set of shapes is a difficult problem since the geometry of the shapes alone cannot always fully describe the semantics of the shape parts. In this paper, we propose a semi-supervised learning method where the user actively assists in the co-analysis by iteratively providing inputs that progressively constrain the system. We introduce a novel constrained clustering method based on a spring system which embeds elements to better respect their inter-distances in feature space together with the user-given set of constraints. We also present an active learning method that suggests to the user where his input is likely to be the most effective in refining the results. We show that each single pair of constraints affects many relations across the set. Thus, the method requires only a sparse set of constraints to quickly converge toward a consistent and error-free semantic labeling of the set.

Deep similarity network fusion for 3D shape classification

The Visual Computer, 2019

In this paper, we introduce a deep similarity network fusion framework for 3D shape classification using a graph convolutional neural network, which is an efficient and scalable deep learning model for graph-structured data. The proposed approach coalesces the geometrical discriminative power of geodesic moments and similarity network fusion in an effort to design a simple, yet discriminative shape descriptor. This geometric shape descriptor is then fed into a graph convolutional neural network to learn a deep feature representation of a 3D shape. We validate the predictive power of our method on ModelNet shape benchmarks, demonstrating that the proposed framework yields significant performance gains compared to state-ofthe-art approaches.