Visual Tracking of Deforming Objects Using Physics-based Models (original) (raw)

Simultaneous Tracking and Elasticity Parameter Estimation of Deformable Objects

2020 International Conference on Robotics and Automation (ICRA), 2020

In this paper, we propose a novel method to simultaneously track the deformation of soft objects and estimate their elasticity parameters. The tracking of the deformable object is performed by combining the visual information acquired with a RGB-D sensor with interactive Finite Element Method simulations of the object. The visual information is more particularly used to distort the simulated object. In parallel, the elasticity parameter estimation minimizes the error between the tracked object and a simulated object deformed by the forces that are measured using a force sensor. Once the elasticity parameters are estimated, our tracking algorithm can then also be used to estimate the deformation forces applied on an object without the use of a force sensor. We validated our method on several soft objects with di erent shape complexities. Our evaluations show the ability of our method to estimate the elasticity parameters as well as its use to estimate the forces applied on a deformab...

Tracking of Non-Rigid Objects using RGB-D Camera

SMC 2019 - IEEE International Conference on Systems, Man, and Cybernetics, Oct 2019, Bari, Italy, 2019

A method to accurately track deformable objects using a RGB-D camera with the help of a coarse object model is presented in this paper. The deformation model is based on corotational FEM formulation. The physical model of the object does not need to be exact, nor do we require the precise physical properties for accurately tracking the object. The position of the vertices of the surface mesh of the tracked object is deformed using a set of virtual forces. A point-to-plane distance based geometric error between the pointcloud and the mesh is minimized with respect to these virtual forces. The point of application of force is determined by analysis of the error obtained from rigid tracking, which is done in parallel with the non-rigid tracking. This architecture also enables the overall system to be realtime. The proposed approach is evaluated on a synthetic data with ground-truth for deformation at every frame, as well as on real data.

Automatic Modelling and Efficient Tracking of Deformable Objects

2005

This paper presents a system for efficiently tracking a deformable object in 3D. It is based on a model of the target represented as a set of textured features in 3D space and a set of shape bases, which encode the nonrigid modes of deformation. The model is constructed from a monocular sequence of 2D feature tracks using a non-rigid factorization algorithm followed by a non-linear optimization of the model parameters. Once the model is available, it is used for efficiently tracking the target using a 3D extension of the Inverse Compositional Algorithm which uses a projective camera model. In the experiments we show the performance of the system on synthetic and real sequences of a human face undergoing different facial expressions.

A finite element model for 3D shape reconstruction and nonrigid motion tracking

Computer Vision, 1993. …, 1993

The authors present a physics-based approach for recovering the 3-D shape and tracking the motion of nonrigid objects using a 3-D elastically deformable balloon model. The balloon model is based on a thin-plate under tension spline which deforms to fit visual data according to internal forces stemming from the elastic properties of the surface and external forces which are produced from the data. The finite element method is used to represent the model as a continuous surface. A natural finite element is used whose nodal ...

Using Physical Modeling and RGB-D Registration for Contact Force Sensing on Deformable Objects

Proceedings of the 14th International Conference on Informatics in Control, Automation and Robotics, 2017

In this paper we propose a method to estimate the force applied to a manipulated deformable object by processing information provided by an external vision sensor, in this case a consumer RGB-D camera. By measuring the deformations undergone by the object through a registration technique, the idea is to retrieve the contact force which minimises the deviation between the measured and the simulated deformations, given a simple interaction model and by employing a fitting process. The system resorts to a realistic mesh-based Finite Element Method model to accurately model deformations, whose elastic parameters are estimated in advance using the vision system and a force sensor. Experimental results are presented for the case of a compressive point-wise contact force applied, at static equilibrium, on a deformable object.

Shape Function-Based Estimation of Deformation with Moving Cameras Attached to the Deforming Body

ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, 2016

The paper presents a novel method to measure 3-D deformation of a large metallic frame structure of a crane under loading from one to several images, when the cameras need to be attached to the self deforming body, the structure sways during loading, and the imaging geometry is not optimal due to physical limitations. The solution is based on modeling the deformation with adequate shape functions and taking into account that the cameras move depending on the frame deformation. It is shown that the deformation can be estimated even from a single image of targeted points if the 3-D coordinates of the points are known or have been measured before loading using multiple cameras or some other measuring technique. The precision of the method is evaluated to be 1 mm at best, corresponding to 1:11400 of the average distance to the target.

Parametric modeling and linear estimation of elastic deformations

2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2011

We present a novel method to model and estimate elastic geometric deformations of an observed object, whether they are caused by the object's own dynamic behavior, or by the dynamic behavior of the imaging device, or both. A procedure for estimating the space of possible deformations the object may undergo based only on a set of observations is derived. This information is then employed to derive a linear estimator of the elastic deformation in any given observation. Application to change detection is presented.

Accurate Real Time Deformable Objects

cs.ubc.ca

We present an algorithm for fast, physically accurate sim-ulation of deformable objects suitable for real time ani-mation and virtual environment interaction. We describe the boundary integral equation formulation of static linear elasticity as well as the related Boundary Element ...

Tracking deformable surfaces that undergo topological changes using an RGB-D camera

We present a method for 3D tracking of deformable surfaces with dynamic topology, for instance a paper that undergoes cutting or tearing. Existing template-based methods assume a template of fixed topology. Thus, they fail in tracking deformable objects that undergo topological changes. In our work, we employ a dynamic template (3D mesh) whose topology evolves based on the topological changes of the observed geometry. Our tracking framework deforms the defined template based on three types of constraints (a) the surface of the template has to be registered to the 3D shape of the tracked surface, (b) the template deformation should respect feature (SIFT) correspondences between selected pairs of frames and, (c) the lengths of the template edges should be preserved. The latter constraint is relaxed when an edge is found to lie on a “geometric gap”, that is, when a significant depth discontinuity is detected along this edge. The topology of the template is updated on the fly by removing overstretched edges that lie on a geometric gap. The proposed method has been evaluated quantitatively and qualitatively in both synthetic and real sequences of monocular RGB-D views of surfaces that undergo various types of topological changes. The obtained results show that our approach tracks effectively objects with evolving topology and outperforms state of the art methods in tracking accuracy.