Patrick van der Smagt | Technische Universität München (original) (raw)

Papers by Patrick van der Smagt

Research paper thumbnail of Two-Stream RNN/CNN for Action Recognition in 3D Videos

The recognition of actions from video sequences has many applications in health monitoring, assis... more The recognition of actions from video sequences has many applications in health monitoring, assisted living, surveillance, and smart homes. Despite advances in sensing, in particular related to 3D video, the methodologies to process the data are still subject to research. We demonstrate superior results by a system which combines recurrent neural networks with convolutional neural networks in a voting approach. The gated-recurrent-unit-based neural networks are particularly well-suited to distinguish actions based on long-term information from optical tracking data; the 3D-CNNs focus more on detailed, recent information from video data. The resulting features are merged in an SVM which then classifies the movement. In this architecture, our method improves recognition rates of state-of-the-art methods by 14% on standard data sets.

Research paper thumbnail of Deep Variational Bayes Filters: Unsupervised learning of state space models from raw data

We introduce Deep Variational Bayes Filters (DVBF), a new method for unsupervised learning and id... more We introduce Deep Variational Bayes Filters (DVBF), a new method for unsupervised learning and identification of latent Markovian state space models. Leveraging recent advances in Stochastic Gradient Variational Bayes, DVBF can overcome intractable inference distributions via variational inference. Thus, it can handle highly nonlinear input data with temporal and spatial dependencies such as image sequences without domain knowledge. Our experiments show that enabling back-propagation through transitions enforces state space assumptions and significantly improves information content of the latent embedding. This also enables realistic long-term prediction.

Research paper thumbnail of CNN-based Segmentation of Medical Imaging Data

Convolutional neural networks have been applied to a wide variety of computer vision tasks. Recen... more Convolutional neural networks have been applied to a wide variety of computer vision tasks. Recent advances in semantic segmentation have enabled their application to medical image segmentation. While most CNNs use two-dimensional kernels, recent CNN-based publications on medical image segmentation featured three-dimensional kernels, allowing full access to the three-dimensional structure of medical images. Though closely related to semantic segmentation, medical image segmentation includes specific challenges that need to be addressed, such as the scarcity of labelled data, the high class imbalance found in the ground truth and the high memory demand of three-dimensional images. In this work, a CNN-based method with three-dimensional filters is demonstrated and applied to hand and brain MRI. Two modifications to an existing CNN architecture are discussed, along with methods on addressing the aforementioned challenges. While most of the existing literature on medical image segmentation focuses on soft tissue and the major organs, this work is validated on data both from the central nervous system as well as the bones of the hand.

Research paper thumbnail of Evaluation of joint type modelling in the human hand

This short communication presents preliminary results from an extensive investigation of joint mo... more This short communication presents preliminary results from an extensive investigation of joint modelling for the human hand. We use finger and hand movement data recorded from both hands of 110 subjects using passive reflective markers on the skin. Furthermore, we use data which was recorded from a single Thiel-fixated cadaver hand using also passive reflective markers but fixed to the bone. Our data clearly demonstrate that, for wrist and finger joints, hinge joint models are suciently accurate to describe their movement in Cartesian space.

Research paper thumbnail of Measuring fingertip forces from camera images for random finger poses

2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015

Robust fingertip force detection from fingernail image is a critical strategy that can be applied... more Robust fingertip force detection from fingernail image is a critical strategy that can be applied in many areas. However, prior research fixed many variables that influence the finger color change. This paper analyzes the effect of the finger joint on the force detection in order to deal with the constrained finger position setting. A force estimator method is designed: a model to predict the fingertip force from finger joints measured from 2D cameras and 3 rectangular markers in cooperation with the fingernail images are trained. Then the error caused by the color changes of the joint bending can be avoided. This strategy is a significant step forward from a finger force estimator that requires tedious finger joint setting.

Research paper thumbnail of OsiSma2013

Research paper thumbnail of On fast dropout and its applicability to recurrent networks

Recurrent Neural Networks (RNNs) are rich models for the processing of sequential data. Recent wo... more Recurrent Neural Networks (RNNs) are rich models for the processing of sequential data. Recent work on advancing the state of the art has been focused on the optimization or modelling of RNNs, mostly motivated by adressing the problems of the vanishing and exploding gradients. The control of overfitting has seen considerably less attention. This paper contributes to that by analyzing fast dropout, a recent regularization method for generalized linear models and neural networks from a back-propagation inspired perspective. We show that fast dropout implements a quadratic form of an adaptive, per-parameter regularizer, which rewards large weights in the light of underfitting, penalizes them when overfitting and vanishes at minima of an unregularized training loss. The derivatives of that regularizer are exclusively based on the training error signal. One consequence of this is the absense of a global weight attractor, which is particularly appealing for RNNs, since the dynamics are not biased towards a certain regime. We positively test the hypothesis that this improves the performance of RNNs on four musical data sets and a natural language processing (NLP) task.

Research paper thumbnail of Computing grip force and torque from finger nail images using Gaussian processes

2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013

We demonstrate a simple approach with which finger force can be measured from nail coloration. By... more We demonstrate a simple approach with which finger force can be measured from nail coloration. By automatically extracting features from nail images of a finger-mounted CCD camera, we can directly relate these images to the force measured by a force-torque sensor. The method automatically corrects orientation and illumination di erences.

Research paper thumbnail of Estimating finger grip force from an image of the hand using Convolutional Neural Networks and Gaussian processes

2014 IEEE International Conference on Robotics and Automation (ICRA), 2014

ABSTRACT Estimating human fingertip forces is required to understand force distribution in graspi... more ABSTRACT Estimating human fingertip forces is required to understand force distribution in grasping and manipulation. Human grasping behavior can then be used to develop force-and impedance-based grasping and manipulation strategies for robotic hands. However, estimating human grip force naturally is only possible with instrumented objects or unnatural gloves, thus greatly limiting the type of objects used. In this paper we describe an approach which uses images of the human fingertip to reconstruct grip force and torque at the finger. Our approach does not use finger-mounted equipment, but instead a steady camera observing the fingers of the hand from a distance. This allows for finger force estimation without any physical interference with the hand or object itself, and is therefore universally applicable. We construct a 3-dimensional finger model from 2D images. Convolutional Neural Networks (CNN) are used to predict the 2D image to a 3D model transformation matrix. Two methods of CNN are designed for separate and combined outputs of orientation and position. After learning, our system shows an alignment accuracy over 98% on unknown data. In the final step, a Gaussian process estimates finger force and torque from the aligned images based on color changes and deformations of the nail and its surrounding skin. Experimental results shows that the accuracy achieves about 95% in the force estimation and 90% in the torque.

Research paper thumbnail of Convolutional Neural Networks Learn Compact Local Image Descriptors

Lecture Notes in Computer Science, 2013

A standard deep convolutional neural network paired with a suitable loss function learns compact ... more A standard deep convolutional neural network paired with a suitable loss function learns compact local image descriptors that perform comparably to state-of-the art approaches.

Research paper thumbnail of Training Neural Networks with Implicit Variance

Lecture Notes in Computer Science, 2013

We present a novel method to train predictive Gaussian distributions p(z|x) for regression proble... more We present a novel method to train predictive Gaussian distributions p(z|x) for regression problems with neural networks. While most approaches either ignore or explicitly model the variance as another response variable, it is trained implicitly in our case. Establishing stochasticty by the injection of noise into the input and hidden units, the outputs are approximated with a Gaussian distribution by the forward propagation method introduced for fast dropout [1]. We have designed our method to respect that probabilistic interpretation of the output units in the loss function. The method is evaluated on a synthetic and a inverse robot dynamics task, yielding superior performance to plain neural networks, Gaussian processes and LWPR in terms of mean squared error and likelihood.

Research paper thumbnail of Unsupervised Feature Learning for low-level Local Image Descriptors

Unsupervised feature learning has shown impressive results for a wide range of input modalities, ... more Unsupervised feature learning has shown impressive results for a wide range of input modalities, in particular for object classification tasks in computer vision. Using a large amount of unlabeled data, unsupervised feature learning methods are utilized to construct high-level representations that are discriminative enough for subsequently trained supervised classification algorithms. However, it has never been \emph{quantitatively} investigated yet how well unsupervised learning methods can find \emph{low-level representations} for image patches without any additional supervision. In this paper we examine the performance of pure unsupervised methods on a low-level correspondence task, a problem that is central to many Computer Vision applications. We find that a special type of Restricted Boltzmann Machines (RBMs) performs comparably to hand-crafted descriptors. Additionally, a simple binarization scheme produces compact representations that perform better than several state-of-the...

Research paper thumbnail of Continuous robot control using surface electromyography of atrophic muscles

2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013

The development of new, light robotic systems has opened up a wealth of human-robot interaction a... more The development of new, light robotic systems has opened up a wealth of human-robot interaction applications. In particular, the use of robot manipulators as personal assistant for the disabled is realistic and affordable, but still requires research as to the brain-computer interface.

Research paper thumbnail of Identification of Human Limb Stiffness in 5 DoF and Estimation via EMG

Springer Tracts in Advanced Robotics, 2013

To approach robustness and optimal performance, biological musculoskeletal systems can adapt thei... more To approach robustness and optimal performance, biological musculoskeletal systems can adapt their impedance while interacting with their environment. This property has motivated modern robotic designs including variable-impedance actuators and control methods, based on the capability to vary visco-elastic properties actively or passively. Even though variable-impedance actuation and impedance control in robotics is resolved to a great part, a general set of rules by which impedance is adjusted related to the task at hand is still lacking. This paper aims to fill this gap by providing a method to estimate the stiffness of the human arm in more than two degrees of freedom by perturbation. To overcome ill-conditionedness of the impedance and inertial matrices, we propose and validate methods to separately identify inertial and stiffness parameters. Finally, a model is proposed to estimate the joint stiffness from EMG-measurements of muscle activities.

Research paper thumbnail of Learning Sequence Neighbourhood Metrics

Lecture Notes in Computer Science, 2012

Recurrent neural networks (RNNs) in combination with a pooling operator and the neighbourhood com... more Recurrent neural networks (RNNs) in combination with a pooling operator and the neighbourhood components analysis (NCA) objective function are able to detect the characterizing dynamics of sequences and embed them into a fixed-length vector space of arbitrary dimensionality. Subsequently, the resulting features are meaningful and can be used for visualization or nearest neighbour classification in linear time. This kind of metric learning for sequential data enables the use of algorithms tailored towards fixed length vector spaces such as R n .

Research paper thumbnail of Estimating finger grip force from an image of the hand using Convolutional Neural Networks and Gaussian processes

2014 IEEE International Conference on Robotics and Automation (ICRA), 2014

ABSTRACT Estimating human fingertip forces is required to understand force distribution in graspi... more ABSTRACT Estimating human fingertip forces is required to understand force distribution in grasping and manipulation. Human grasping behavior can then be used to develop force-and impedance-based grasping and manipulation strategies for robotic hands. However, estimating human grip force naturally is only possible with instrumented objects or unnatural gloves, thus greatly limiting the type of objects used. In this paper we describe an approach which uses images of the human fingertip to reconstruct grip force and torque at the finger. Our approach does not use finger-mounted equipment, but instead a steady camera observing the fingers of the hand from a distance. This allows for finger force estimation without any physical interference with the hand or object itself, and is therefore universally applicable. We construct a 3-dimensional finger model from 2D images. Convolutional Neural Networks (CNN) are used to predict the 2D image to a 3D model transformation matrix. Two methods of CNN are designed for separate and combined outputs of orientation and position. After learning, our system shows an alignment accuracy over 98% on unknown data. In the final step, a Gaussian process estimates finger force and torque from the aligned images based on color changes and deformations of the nail and its surrounding skin. Experimental results shows that the accuracy achieves about 95% in the force estimation and 90% in the torque.

Research paper thumbnail of Training Neural Networks with Implicit Variance

Lecture Notes in Computer Science, 2013

We present a novel method to train predictive Gaussian distributions p(z|x) for regression proble... more We present a novel method to train predictive Gaussian distributions p(z|x) for regression problems with neural networks. While most approaches either ignore or explicitly model the variance as another response variable, it is trained implicitly in our case. Establishing stochasticty by the injection of noise into the input and hidden units, the outputs are approximated with a Gaussian distribution by the forward propagation method introduced for fast dropout [1]. We have designed our method to respect that probabilistic interpretation of the output units in the loss function. The method is evaluated on a synthetic and a inverse robot dynamics task, yielding superior performance to plain neural networks, Gaussian processes and LWPR in terms of mean squared error and likelihood.

Research paper thumbnail of On fast dropout and its applicability to recurrent networks

Recurrent Neural Networks (RNNs) are rich models for the processing of sequential data. Recent wo... more Recurrent Neural Networks (RNNs) are rich models for the processing of sequential data. Recent work on advancing the state of the art has been focused on the optimization or modelling of RNNs, mostly motivated by adressing the problems of the vanishing and exploding gradients. The control of overfitting has seen considerably less attention. This paper contributes to that by analyzing fast dropout, a recent regularization method for generalized linear models and neural networks from a back-propagation inspired perspective. We show that fast dropout implements a quadratic form of an adaptive, per-parameter regularizer, which rewards large weights in the light of underfitting, penalizes them when overfitting and vanishes at minima of an unregularized training loss. The derivatives of that regularizer are exclusively based on the training error signal. One consequence of this is the absense of a global weight attractor, which is particularly appealing for RNNs, since the dynamics are not biased towards a certain regime. We positively test the hypothesis that this improves the performance of RNNs on four musical data sets and a natural language processing (NLP) task.

Research paper thumbnail of Estimating finger grip force from an image of the hand using Convolutional Neural Networks and Gaussian Processes

ABSTRACT Estimating human fingertip forces is required to understand force distribution in graspi... more ABSTRACT Estimating human fingertip forces is required to understand force distribution in grasping and manipulation. Human grasping behavior can then be used to develop force-and impedance-based grasping and manipulation strategies for robotic hands. However, estimating human grip force naturally is only possible with instrumented objects or unnatural gloves, thus greatly limiting the type of objects used. In this paper we describe an approach which uses images of the human fingertip to reconstruct grip force and torque at the finger. Our approach does not use finger-mounted equipment, but instead a steady camera observing the fingers of the hand from a distance. This allows for finger force estimation without any physical interference with the hand or object itself, and is therefore universally applicable. We construct a 3-dimensional finger model from 2D images. Convolutional Neural Networks (CNN) are used to predict the 2D image to a 3D model transformation matrix. Two methods of CNN are designed for separate and combined outputs of orientation and position. After learning, our system shows an alignment accuracy over 98% on unknown data. In the final step, a Gaussian process estimates finger force and torque from the aligned images based on color changes and deformations of the nail and its surrounding skin. Experimental results shows that the accuracy achieves about 95% in the force estimation and 90% in the torque.

Research paper thumbnail of The Grasp Perturbator: Calibrating human grasp stiffness during a graded force task

2011 IEEE International Conference on Robotics and Automation, 2011

In this paper we present a novel and simple handheld device for measuring in vivo human grasp imp... more In this paper we present a novel and simple handheld device for measuring in vivo human grasp impedance. The measurement method is based on a static identification method and intrinsic impedance is identified inbetween 25 ms. Using this device it is possbile to develop continuous grasp impedance measurement methods as it is an active research topic in physiology as well as in robotics, especially since nowadays (bio-inspired) robotics can be impedance-controlled. Potential applications of human impedance estimation range from impedance-controlled telesurgery to limb prosthetics and rehabilitation robotics. We validate the device through a physiological experiment in which the device is used to show a linear relationship between finger stiffness and grip force.

Research paper thumbnail of Two-Stream RNN/CNN for Action Recognition in 3D Videos

The recognition of actions from video sequences has many applications in health monitoring, assis... more The recognition of actions from video sequences has many applications in health monitoring, assisted living, surveillance, and smart homes. Despite advances in sensing, in particular related to 3D video, the methodologies to process the data are still subject to research. We demonstrate superior results by a system which combines recurrent neural networks with convolutional neural networks in a voting approach. The gated-recurrent-unit-based neural networks are particularly well-suited to distinguish actions based on long-term information from optical tracking data; the 3D-CNNs focus more on detailed, recent information from video data. The resulting features are merged in an SVM which then classifies the movement. In this architecture, our method improves recognition rates of state-of-the-art methods by 14% on standard data sets.

Research paper thumbnail of Deep Variational Bayes Filters: Unsupervised learning of state space models from raw data

We introduce Deep Variational Bayes Filters (DVBF), a new method for unsupervised learning and id... more We introduce Deep Variational Bayes Filters (DVBF), a new method for unsupervised learning and identification of latent Markovian state space models. Leveraging recent advances in Stochastic Gradient Variational Bayes, DVBF can overcome intractable inference distributions via variational inference. Thus, it can handle highly nonlinear input data with temporal and spatial dependencies such as image sequences without domain knowledge. Our experiments show that enabling back-propagation through transitions enforces state space assumptions and significantly improves information content of the latent embedding. This also enables realistic long-term prediction.

Research paper thumbnail of CNN-based Segmentation of Medical Imaging Data

Convolutional neural networks have been applied to a wide variety of computer vision tasks. Recen... more Convolutional neural networks have been applied to a wide variety of computer vision tasks. Recent advances in semantic segmentation have enabled their application to medical image segmentation. While most CNNs use two-dimensional kernels, recent CNN-based publications on medical image segmentation featured three-dimensional kernels, allowing full access to the three-dimensional structure of medical images. Though closely related to semantic segmentation, medical image segmentation includes specific challenges that need to be addressed, such as the scarcity of labelled data, the high class imbalance found in the ground truth and the high memory demand of three-dimensional images. In this work, a CNN-based method with three-dimensional filters is demonstrated and applied to hand and brain MRI. Two modifications to an existing CNN architecture are discussed, along with methods on addressing the aforementioned challenges. While most of the existing literature on medical image segmentation focuses on soft tissue and the major organs, this work is validated on data both from the central nervous system as well as the bones of the hand.

Research paper thumbnail of Evaluation of joint type modelling in the human hand

This short communication presents preliminary results from an extensive investigation of joint mo... more This short communication presents preliminary results from an extensive investigation of joint modelling for the human hand. We use finger and hand movement data recorded from both hands of 110 subjects using passive reflective markers on the skin. Furthermore, we use data which was recorded from a single Thiel-fixated cadaver hand using also passive reflective markers but fixed to the bone. Our data clearly demonstrate that, for wrist and finger joints, hinge joint models are suciently accurate to describe their movement in Cartesian space.

Research paper thumbnail of Measuring fingertip forces from camera images for random finger poses

2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015

Robust fingertip force detection from fingernail image is a critical strategy that can be applied... more Robust fingertip force detection from fingernail image is a critical strategy that can be applied in many areas. However, prior research fixed many variables that influence the finger color change. This paper analyzes the effect of the finger joint on the force detection in order to deal with the constrained finger position setting. A force estimator method is designed: a model to predict the fingertip force from finger joints measured from 2D cameras and 3 rectangular markers in cooperation with the fingernail images are trained. Then the error caused by the color changes of the joint bending can be avoided. This strategy is a significant step forward from a finger force estimator that requires tedious finger joint setting.

Research paper thumbnail of OsiSma2013

Research paper thumbnail of On fast dropout and its applicability to recurrent networks

Recurrent Neural Networks (RNNs) are rich models for the processing of sequential data. Recent wo... more Recurrent Neural Networks (RNNs) are rich models for the processing of sequential data. Recent work on advancing the state of the art has been focused on the optimization or modelling of RNNs, mostly motivated by adressing the problems of the vanishing and exploding gradients. The control of overfitting has seen considerably less attention. This paper contributes to that by analyzing fast dropout, a recent regularization method for generalized linear models and neural networks from a back-propagation inspired perspective. We show that fast dropout implements a quadratic form of an adaptive, per-parameter regularizer, which rewards large weights in the light of underfitting, penalizes them when overfitting and vanishes at minima of an unregularized training loss. The derivatives of that regularizer are exclusively based on the training error signal. One consequence of this is the absense of a global weight attractor, which is particularly appealing for RNNs, since the dynamics are not biased towards a certain regime. We positively test the hypothesis that this improves the performance of RNNs on four musical data sets and a natural language processing (NLP) task.

Research paper thumbnail of Computing grip force and torque from finger nail images using Gaussian processes

2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013

We demonstrate a simple approach with which finger force can be measured from nail coloration. By... more We demonstrate a simple approach with which finger force can be measured from nail coloration. By automatically extracting features from nail images of a finger-mounted CCD camera, we can directly relate these images to the force measured by a force-torque sensor. The method automatically corrects orientation and illumination di erences.

Research paper thumbnail of Estimating finger grip force from an image of the hand using Convolutional Neural Networks and Gaussian processes

2014 IEEE International Conference on Robotics and Automation (ICRA), 2014

ABSTRACT Estimating human fingertip forces is required to understand force distribution in graspi... more ABSTRACT Estimating human fingertip forces is required to understand force distribution in grasping and manipulation. Human grasping behavior can then be used to develop force-and impedance-based grasping and manipulation strategies for robotic hands. However, estimating human grip force naturally is only possible with instrumented objects or unnatural gloves, thus greatly limiting the type of objects used. In this paper we describe an approach which uses images of the human fingertip to reconstruct grip force and torque at the finger. Our approach does not use finger-mounted equipment, but instead a steady camera observing the fingers of the hand from a distance. This allows for finger force estimation without any physical interference with the hand or object itself, and is therefore universally applicable. We construct a 3-dimensional finger model from 2D images. Convolutional Neural Networks (CNN) are used to predict the 2D image to a 3D model transformation matrix. Two methods of CNN are designed for separate and combined outputs of orientation and position. After learning, our system shows an alignment accuracy over 98% on unknown data. In the final step, a Gaussian process estimates finger force and torque from the aligned images based on color changes and deformations of the nail and its surrounding skin. Experimental results shows that the accuracy achieves about 95% in the force estimation and 90% in the torque.

Research paper thumbnail of Convolutional Neural Networks Learn Compact Local Image Descriptors

Lecture Notes in Computer Science, 2013

A standard deep convolutional neural network paired with a suitable loss function learns compact ... more A standard deep convolutional neural network paired with a suitable loss function learns compact local image descriptors that perform comparably to state-of-the art approaches.

Research paper thumbnail of Training Neural Networks with Implicit Variance

Lecture Notes in Computer Science, 2013

We present a novel method to train predictive Gaussian distributions p(z|x) for regression proble... more We present a novel method to train predictive Gaussian distributions p(z|x) for regression problems with neural networks. While most approaches either ignore or explicitly model the variance as another response variable, it is trained implicitly in our case. Establishing stochasticty by the injection of noise into the input and hidden units, the outputs are approximated with a Gaussian distribution by the forward propagation method introduced for fast dropout [1]. We have designed our method to respect that probabilistic interpretation of the output units in the loss function. The method is evaluated on a synthetic and a inverse robot dynamics task, yielding superior performance to plain neural networks, Gaussian processes and LWPR in terms of mean squared error and likelihood.

Research paper thumbnail of Unsupervised Feature Learning for low-level Local Image Descriptors

Unsupervised feature learning has shown impressive results for a wide range of input modalities, ... more Unsupervised feature learning has shown impressive results for a wide range of input modalities, in particular for object classification tasks in computer vision. Using a large amount of unlabeled data, unsupervised feature learning methods are utilized to construct high-level representations that are discriminative enough for subsequently trained supervised classification algorithms. However, it has never been \emph{quantitatively} investigated yet how well unsupervised learning methods can find \emph{low-level representations} for image patches without any additional supervision. In this paper we examine the performance of pure unsupervised methods on a low-level correspondence task, a problem that is central to many Computer Vision applications. We find that a special type of Restricted Boltzmann Machines (RBMs) performs comparably to hand-crafted descriptors. Additionally, a simple binarization scheme produces compact representations that perform better than several state-of-the...

Research paper thumbnail of Continuous robot control using surface electromyography of atrophic muscles

2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013

The development of new, light robotic systems has opened up a wealth of human-robot interaction a... more The development of new, light robotic systems has opened up a wealth of human-robot interaction applications. In particular, the use of robot manipulators as personal assistant for the disabled is realistic and affordable, but still requires research as to the brain-computer interface.

Research paper thumbnail of Identification of Human Limb Stiffness in 5 DoF and Estimation via EMG

Springer Tracts in Advanced Robotics, 2013

To approach robustness and optimal performance, biological musculoskeletal systems can adapt thei... more To approach robustness and optimal performance, biological musculoskeletal systems can adapt their impedance while interacting with their environment. This property has motivated modern robotic designs including variable-impedance actuators and control methods, based on the capability to vary visco-elastic properties actively or passively. Even though variable-impedance actuation and impedance control in robotics is resolved to a great part, a general set of rules by which impedance is adjusted related to the task at hand is still lacking. This paper aims to fill this gap by providing a method to estimate the stiffness of the human arm in more than two degrees of freedom by perturbation. To overcome ill-conditionedness of the impedance and inertial matrices, we propose and validate methods to separately identify inertial and stiffness parameters. Finally, a model is proposed to estimate the joint stiffness from EMG-measurements of muscle activities.

Research paper thumbnail of Learning Sequence Neighbourhood Metrics

Lecture Notes in Computer Science, 2012

Recurrent neural networks (RNNs) in combination with a pooling operator and the neighbourhood com... more Recurrent neural networks (RNNs) in combination with a pooling operator and the neighbourhood components analysis (NCA) objective function are able to detect the characterizing dynamics of sequences and embed them into a fixed-length vector space of arbitrary dimensionality. Subsequently, the resulting features are meaningful and can be used for visualization or nearest neighbour classification in linear time. This kind of metric learning for sequential data enables the use of algorithms tailored towards fixed length vector spaces such as R n .

Research paper thumbnail of Estimating finger grip force from an image of the hand using Convolutional Neural Networks and Gaussian processes

2014 IEEE International Conference on Robotics and Automation (ICRA), 2014

ABSTRACT Estimating human fingertip forces is required to understand force distribution in graspi... more ABSTRACT Estimating human fingertip forces is required to understand force distribution in grasping and manipulation. Human grasping behavior can then be used to develop force-and impedance-based grasping and manipulation strategies for robotic hands. However, estimating human grip force naturally is only possible with instrumented objects or unnatural gloves, thus greatly limiting the type of objects used. In this paper we describe an approach which uses images of the human fingertip to reconstruct grip force and torque at the finger. Our approach does not use finger-mounted equipment, but instead a steady camera observing the fingers of the hand from a distance. This allows for finger force estimation without any physical interference with the hand or object itself, and is therefore universally applicable. We construct a 3-dimensional finger model from 2D images. Convolutional Neural Networks (CNN) are used to predict the 2D image to a 3D model transformation matrix. Two methods of CNN are designed for separate and combined outputs of orientation and position. After learning, our system shows an alignment accuracy over 98% on unknown data. In the final step, a Gaussian process estimates finger force and torque from the aligned images based on color changes and deformations of the nail and its surrounding skin. Experimental results shows that the accuracy achieves about 95% in the force estimation and 90% in the torque.

Research paper thumbnail of Training Neural Networks with Implicit Variance

Lecture Notes in Computer Science, 2013

We present a novel method to train predictive Gaussian distributions p(z|x) for regression proble... more We present a novel method to train predictive Gaussian distributions p(z|x) for regression problems with neural networks. While most approaches either ignore or explicitly model the variance as another response variable, it is trained implicitly in our case. Establishing stochasticty by the injection of noise into the input and hidden units, the outputs are approximated with a Gaussian distribution by the forward propagation method introduced for fast dropout [1]. We have designed our method to respect that probabilistic interpretation of the output units in the loss function. The method is evaluated on a synthetic and a inverse robot dynamics task, yielding superior performance to plain neural networks, Gaussian processes and LWPR in terms of mean squared error and likelihood.

Research paper thumbnail of On fast dropout and its applicability to recurrent networks

Recurrent Neural Networks (RNNs) are rich models for the processing of sequential data. Recent wo... more Recurrent Neural Networks (RNNs) are rich models for the processing of sequential data. Recent work on advancing the state of the art has been focused on the optimization or modelling of RNNs, mostly motivated by adressing the problems of the vanishing and exploding gradients. The control of overfitting has seen considerably less attention. This paper contributes to that by analyzing fast dropout, a recent regularization method for generalized linear models and neural networks from a back-propagation inspired perspective. We show that fast dropout implements a quadratic form of an adaptive, per-parameter regularizer, which rewards large weights in the light of underfitting, penalizes them when overfitting and vanishes at minima of an unregularized training loss. The derivatives of that regularizer are exclusively based on the training error signal. One consequence of this is the absense of a global weight attractor, which is particularly appealing for RNNs, since the dynamics are not biased towards a certain regime. We positively test the hypothesis that this improves the performance of RNNs on four musical data sets and a natural language processing (NLP) task.

Research paper thumbnail of Estimating finger grip force from an image of the hand using Convolutional Neural Networks and Gaussian Processes

ABSTRACT Estimating human fingertip forces is required to understand force distribution in graspi... more ABSTRACT Estimating human fingertip forces is required to understand force distribution in grasping and manipulation. Human grasping behavior can then be used to develop force-and impedance-based grasping and manipulation strategies for robotic hands. However, estimating human grip force naturally is only possible with instrumented objects or unnatural gloves, thus greatly limiting the type of objects used. In this paper we describe an approach which uses images of the human fingertip to reconstruct grip force and torque at the finger. Our approach does not use finger-mounted equipment, but instead a steady camera observing the fingers of the hand from a distance. This allows for finger force estimation without any physical interference with the hand or object itself, and is therefore universally applicable. We construct a 3-dimensional finger model from 2D images. Convolutional Neural Networks (CNN) are used to predict the 2D image to a 3D model transformation matrix. Two methods of CNN are designed for separate and combined outputs of orientation and position. After learning, our system shows an alignment accuracy over 98% on unknown data. In the final step, a Gaussian process estimates finger force and torque from the aligned images based on color changes and deformations of the nail and its surrounding skin. Experimental results shows that the accuracy achieves about 95% in the force estimation and 90% in the torque.

Research paper thumbnail of The Grasp Perturbator: Calibrating human grasp stiffness during a graded force task

2011 IEEE International Conference on Robotics and Automation, 2011

In this paper we present a novel and simple handheld device for measuring in vivo human grasp imp... more In this paper we present a novel and simple handheld device for measuring in vivo human grasp impedance. The measurement method is based on a static identification method and intrinsic impedance is identified inbetween 25 ms. Using this device it is possbile to develop continuous grasp impedance measurement methods as it is an active research topic in physiology as well as in robotics, especially since nowadays (bio-inspired) robotics can be impedance-controlled. Potential applications of human impedance estimation range from impedance-controlled telesurgery to limb prosthetics and rehabilitation robotics. We validate the device through a physiological experiment in which the device is used to show a linear relationship between finger stiffness and grip force.