Alexandre Krupa | Institut National de Recherche en Informatique et Automatique (INRIA) (original) (raw)
Papers by Alexandre Krupa
Ces dernieres annees, le traitement du signal et l’automatique ont revolutionne la medecine et la... more Ces dernieres annees, le traitement du signal et l’automatique ont revolutionne la medecine et la chirurgie...
Contributions à l’asservissement visuel échographique soutenue le 21 décembre 2012 devant le jury... more Contributions à l’asservissement visuel échographique soutenue le 21 décembre 2012 devant le jury composé de
This paper presents a new visual servoing method that is able to stabilize a moving area of soft ... more This paper presents a new visual servoing method that is able to stabilize a moving area of soft tissue within an ultrasound B-mode imaging plane. The approach consists of moving the probe in order to minimize the relative position between a target imaging plane and the ultrasound plane observed by the probe of the moving tissue target. The problem is decoupled into motion out-of-plane and motion within plane. For the former, a new original method based on the speckle information contained in the images is developed. For the latter, an image region tracker is used to provide the in-plane motion. A visual servoing control scheme is then developed to perform the tracking robotic task. The method is validated on simulated motions of a probe on a static ultrasound volume acquired from a phantom.
Cet article presente une methode optimale de contour actif parametrique base sur les descripteurs... more Cet article presente une methode optimale de contour actif parametrique base sur les descripteurs de Fourier qui permet de detecter et de suivre en temps reel un contour dans une sequence d'images. L'avantage de cette approche reside dans sa capacite a traiter non seulement les contours de formes convexes, mais egalement les contours concaves. Cette methode permet egalement de re-initialiser automatiquement le contour actif lorsque des changements de topologie sont detectes au cours du suivi.
Twoand three-dimensional ultrasound imaging devices are widely used in medicine for non-invasive ... more Twoand three-dimensional ultrasound imaging devices are widely used in medicine for non-invasive observations of inner organs. In this context, robotic systems have been developed to automatically compensate for patient movements in order to keep the observed ultrasound image as stable as possible. This is typically achieved by continuously adjusting the position of an ultrasound probe such that the error between a currently observed set of features s ∈ R and a desired set of features s∗ ∈ R
2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), Oct 1, 2019
2020 International Conference on Robotics and Automation (ICRA), 2020
In this paper, we propose a novel method to simultaneously track the deformation of soft objects ... more In this paper, we propose a novel method to simultaneously track the deformation of soft objects and estimate their elasticity parameters. The tracking of the deformable object is performed by combining the visual information acquired with a RGB-D sensor with interactive Finite Element Method simulations of the object. The visual information is more particularly used to distort the simulated object. In parallel, the elasticity parameter estimation minimizes the error between the tracked object and a simulated object deformed by the forces that are measured using a force sensor. Once the elasticity parameters are estimated, our tracking algorithm can then also be used to estimate the deformation forces applied on an object without the use of a force sensor. We validated our method on several soft objects with di erent shape complexities. Our evaluations show the ability of our method to estimate the elasticity parameters as well as its use to estimate the forces applied on a deformab...
IEEE Robotics and Automation Letters
Needle insertion procedures under ultrasound guidance are commonly used for diagnosis and therapy... more Needle insertion procedures under ultrasound guidance are commonly used for diagnosis and therapy. It is often critical to accurately reach a targeted region, and this can be difficult to achieve due to intra-operative tissue motion. In this paper, we present a method to steer a beveledtip flexible needle towards a target embedded in moving tissue. Needle steering is performed using a needle insertion device attached to a robot arm. Closed-loop 3D steering of the needle is achieved using tracking of an artificial target in 2D ultrasound images and tracking of the needle tip position and orientation with an electromagnetic tracker. Tissue motion compensation is performed using force feedback to reduce targeting error and forces applied to the tissue. The method uses a mechanics-based interaction model that is updated online. A novel control law using task functions is proposed to fuse motion compensation, steering via base manipulation and tip-based steering. Validation of the tracking and steering algorithms are performed in gelatin phantom and bovine liver. Tissue motion up to 15mm is applied and average targeting error is 1.2±0.8mm and 2.5±0.7mm in gelatin and liver, respectively, which is sufficiently accurate for commonly performed needle insertion procedures.
IEEE Transactions on Robotics
The International Journal of Robotics Research
A visual servoing scheme consists of a closed-loop control approach that uses visual information ... more A visual servoing scheme consists of a closed-loop control approach that uses visual information feedback to control the motion of a robotic system. Probably the most popular visual servoing method is image-based visual servoing (IBVS). This kind of method uses geometric visual features extracted from the image to design the control law. However, extracting, matching, and tracking geometric visual features over time significantly limits the versatility of visual servoing controllers in various industrial and medical applications, in particular for “low-structured” medical images, e.g. ultrasounds and optical coherence tomography modalities. To overcome the limits of conventional IBVS, one can consider novel visual servoing paradigms known as “ direct” or “ featureless” approaches. This paper deals with the development of a new generation of direct visual servoing methods in which the signal control inputs are the coefficients of a multiscale image representation. In particular, we c...
This work deals with the use of speckle information contained in 2D B-mode ultrasound images to c... more This work deals with the use of speckle information contained in 2D B-mode ultrasound images to control the displacement of a robotized ultrasound probe with a visual servoing control scheme. The objective is to automatically synchronize the displacement of a conventional 2D ultrasound probe held by a medical robot in such a way to stabilize the image of a moving soft tissue. Such a workable method operating on 2D ultrasound images could be exploited in a host of clinical applications. For example, in diagnostic ultrasound imaging, one could automatically move the ultrasound probe to maintain the optimal view of moving soft tissue targets; or in biopsies and localized therapy procedures, one could synchronize the insertion of needles or other surgical tools into a moving target observed in live ultrasound.
Palpation process using elastography has been studied for malign tissue localization. This techni... more Palpation process using elastography has been studied for malign tissue localization. This technique has been applied in ultrasound and magnetic resonance imaging modalities. However, the ultrasound features make it suitable for roboticassisted systems in real-time [1], [2], [3]. The basic principle of ultrasound elastography is based on quasi-static mechanic compression of a target tissue with an ultrasound probe to obtain its elastic parameters. This parameters are generally imaged in an elastogram for visual purposes. The procedure to obtain an elastogram in hospitals is done by hand. Nevertheless, drawbacks like abrupt changes in force compression and motion stability of the hand affect the elastogram quality an sometimes ending in a wrong diagnosis. The dexterity of a robot can tackle these issues using a stable compression motion of the ultrasound probe. Very few investigations have been undertaken regarding the use of ultrasound elastography in robot-assisted procedures. Thes...
In minimally-invasive procedures like biopsy, the physician has to insert a needle into the tissu... more In minimally-invasive procedures like biopsy, the physician has to insert a needle into the tissues of a patient to reach a target. Currently, this task is mostly performed manually. This can result in a large nal positioning error of the tip that might lead to misdiagnosis and inadequate treatment. In this paper, we present 5 haptic guides to assist the clinical gesture of needle insertion. Those guides were evaluated through a preliminary user study involving two physicians, both experts in needle manipulation.
Collaboration with the food processing industry The iProcess project has been designed to address... more Collaboration with the food processing industry The iProcess project has been designed to address relevant Norwegian food industry needs in the fields of robotic automation, process analytical technology, information flow management and new business models. In order to map industry needs and to identify relevant case studies for use in developing the methodologies being researched by iProcess, the researchers have visited the industrial partners and their facilities. Discussions that took place during these visits, combined with annual project meetings, '' The number of animal hides we deal with has increased by 50%, thus boosting our requirement for better traceability. Tufan Yurt (Norilia) Industry involvement is key to ensuring the commercial impact of the research conducted as part of the iProcess project. The food processing industry has been involved in the selection of the relevant case studies around which research activities have been focused. Since these studies have involved a high degree of complexity, and in order to open the door to radical and innovative approaches, iProcess has focused on fundamental technology-oriented research with the aim of bridging the gap between current state-of-the-art and future industry needs. The nature of this approach has required regular and close collaboration with the food and vendor industry.
Cet article presente les avancees recentes dans le domaine de l'asservissement visuel par ima... more Cet article presente les avancees recentes dans le domaine de l'asservissement visuel par imagerie medicale, realisees notamment depuis les dernieres JNRR'07. Le domaine applicatif concerne la robotique medicale ou la tâche est realisee selon un schema de commande en « boucle fermee » utilisant directement l'information visuelle fournie par une modalite d'imagerie medicale telles l'endoscopie, la fibroscopie, l'echographie, la tomodensitometrie (scanner X), la resonance magnetique (IRM) ou autres.
This paper deals with the development of an Optical Coherence Tomography (OCT) based visual servo... more This paper deals with the development of an Optical Coherence Tomography (OCT) based visual servoing. The proposed control law uses the wavelet coefficients of the OCT images as the signal control inputs instead of the conventional geometric visual features (points, lines, moments, etc.). An important contribution is the determination of the interaction matrix that links the variation of the wavelet coefficients to the OCT probe (respectively to the robotic platform) spatial velocity. This interaction matrix, required in the visual control law, is obtained from time-derivation of the wavelet coefficients. This work is carried out in a medical context which consists of automatically moving a biological sample in such a way to go back to the position of a sample region that corresponds to a previous optical biopsy (OCT image). For instance, the proposed methodology makes it possible to follow accurately the progress of a pathological tissue between an optical biopsy and a former one. ...
Three dimensional ultrasound (3DUS) guided robotic beating heart surgery is an emerging computer-... more Three dimensional ultrasound (3DUS) guided robotic beating heart surgery is an emerging computer-assisted surgical technique. Motion tracking of both the robot and the heart tissue is one of the key open problems in these procedures and the use of 3DUS imaging offers the possibility to extract, intra-operatively and in real time, the motion of one or more targets. In this paper, we present a visual servoing approach to track the motion of a single target that can consist of either the surgical robot or cardiac tissues. To deal with the low quality of the 3DUS volumes, the proposed intensity-based approach requires no primitive extraction or image segmentation. This approach does not involve time consuming processing steps and can be applied to a wide range of tissue types and medical instruments. Unlike previous work [1] where the motion compensation task was realized physically by moving the probe attached to a robotic arm to compensate for abdominal organ motion, we propose here t...
IEEE Robotics and Automation Letters
We present a haptic teleoperation system capable of steering flexible needles under ultrasound im... more We present a haptic teleoperation system capable of steering flexible needles under ultrasound imaging toward a target. With respect to similar works, this approach enables intuitive control of the needle motion while providing the user with 3D navigation and needle tip cutting force using a combination of kinesthetic and vibrotactile haptic feedback. The needle is tracked during the insertion using a 3D ultrasound probe. A friction estimation algorithm extracts salient information about the cutting force at the needle tip from a force sensor placed at the needle base. A grounded haptic interface enables natural 6-DoF control of the needle motion while providing kinesthetic feedback, and a wearable cutaneous interface on the forearm provides distributed vibrotactile sensations. We carried out a human subject study to validate the insertion system in a gelatine phantom and compare seven different feedback techniques. The best performance was registered when providing navigation cues through kinesthetic feedback and needle tip cutting force through cutaneous vibrotactile feedback. In this modality, results showed an 87% accuracy improvement with respect to providing no haptic feedback at all.
2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Sep 1, 2017
This paper presents a robot-assisted system to obtain elastic information of a moving tissue usin... more This paper presents a robot-assisted system to obtain elastic information of a moving tissue using a 2-D ultrasound probe actuated by a 6 degrees of freedom robotic arm. The proposed method combines ultrasound image-based visual servoing, force control and non-rigid motion estimation. We present how the motion estimation, while the force control and visual sevoing are enabled, is useful to compute the strain map of the tissue. Ex-vivo experiments performed on a moving abdominal phantom demonstrate the efficiency and robustness of this methodology to compute the strain information of a tissue in motion.
Ces dernieres annees, le traitement du signal et l’automatique ont revolutionne la medecine et la... more Ces dernieres annees, le traitement du signal et l’automatique ont revolutionne la medecine et la chirurgie...
Contributions à l’asservissement visuel échographique soutenue le 21 décembre 2012 devant le jury... more Contributions à l’asservissement visuel échographique soutenue le 21 décembre 2012 devant le jury composé de
This paper presents a new visual servoing method that is able to stabilize a moving area of soft ... more This paper presents a new visual servoing method that is able to stabilize a moving area of soft tissue within an ultrasound B-mode imaging plane. The approach consists of moving the probe in order to minimize the relative position between a target imaging plane and the ultrasound plane observed by the probe of the moving tissue target. The problem is decoupled into motion out-of-plane and motion within plane. For the former, a new original method based on the speckle information contained in the images is developed. For the latter, an image region tracker is used to provide the in-plane motion. A visual servoing control scheme is then developed to perform the tracking robotic task. The method is validated on simulated motions of a probe on a static ultrasound volume acquired from a phantom.
Cet article presente une methode optimale de contour actif parametrique base sur les descripteurs... more Cet article presente une methode optimale de contour actif parametrique base sur les descripteurs de Fourier qui permet de detecter et de suivre en temps reel un contour dans une sequence d'images. L'avantage de cette approche reside dans sa capacite a traiter non seulement les contours de formes convexes, mais egalement les contours concaves. Cette methode permet egalement de re-initialiser automatiquement le contour actif lorsque des changements de topologie sont detectes au cours du suivi.
Twoand three-dimensional ultrasound imaging devices are widely used in medicine for non-invasive ... more Twoand three-dimensional ultrasound imaging devices are widely used in medicine for non-invasive observations of inner organs. In this context, robotic systems have been developed to automatically compensate for patient movements in order to keep the observed ultrasound image as stable as possible. This is typically achieved by continuously adjusting the position of an ultrasound probe such that the error between a currently observed set of features s ∈ R and a desired set of features s∗ ∈ R
2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), Oct 1, 2019
2020 International Conference on Robotics and Automation (ICRA), 2020
In this paper, we propose a novel method to simultaneously track the deformation of soft objects ... more In this paper, we propose a novel method to simultaneously track the deformation of soft objects and estimate their elasticity parameters. The tracking of the deformable object is performed by combining the visual information acquired with a RGB-D sensor with interactive Finite Element Method simulations of the object. The visual information is more particularly used to distort the simulated object. In parallel, the elasticity parameter estimation minimizes the error between the tracked object and a simulated object deformed by the forces that are measured using a force sensor. Once the elasticity parameters are estimated, our tracking algorithm can then also be used to estimate the deformation forces applied on an object without the use of a force sensor. We validated our method on several soft objects with di erent shape complexities. Our evaluations show the ability of our method to estimate the elasticity parameters as well as its use to estimate the forces applied on a deformab...
IEEE Robotics and Automation Letters
Needle insertion procedures under ultrasound guidance are commonly used for diagnosis and therapy... more Needle insertion procedures under ultrasound guidance are commonly used for diagnosis and therapy. It is often critical to accurately reach a targeted region, and this can be difficult to achieve due to intra-operative tissue motion. In this paper, we present a method to steer a beveledtip flexible needle towards a target embedded in moving tissue. Needle steering is performed using a needle insertion device attached to a robot arm. Closed-loop 3D steering of the needle is achieved using tracking of an artificial target in 2D ultrasound images and tracking of the needle tip position and orientation with an electromagnetic tracker. Tissue motion compensation is performed using force feedback to reduce targeting error and forces applied to the tissue. The method uses a mechanics-based interaction model that is updated online. A novel control law using task functions is proposed to fuse motion compensation, steering via base manipulation and tip-based steering. Validation of the tracking and steering algorithms are performed in gelatin phantom and bovine liver. Tissue motion up to 15mm is applied and average targeting error is 1.2±0.8mm and 2.5±0.7mm in gelatin and liver, respectively, which is sufficiently accurate for commonly performed needle insertion procedures.
IEEE Transactions on Robotics
The International Journal of Robotics Research
A visual servoing scheme consists of a closed-loop control approach that uses visual information ... more A visual servoing scheme consists of a closed-loop control approach that uses visual information feedback to control the motion of a robotic system. Probably the most popular visual servoing method is image-based visual servoing (IBVS). This kind of method uses geometric visual features extracted from the image to design the control law. However, extracting, matching, and tracking geometric visual features over time significantly limits the versatility of visual servoing controllers in various industrial and medical applications, in particular for “low-structured” medical images, e.g. ultrasounds and optical coherence tomography modalities. To overcome the limits of conventional IBVS, one can consider novel visual servoing paradigms known as “ direct” or “ featureless” approaches. This paper deals with the development of a new generation of direct visual servoing methods in which the signal control inputs are the coefficients of a multiscale image representation. In particular, we c...
This work deals with the use of speckle information contained in 2D B-mode ultrasound images to c... more This work deals with the use of speckle information contained in 2D B-mode ultrasound images to control the displacement of a robotized ultrasound probe with a visual servoing control scheme. The objective is to automatically synchronize the displacement of a conventional 2D ultrasound probe held by a medical robot in such a way to stabilize the image of a moving soft tissue. Such a workable method operating on 2D ultrasound images could be exploited in a host of clinical applications. For example, in diagnostic ultrasound imaging, one could automatically move the ultrasound probe to maintain the optimal view of moving soft tissue targets; or in biopsies and localized therapy procedures, one could synchronize the insertion of needles or other surgical tools into a moving target observed in live ultrasound.
Palpation process using elastography has been studied for malign tissue localization. This techni... more Palpation process using elastography has been studied for malign tissue localization. This technique has been applied in ultrasound and magnetic resonance imaging modalities. However, the ultrasound features make it suitable for roboticassisted systems in real-time [1], [2], [3]. The basic principle of ultrasound elastography is based on quasi-static mechanic compression of a target tissue with an ultrasound probe to obtain its elastic parameters. This parameters are generally imaged in an elastogram for visual purposes. The procedure to obtain an elastogram in hospitals is done by hand. Nevertheless, drawbacks like abrupt changes in force compression and motion stability of the hand affect the elastogram quality an sometimes ending in a wrong diagnosis. The dexterity of a robot can tackle these issues using a stable compression motion of the ultrasound probe. Very few investigations have been undertaken regarding the use of ultrasound elastography in robot-assisted procedures. Thes...
In minimally-invasive procedures like biopsy, the physician has to insert a needle into the tissu... more In minimally-invasive procedures like biopsy, the physician has to insert a needle into the tissues of a patient to reach a target. Currently, this task is mostly performed manually. This can result in a large nal positioning error of the tip that might lead to misdiagnosis and inadequate treatment. In this paper, we present 5 haptic guides to assist the clinical gesture of needle insertion. Those guides were evaluated through a preliminary user study involving two physicians, both experts in needle manipulation.
Collaboration with the food processing industry The iProcess project has been designed to address... more Collaboration with the food processing industry The iProcess project has been designed to address relevant Norwegian food industry needs in the fields of robotic automation, process analytical technology, information flow management and new business models. In order to map industry needs and to identify relevant case studies for use in developing the methodologies being researched by iProcess, the researchers have visited the industrial partners and their facilities. Discussions that took place during these visits, combined with annual project meetings, '' The number of animal hides we deal with has increased by 50%, thus boosting our requirement for better traceability. Tufan Yurt (Norilia) Industry involvement is key to ensuring the commercial impact of the research conducted as part of the iProcess project. The food processing industry has been involved in the selection of the relevant case studies around which research activities have been focused. Since these studies have involved a high degree of complexity, and in order to open the door to radical and innovative approaches, iProcess has focused on fundamental technology-oriented research with the aim of bridging the gap between current state-of-the-art and future industry needs. The nature of this approach has required regular and close collaboration with the food and vendor industry.
Cet article presente les avancees recentes dans le domaine de l'asservissement visuel par ima... more Cet article presente les avancees recentes dans le domaine de l'asservissement visuel par imagerie medicale, realisees notamment depuis les dernieres JNRR'07. Le domaine applicatif concerne la robotique medicale ou la tâche est realisee selon un schema de commande en « boucle fermee » utilisant directement l'information visuelle fournie par une modalite d'imagerie medicale telles l'endoscopie, la fibroscopie, l'echographie, la tomodensitometrie (scanner X), la resonance magnetique (IRM) ou autres.
This paper deals with the development of an Optical Coherence Tomography (OCT) based visual servo... more This paper deals with the development of an Optical Coherence Tomography (OCT) based visual servoing. The proposed control law uses the wavelet coefficients of the OCT images as the signal control inputs instead of the conventional geometric visual features (points, lines, moments, etc.). An important contribution is the determination of the interaction matrix that links the variation of the wavelet coefficients to the OCT probe (respectively to the robotic platform) spatial velocity. This interaction matrix, required in the visual control law, is obtained from time-derivation of the wavelet coefficients. This work is carried out in a medical context which consists of automatically moving a biological sample in such a way to go back to the position of a sample region that corresponds to a previous optical biopsy (OCT image). For instance, the proposed methodology makes it possible to follow accurately the progress of a pathological tissue between an optical biopsy and a former one. ...
Three dimensional ultrasound (3DUS) guided robotic beating heart surgery is an emerging computer-... more Three dimensional ultrasound (3DUS) guided robotic beating heart surgery is an emerging computer-assisted surgical technique. Motion tracking of both the robot and the heart tissue is one of the key open problems in these procedures and the use of 3DUS imaging offers the possibility to extract, intra-operatively and in real time, the motion of one or more targets. In this paper, we present a visual servoing approach to track the motion of a single target that can consist of either the surgical robot or cardiac tissues. To deal with the low quality of the 3DUS volumes, the proposed intensity-based approach requires no primitive extraction or image segmentation. This approach does not involve time consuming processing steps and can be applied to a wide range of tissue types and medical instruments. Unlike previous work [1] where the motion compensation task was realized physically by moving the probe attached to a robotic arm to compensate for abdominal organ motion, we propose here t...
IEEE Robotics and Automation Letters
We present a haptic teleoperation system capable of steering flexible needles under ultrasound im... more We present a haptic teleoperation system capable of steering flexible needles under ultrasound imaging toward a target. With respect to similar works, this approach enables intuitive control of the needle motion while providing the user with 3D navigation and needle tip cutting force using a combination of kinesthetic and vibrotactile haptic feedback. The needle is tracked during the insertion using a 3D ultrasound probe. A friction estimation algorithm extracts salient information about the cutting force at the needle tip from a force sensor placed at the needle base. A grounded haptic interface enables natural 6-DoF control of the needle motion while providing kinesthetic feedback, and a wearable cutaneous interface on the forearm provides distributed vibrotactile sensations. We carried out a human subject study to validate the insertion system in a gelatine phantom and compare seven different feedback techniques. The best performance was registered when providing navigation cues through kinesthetic feedback and needle tip cutting force through cutaneous vibrotactile feedback. In this modality, results showed an 87% accuracy improvement with respect to providing no haptic feedback at all.
2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Sep 1, 2017
This paper presents a robot-assisted system to obtain elastic information of a moving tissue usin... more This paper presents a robot-assisted system to obtain elastic information of a moving tissue using a 2-D ultrasound probe actuated by a 6 degrees of freedom robotic arm. The proposed method combines ultrasound image-based visual servoing, force control and non-rigid motion estimation. We present how the motion estimation, while the force control and visual sevoing are enabled, is useful to compute the strain map of the tissue. Ex-vivo experiments performed on a moving abdominal phantom demonstrate the efficiency and robustness of this methodology to compute the strain information of a tissue in motion.