ruofeng wei - Academia.edu (original) (raw)

Papers by ruofeng wei

Research paper thumbnail of Distilled Visual and Robot Kinematics Embeddings for Metric Depth Estimation in Monocular Scene Reconstruction

2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Estimating precise metric depth and scene reconstruction from monocular endoscopy is a fundamenta... more Estimating precise metric depth and scene reconstruction from monocular endoscopy is a fundamental task for surgical navigation in robotic surgery. However, traditional stereo matching adopts binocular images to perceive the depth information, which is difficult to transfer to the soft roboticsbased surgical systems due to the use of monocular endoscopy. In this paper, we present a novel framework that combines robot kinematics and monocular endoscope images with deep unsupervised learning into a single network for metric depth estimation and then achieve 3D reconstruction of complex anatomy. Specifically, we first obtain the relative depth maps of surgical scenes by leveraging a brightness-aware monocular depth estimation method. Then, the corresponding endoscope poses are computed based on non-linear optimization of geometric and photometric reprojection residuals. Afterwards, we develop a Depth-driven Sliding Optimization (DDSO) algorithm to extract the scaling coefficient from kinematics and calculated poses offline. By coupling the metric scale and relative depth data, we form a robust ensemble that represents the metric and consistent depth. Next, we treat the ensemble as supervisory labels to train a metric depth estimation network for surgeries (i.e., MetricDepthS-Net) that distills the embeddings from the robot kinematics, endoscopic videos, and poses. With accurate metric depth estimation, we utilize a dense visual reconstruction method to recover the 3D structure of the whole surgical site. We have extensively evaluated the proposed framework on public SCARED and achieved comparable performance with stereo-based depth estimation methods. Our results demonstrate the feasibility of the proposed approach to recover the metric depth and 3D structure with monocular inputs.

Research paper thumbnail of Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery

IEEE Transactions on Biomedical Engineering

The computation of anatomical information and laparoscope position is a fundamental block of robo... more The computation of anatomical information and laparoscope position is a fundamental block of robot-assisted surgical navigation in Minimally Invasive Surgery (MIS). Recovering a dense 3D structure of surgical scene using visual cues remains a challenge, and the online laparoscopic tracking mostly relies on external sensors, which increases system complexity. In this paper, we propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is hereby achieved. To reconstruct the 3D structure of the whole surgical environment, we first fine-tune a learning-based stereoscopic depth perception method, which is robust to the texture-less and variant soft tissues, for depth estimation. Then, we develop a dense visual reconstruction algorithm to represent the scene by surfels, estimate the laparoscope pose and fuse the depth data into a unified reference coordinate for tissue reconstruction. To estimate poses of new laparoscope views, we realize a coarse-to-fine localization method, which incorporates our reconstructed 3D model. We evaluate the reconstruction method and the localization module on three datasets, namely, the stereo correspondence and reconstruction of endoscopic data (SCARED), the ex-vivo phantom and tissue data collected with Universal Robot (UR) and Karl Storz Laparoscope, and the in-vivo DaVinci robotic surgery dataset. Extensive experiments have been conducted to prove the superior performance of our method in 3D anatomy reconstruction and laparoscopic localization, which demonstrates its potential implementation to surgical navigation system.

Research paper thumbnail of Research on Recognition Technology of Aluminum Profile Surface Defects Based on Deep Learning

Aluminum profile surface defects can greatly affect the performance, safety and reliability of pr... more Aluminum profile surface defects can greatly affect the performance, safety and reliability of products. Traditional human-based visual inspection is low accuracy and time consuming, and machine vision-based methods depend on hand-crafted features which need to be carefully designed and lack robustness. To recognize the multiple types of defects with various size on aluminum profiles, a multiscale defect detection network based on deep learning is proposed. Then, the network is trained and evaluated using aluminum profile surface defects images. Results show 84.6%, 48.5%, 96.9%, 97.9%, 96.9%, 42.5%, 47.2%, 100%, 100%, 43.3% average precision(AP) for the ten defect categories, respectively, with a mean AP of 75.8%, which illustrate the effectiveness of the network in aluminum profile surface defects detection. In addition, saliency maps also show the feasibility of the proposed network.

Research paper thumbnail of 3D Perception based Imitation Learning under Limited Demonstration for Laparoscope Control in Robotic Surgery

3D Perception based Imitation Learning under Limited Demonstration for Laparoscope Control in Robotic Surgery

2022 International Conference on Robotics and Automation (ICRA)

Research paper thumbnail of Control of a Flexible Continuum Manipulator for Laser Beam Steering

Control of a Flexible Continuum Manipulator for Laser Beam Steering

IEEE Robotics and Automation Letters

Continuum manipulators with structural compliance can be utilized to steer a laser beam in constr... more Continuum manipulators with structural compliance can be utilized to steer a laser beam in constrained environments. However, the flexibility and nonlinear characteristics of continuum manipulators bring difficulty in precision manipulation. This study proposes a model-free control approach with visual feedback to control a tendon-driven flexible manipulator that can be integrated into an endoscope to steer a laser beam accurately. To overcome the noise from the visual feedback and disturbances from environment during operation, a local Jacobian matrix that maps the actuation space to the image space is approximated online by using the sensing data stored in limited memory, where the influence of outliers can be filtered. Then, a second-order sliding mode controller is developed to achieve robust control of the flexible manipulator in laser beam steering. Simulations and experiments are performed to demonstrate the effectiveness of the proposed control methods in different configurations.

Research paper thumbnail of Distilled Visual and Robot Kinematics Embeddings for Metric Depth Estimation in Monocular Scene Reconstruction

2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Estimating precise metric depth and scene reconstruction from monocular endoscopy is a fundamenta... more Estimating precise metric depth and scene reconstruction from monocular endoscopy is a fundamental task for surgical navigation in robotic surgery. However, traditional stereo matching adopts binocular images to perceive the depth information, which is difficult to transfer to the soft roboticsbased surgical systems due to the use of monocular endoscopy. In this paper, we present a novel framework that combines robot kinematics and monocular endoscope images with deep unsupervised learning into a single network for metric depth estimation and then achieve 3D reconstruction of complex anatomy. Specifically, we first obtain the relative depth maps of surgical scenes by leveraging a brightness-aware monocular depth estimation method. Then, the corresponding endoscope poses are computed based on non-linear optimization of geometric and photometric reprojection residuals. Afterwards, we develop a Depth-driven Sliding Optimization (DDSO) algorithm to extract the scaling coefficient from kinematics and calculated poses offline. By coupling the metric scale and relative depth data, we form a robust ensemble that represents the metric and consistent depth. Next, we treat the ensemble as supervisory labels to train a metric depth estimation network for surgeries (i.e., MetricDepthS-Net) that distills the embeddings from the robot kinematics, endoscopic videos, and poses. With accurate metric depth estimation, we utilize a dense visual reconstruction method to recover the 3D structure of the whole surgical site. We have extensively evaluated the proposed framework on public SCARED and achieved comparable performance with stereo-based depth estimation methods. Our results demonstrate the feasibility of the proposed approach to recover the metric depth and 3D structure with monocular inputs.

Research paper thumbnail of Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery

IEEE Transactions on Biomedical Engineering

The computation of anatomical information and laparoscope position is a fundamental block of robo... more The computation of anatomical information and laparoscope position is a fundamental block of robot-assisted surgical navigation in Minimally Invasive Surgery (MIS). Recovering a dense 3D structure of surgical scene using visual cues remains a challenge, and the online laparoscopic tracking mostly relies on external sensors, which increases system complexity. In this paper, we propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is hereby achieved. To reconstruct the 3D structure of the whole surgical environment, we first fine-tune a learning-based stereoscopic depth perception method, which is robust to the texture-less and variant soft tissues, for depth estimation. Then, we develop a dense visual reconstruction algorithm to represent the scene by surfels, estimate the laparoscope pose and fuse the depth data into a unified reference coordinate for tissue reconstruction. To estimate poses of new laparoscope views, we realize a coarse-to-fine localization method, which incorporates our reconstructed 3D model. We evaluate the reconstruction method and the localization module on three datasets, namely, the stereo correspondence and reconstruction of endoscopic data (SCARED), the ex-vivo phantom and tissue data collected with Universal Robot (UR) and Karl Storz Laparoscope, and the in-vivo DaVinci robotic surgery dataset. Extensive experiments have been conducted to prove the superior performance of our method in 3D anatomy reconstruction and laparoscopic localization, which demonstrates its potential implementation to surgical navigation system.

Research paper thumbnail of Research on Recognition Technology of Aluminum Profile Surface Defects Based on Deep Learning

Aluminum profile surface defects can greatly affect the performance, safety and reliability of pr... more Aluminum profile surface defects can greatly affect the performance, safety and reliability of products. Traditional human-based visual inspection is low accuracy and time consuming, and machine vision-based methods depend on hand-crafted features which need to be carefully designed and lack robustness. To recognize the multiple types of defects with various size on aluminum profiles, a multiscale defect detection network based on deep learning is proposed. Then, the network is trained and evaluated using aluminum profile surface defects images. Results show 84.6%, 48.5%, 96.9%, 97.9%, 96.9%, 42.5%, 47.2%, 100%, 100%, 43.3% average precision(AP) for the ten defect categories, respectively, with a mean AP of 75.8%, which illustrate the effectiveness of the network in aluminum profile surface defects detection. In addition, saliency maps also show the feasibility of the proposed network.

Research paper thumbnail of 3D Perception based Imitation Learning under Limited Demonstration for Laparoscope Control in Robotic Surgery

3D Perception based Imitation Learning under Limited Demonstration for Laparoscope Control in Robotic Surgery

2022 International Conference on Robotics and Automation (ICRA)

Research paper thumbnail of Control of a Flexible Continuum Manipulator for Laser Beam Steering

Control of a Flexible Continuum Manipulator for Laser Beam Steering

IEEE Robotics and Automation Letters

Continuum manipulators with structural compliance can be utilized to steer a laser beam in constr... more Continuum manipulators with structural compliance can be utilized to steer a laser beam in constrained environments. However, the flexibility and nonlinear characteristics of continuum manipulators bring difficulty in precision manipulation. This study proposes a model-free control approach with visual feedback to control a tendon-driven flexible manipulator that can be integrated into an endoscope to steer a laser beam accurately. To overcome the noise from the visual feedback and disturbances from environment during operation, a local Jacobian matrix that maps the actuation space to the image space is approximated online by using the sensing data stored in limited memory, where the influence of outliers can be filtered. Then, a second-order sliding mode controller is developed to achieve robust control of the flexible manipulator in laser beam steering. Simulations and experiments are performed to demonstrate the effectiveness of the proposed control methods in different configurations.