Jing Dao Chen - Academia.edu (original) (raw)

Uploads

Papers by Jing Dao Chen

Research paper thumbnail of LRGNet: Learnable Region Growing for Class-Agnostic Point Cloud Segmentation

Robotics and Automation Letters, 2021

3D point cloud segmentation is an important function that helps robots understand the layout of t... more 3D point cloud segmentation is an important function that helps robots understand the layout of their surrounding environment and perform tasks such as grasping objects, avoiding obstacles, and finding landmarks. Current segmentation methods are mostly class-specific, many of which are tuned to work with specific object categories and may not be generalizable to different types of scenes. This research proposes a learnable region growing method for class-agnostic point cloud segmentation, specifically for the task of instance label prediction. The proposed method is able to segment any class of objects using a single deep neural network without any assumptions about their shapes and sizes. The deep neural network is trained to predict how to add or remove points from a point cloud region to morph it into incrementally more complete regions of an object instance. Segmentation results on the S3DIS and ScanNet datasets show that the proposed method outperforms competing methods by 1%-9% on 6 different evaluation metrics.

Research paper thumbnail of Dynamic Crane Workspace Update for Collision Avoidance during Blind Lift Operations

Proceedings of the 18th International Conference on Computing in Civil and Building Engineering, 2020

Cranes are associated with high levels of accidents and struck-by fatalities in many industrial s... more Cranes are associated with high levels of accidents and struck-by fatalities in many industrial settings. This is due to the congested nature of crane operation sites and the high cognitive load for the crane operator to track the load position throughout the lifting operation, especially during blind lifts. Current research on crane safety assistance systems focuses on preventing collisions with static entities such as existing building structures but do not adequately handle dynamic entities such as workers and movable containers. This research proposes a dynamic crane workspace updating method for collision avoidance during blind lift operations. The position and orientation of the crane load, as well as the position and orientation of surrounding obstacles, are automatically tracked and updated in a 3D crane workspace model during lifting operations. The load base position is first estimated using forward kinematics obtained from an encoder system. Then the load swing and load rotation are corrected for using a vision-based load detection algorithm. A static map of surrounding obstacles is initially obtained using 3D laser scanning. The map is then updated over the course of a lifting operation through semi-automated obstacle placement. In addition, a vision-based worker detection algorithm is used to track the position of workers in proximity to the load. Finally, the crane workspace is updated in real-time for the crane operator through a 3D user interface and warnings are issued whenever a potential collision scenario is detected. The proposed method was evaluated at a crane yard where multiple blind lift scenarios were carried out. Experimental results indicate that the proposed method can potentially improve the situational awareness of the crane operator for blind lift operations.

Research paper thumbnail of User Exemplar-based Building Element Retrieval from Raw Point Clouds using Deep Point-level Features

Automation in Construction, 2020

3D point cloud data can be utilized for site inspection and reverse engineering of building model... more 3D point cloud data can be utilized for site inspection and reverse engineering of building models. However, conventional methods for building element retrieval require a database of 3D CAD or BIM models which are unsuitable for the case of historical buildings without as-planned models or temporary structures that are not in the pre-built model. Thus, this paper proposes a semi-automated method to efficiently retrieve duplicate building elements without these constraints. First, the point cloud is processed with a pre-trained deep feature extractor to generate a 50-dimensional feature vector for each point. Next, the point cloud is segmented through feature clustering and region-growing algorithms, then displayed on a user interface for selection. Lastly, the selected exemplar is provided as input to a peak-finding algorithm to determine positive matches. Experimental results on five different datasets show that the proposed method obtains average rates above 90% for precision and recall.

Research paper thumbnail of Deep Learning Approach to Point Cloud Scene Understanding for Automated Scan to 3D Reconstruction

Journal of Computing in Civil Engineering, 2019

Construction progress estimation is an essential component of the daily construction cycle to ens... more Construction progress estimation is an essential component of the daily construction cycle to ensure high productivity and quality. However, using laser-scanned point clouds for the purpose of measuring the deviation between the as-built structures and the as-planned Building Information Model (BIM) remains cumbersome due to the difficulty in data registration, segmentation, annotation and modeling in large-scale point clouds. This research proposes the use of a data-driven deep learning framework to automatically detect and classify building elements from a laser-scanned point cloud scene. The point cloud is first converted into a graph representation, where vertices represent points and edges represent connections between points within a fixed distance. An edge-based classifier is used to discard edges connecting points from different objects and form connected components from points in the same object. Next, a point-based object classifier is used to determine the type of building component based on the 1

Research paper thumbnail of Multi-view Incremental Segmentation of 3D Point Clouds for Mobile Robots

Robotics and Automation Letters, 2019

Mobile robots need to create high-definition 3D maps of the environment for applications such as ... more Mobile robots need to create high-definition 3D maps of the environment for applications such as remote surveillance and infrastructure mapping. Accurate semantic processing of the acquired 3D point cloud is critical for allowing the robot to obtain a high-level understanding of the surrounding objects and perform context-aware decision making. Existing techniques for point cloud semantic segmentation are mostly applied on a single-frame or offline basis, with no way to integrate the segmentation results over time. This paper proposes an online method for mobile robots to incrementally build a semantically-rich 3D point cloud of the environment. The proposed deep neural network, MCPNet, is trained to predict class labels and object instance labels for each point in the scanned point cloud in an incremental fashion. A multi-view context pooling (MCP) operator is used to combine point features obtained from multiple viewpoints to improve the classification accuracy. The proposed architecture was trained and evaluated on ray-traced scans derived from the Stanford 3D Indoor Spaces dataset. Results show that the proposed approach led to 15% improvement in point-wise accuracy and 7% improvement in NMI compared to the next best online method, with only a 6% drop in accuracy compared to the PointNet-based offline approach.

Research paper thumbnail of Enhancing Blind Lift Safety on Offshore Platforms through Real-time Sensing and Visualization

CONVR, 2018

Crane lift operations are human-centric tasks where lift safety heavily relies on the operator's ... more Crane lift operations are human-centric tasks where lift safety heavily relies on the operator's awareness of potential spatial constraints and associated safety risks. Blind lifts on congested offshore platform (OP) environments, therefore, are inherently dangerous because of substantial presence of spatial conflicts in the crane workspace and the operator's limited visibility to the load. This research aims to improve operators' spatial awareness in blind lifts in the OP environment through real-time crane states sensing and visualization. We propose a technical framework that consists of two sensing modules (i.e., crane motion monitoring and load sway monitoring) and a visualization module. An inertial measurement unit (IMU)-based approach and a computer vision (CV) based approach are introduced to track load position for load sway monitoring. A prototype system was built and tested in an offshore platform to validate the crane motion monitoring and visualization modules. A lab experiment was conducted to evaluate the CV-based load sway monitoring method. Results from the field and lab tests indicate the proposed framework and methods were able to continuously monitor and visualize the crane states in real-time and thus provide the operator adequate assistance to identify and mitigate unsafe conditions during blind lifts.

Research paper thumbnail of SLAM-driven Intelligent Autonomous Mobile Robot Navigation for Construction Applications

The demand for construction site automation with mobile robots is increasing due to its advantage... more The demand for construction site automation with mobile robots is increasing due to its advantages in potential cost-saving, productivity, and safety. To be realistically deployed in construction sites, mobile robots must be capable of navigating in unstructured and cluttered environments. Furthermore, mobile robots should recognize both static and dynamic obstacles to determine drivable paths. However, existing robot navigation methods are not suitable for construction applications due to the challenging environmental conditions in construction sites. This study introduces an autonomous as-is 3D spatial data collection and perception method for mobile robots specifically aimed for construction job sites with many spatial uncertainties. The proposed Simultaneous Localization and Mapping (SLAM)-based navigation and object recognition methods were implemented and tested with a custom-designed mobile robot platform, Ground Robot of Mapping Infrastructure (GRoMI), which uses multiple laser scanners and a camera to sense and build a 3D environment map. Since SLAM, it self, did not detect uneven surface conditions and spatiotemporal objects on the ground, an obstacle detection algorithm was developed to recognize and avoid obstacles and the highly uneven terrain in real time. Given the 3D real-time scan map generated by 3D laser scanners, a path-finding algorithm was developed for autonomous navigation in an unknown environment with obstacles. Overall, the 3D color-mapped point clouds of construction sites generated by GRoMI were of sufficient quality to be used for many construction management applications such as construction progress monitoring, safety hazard identification, and defect detection.

Research paper thumbnail of Point-to-point Comparison Method for Automated Scan-vs-BIM Deviation Detection

Chen, J. and Cho, Y. (2018). "Point-to-point Comparison Method for Automated Scan-vs-BIM Deviatio... more Chen, J. and Cho, Y. (2018). "Point-to-point Comparison Method for Automated Scan-vs-BIM Deviation Detection. Abstract: Laser scanning is one of the most accurate methods of measuring the geometric accuracy of the as-built condition of a construction site. However, using laser-scanned point clouds for the purpose of measuring the deviation between the as-built structures and the as-planned Building Information Model (BIM) remains cumbersome due to the difficulty in registration, segmentation, and matching of large-scale point clouds. Conventional methods for automated deviation detection are computationally intensive and only work for regular geometric shapes such as planes and cylinders. This research proposes a point-to-point comparison method for automated scan-vs-BIM deviation detection. First, a laser-scanned point cloud is collected and imported into a BIM file. A column-detection routine is used to locate the centroid of each column in the x-y plane for both the BIM and point cloud data. The Random Sample Consensus (RANSAC) method is used to determine the optimal translation and rotation parameters to register the BIM and point cloud data. Next, the BIM model is converted to point cloud format by uniformly sampling points from each face of the building mesh model. The laser-scanned point cloud is similarly down-sampled to be at the same resolution as the BIM-derived point cloud. A point-to-point comparison sequence is carried out to measure the deviation of building elements between the BIM and laser-scanned point clouds. Regions in the point cloud are highlighted according to the degree of deviation to alert the user to the areas that require further inspection. Experiments were carried out using laser-scanned point clouds of an indoor hallway to validate the proposed approach. Results show that the proposed column-based registration method achieved a translation error rate of 0.15 meters and a rotation error rate of 0.068 degrees. The computation time required is 3 seconds for the column-based registration step and 70 seconds for the deviation detection step. The main contribution of this research is to propose a non-parametric, class-agnostic approach to deviation detection in order to handle the variation in the geometric shape of different building elements.

Conference Presentations by Jing Dao Chen

Research paper thumbnail of Workspace Modeling: Visualization and Pose Estimation of Teleoperated Construction Equipment from Point Clouds

37th International Symposium on Automation and Robotics in Construction (ISARC 2020), 2020

In order to teleoperate excavators remotely, human operators need accurate information of the rob... more In order to teleoperate excavators remotely, human operators need accurate information of the robot workspace to carry out manipulation tasks accurately and efficiently. Current visualization methods only allow for limited depth perception and situational awareness for the human operator, leading to high cognitive load when operating the robot in confined spaces or cluttered environments. This research proposes an advanced 3D workspace modeling method for remotely operated construction equipment where the environment is captured in real-time by laser scanning. A real-time 3D workspace state, which contains information such as the pose of end effectors, pose of salient objects, and distances between them, is used to provide feedback to the remote operator concerning the progress of manipulation tasks. The proposed method was validated at a mock urban disaster site where two excavators were teleoperated to pick up and move various debris. A 3D workspace model was constructed by laser scanning which was able to estimate the positions of the excavator and target assets within 0.1-0.2m accuracy.

Research paper thumbnail of LRGNet: Learnable Region Growing for Class-Agnostic Point Cloud Segmentation

Robotics and Automation Letters, 2021

3D point cloud segmentation is an important function that helps robots understand the layout of t... more 3D point cloud segmentation is an important function that helps robots understand the layout of their surrounding environment and perform tasks such as grasping objects, avoiding obstacles, and finding landmarks. Current segmentation methods are mostly class-specific, many of which are tuned to work with specific object categories and may not be generalizable to different types of scenes. This research proposes a learnable region growing method for class-agnostic point cloud segmentation, specifically for the task of instance label prediction. The proposed method is able to segment any class of objects using a single deep neural network without any assumptions about their shapes and sizes. The deep neural network is trained to predict how to add or remove points from a point cloud region to morph it into incrementally more complete regions of an object instance. Segmentation results on the S3DIS and ScanNet datasets show that the proposed method outperforms competing methods by 1%-9% on 6 different evaluation metrics.

Research paper thumbnail of Dynamic Crane Workspace Update for Collision Avoidance during Blind Lift Operations

Proceedings of the 18th International Conference on Computing in Civil and Building Engineering, 2020

Cranes are associated with high levels of accidents and struck-by fatalities in many industrial s... more Cranes are associated with high levels of accidents and struck-by fatalities in many industrial settings. This is due to the congested nature of crane operation sites and the high cognitive load for the crane operator to track the load position throughout the lifting operation, especially during blind lifts. Current research on crane safety assistance systems focuses on preventing collisions with static entities such as existing building structures but do not adequately handle dynamic entities such as workers and movable containers. This research proposes a dynamic crane workspace updating method for collision avoidance during blind lift operations. The position and orientation of the crane load, as well as the position and orientation of surrounding obstacles, are automatically tracked and updated in a 3D crane workspace model during lifting operations. The load base position is first estimated using forward kinematics obtained from an encoder system. Then the load swing and load rotation are corrected for using a vision-based load detection algorithm. A static map of surrounding obstacles is initially obtained using 3D laser scanning. The map is then updated over the course of a lifting operation through semi-automated obstacle placement. In addition, a vision-based worker detection algorithm is used to track the position of workers in proximity to the load. Finally, the crane workspace is updated in real-time for the crane operator through a 3D user interface and warnings are issued whenever a potential collision scenario is detected. The proposed method was evaluated at a crane yard where multiple blind lift scenarios were carried out. Experimental results indicate that the proposed method can potentially improve the situational awareness of the crane operator for blind lift operations.

Research paper thumbnail of User Exemplar-based Building Element Retrieval from Raw Point Clouds using Deep Point-level Features

Automation in Construction, 2020

3D point cloud data can be utilized for site inspection and reverse engineering of building model... more 3D point cloud data can be utilized for site inspection and reverse engineering of building models. However, conventional methods for building element retrieval require a database of 3D CAD or BIM models which are unsuitable for the case of historical buildings without as-planned models or temporary structures that are not in the pre-built model. Thus, this paper proposes a semi-automated method to efficiently retrieve duplicate building elements without these constraints. First, the point cloud is processed with a pre-trained deep feature extractor to generate a 50-dimensional feature vector for each point. Next, the point cloud is segmented through feature clustering and region-growing algorithms, then displayed on a user interface for selection. Lastly, the selected exemplar is provided as input to a peak-finding algorithm to determine positive matches. Experimental results on five different datasets show that the proposed method obtains average rates above 90% for precision and recall.

Research paper thumbnail of Deep Learning Approach to Point Cloud Scene Understanding for Automated Scan to 3D Reconstruction

Journal of Computing in Civil Engineering, 2019

Construction progress estimation is an essential component of the daily construction cycle to ens... more Construction progress estimation is an essential component of the daily construction cycle to ensure high productivity and quality. However, using laser-scanned point clouds for the purpose of measuring the deviation between the as-built structures and the as-planned Building Information Model (BIM) remains cumbersome due to the difficulty in data registration, segmentation, annotation and modeling in large-scale point clouds. This research proposes the use of a data-driven deep learning framework to automatically detect and classify building elements from a laser-scanned point cloud scene. The point cloud is first converted into a graph representation, where vertices represent points and edges represent connections between points within a fixed distance. An edge-based classifier is used to discard edges connecting points from different objects and form connected components from points in the same object. Next, a point-based object classifier is used to determine the type of building component based on the 1

Research paper thumbnail of Multi-view Incremental Segmentation of 3D Point Clouds for Mobile Robots

Robotics and Automation Letters, 2019

Mobile robots need to create high-definition 3D maps of the environment for applications such as ... more Mobile robots need to create high-definition 3D maps of the environment for applications such as remote surveillance and infrastructure mapping. Accurate semantic processing of the acquired 3D point cloud is critical for allowing the robot to obtain a high-level understanding of the surrounding objects and perform context-aware decision making. Existing techniques for point cloud semantic segmentation are mostly applied on a single-frame or offline basis, with no way to integrate the segmentation results over time. This paper proposes an online method for mobile robots to incrementally build a semantically-rich 3D point cloud of the environment. The proposed deep neural network, MCPNet, is trained to predict class labels and object instance labels for each point in the scanned point cloud in an incremental fashion. A multi-view context pooling (MCP) operator is used to combine point features obtained from multiple viewpoints to improve the classification accuracy. The proposed architecture was trained and evaluated on ray-traced scans derived from the Stanford 3D Indoor Spaces dataset. Results show that the proposed approach led to 15% improvement in point-wise accuracy and 7% improvement in NMI compared to the next best online method, with only a 6% drop in accuracy compared to the PointNet-based offline approach.

Research paper thumbnail of Enhancing Blind Lift Safety on Offshore Platforms through Real-time Sensing and Visualization

CONVR, 2018

Crane lift operations are human-centric tasks where lift safety heavily relies on the operator's ... more Crane lift operations are human-centric tasks where lift safety heavily relies on the operator's awareness of potential spatial constraints and associated safety risks. Blind lifts on congested offshore platform (OP) environments, therefore, are inherently dangerous because of substantial presence of spatial conflicts in the crane workspace and the operator's limited visibility to the load. This research aims to improve operators' spatial awareness in blind lifts in the OP environment through real-time crane states sensing and visualization. We propose a technical framework that consists of two sensing modules (i.e., crane motion monitoring and load sway monitoring) and a visualization module. An inertial measurement unit (IMU)-based approach and a computer vision (CV) based approach are introduced to track load position for load sway monitoring. A prototype system was built and tested in an offshore platform to validate the crane motion monitoring and visualization modules. A lab experiment was conducted to evaluate the CV-based load sway monitoring method. Results from the field and lab tests indicate the proposed framework and methods were able to continuously monitor and visualize the crane states in real-time and thus provide the operator adequate assistance to identify and mitigate unsafe conditions during blind lifts.

Research paper thumbnail of SLAM-driven Intelligent Autonomous Mobile Robot Navigation for Construction Applications

The demand for construction site automation with mobile robots is increasing due to its advantage... more The demand for construction site automation with mobile robots is increasing due to its advantages in potential cost-saving, productivity, and safety. To be realistically deployed in construction sites, mobile robots must be capable of navigating in unstructured and cluttered environments. Furthermore, mobile robots should recognize both static and dynamic obstacles to determine drivable paths. However, existing robot navigation methods are not suitable for construction applications due to the challenging environmental conditions in construction sites. This study introduces an autonomous as-is 3D spatial data collection and perception method for mobile robots specifically aimed for construction job sites with many spatial uncertainties. The proposed Simultaneous Localization and Mapping (SLAM)-based navigation and object recognition methods were implemented and tested with a custom-designed mobile robot platform, Ground Robot of Mapping Infrastructure (GRoMI), which uses multiple laser scanners and a camera to sense and build a 3D environment map. Since SLAM, it self, did not detect uneven surface conditions and spatiotemporal objects on the ground, an obstacle detection algorithm was developed to recognize and avoid obstacles and the highly uneven terrain in real time. Given the 3D real-time scan map generated by 3D laser scanners, a path-finding algorithm was developed for autonomous navigation in an unknown environment with obstacles. Overall, the 3D color-mapped point clouds of construction sites generated by GRoMI were of sufficient quality to be used for many construction management applications such as construction progress monitoring, safety hazard identification, and defect detection.

Research paper thumbnail of Point-to-point Comparison Method for Automated Scan-vs-BIM Deviation Detection

Chen, J. and Cho, Y. (2018). "Point-to-point Comparison Method for Automated Scan-vs-BIM Deviatio... more Chen, J. and Cho, Y. (2018). "Point-to-point Comparison Method for Automated Scan-vs-BIM Deviation Detection. Abstract: Laser scanning is one of the most accurate methods of measuring the geometric accuracy of the as-built condition of a construction site. However, using laser-scanned point clouds for the purpose of measuring the deviation between the as-built structures and the as-planned Building Information Model (BIM) remains cumbersome due to the difficulty in registration, segmentation, and matching of large-scale point clouds. Conventional methods for automated deviation detection are computationally intensive and only work for regular geometric shapes such as planes and cylinders. This research proposes a point-to-point comparison method for automated scan-vs-BIM deviation detection. First, a laser-scanned point cloud is collected and imported into a BIM file. A column-detection routine is used to locate the centroid of each column in the x-y plane for both the BIM and point cloud data. The Random Sample Consensus (RANSAC) method is used to determine the optimal translation and rotation parameters to register the BIM and point cloud data. Next, the BIM model is converted to point cloud format by uniformly sampling points from each face of the building mesh model. The laser-scanned point cloud is similarly down-sampled to be at the same resolution as the BIM-derived point cloud. A point-to-point comparison sequence is carried out to measure the deviation of building elements between the BIM and laser-scanned point clouds. Regions in the point cloud are highlighted according to the degree of deviation to alert the user to the areas that require further inspection. Experiments were carried out using laser-scanned point clouds of an indoor hallway to validate the proposed approach. Results show that the proposed column-based registration method achieved a translation error rate of 0.15 meters and a rotation error rate of 0.068 degrees. The computation time required is 3 seconds for the column-based registration step and 70 seconds for the deviation detection step. The main contribution of this research is to propose a non-parametric, class-agnostic approach to deviation detection in order to handle the variation in the geometric shape of different building elements.

Research paper thumbnail of Workspace Modeling: Visualization and Pose Estimation of Teleoperated Construction Equipment from Point Clouds

37th International Symposium on Automation and Robotics in Construction (ISARC 2020), 2020

In order to teleoperate excavators remotely, human operators need accurate information of the rob... more In order to teleoperate excavators remotely, human operators need accurate information of the robot workspace to carry out manipulation tasks accurately and efficiently. Current visualization methods only allow for limited depth perception and situational awareness for the human operator, leading to high cognitive load when operating the robot in confined spaces or cluttered environments. This research proposes an advanced 3D workspace modeling method for remotely operated construction equipment where the environment is captured in real-time by laser scanning. A real-time 3D workspace state, which contains information such as the pose of end effectors, pose of salient objects, and distances between them, is used to provide feedback to the remote operator concerning the progress of manipulation tasks. The proposed method was validated at a mock urban disaster site where two excavators were teleoperated to pick up and move various debris. A 3D workspace model was constructed by laser scanning which was able to estimate the positions of the excavator and target assets within 0.1-0.2m accuracy.