Reinforcement Learning Control of a Forestry Crane Manipulator (original) (raw)
Related papers
Robotic Arm Control and Task Training through Deep Reinforcement Learning
ArXiv, 2020
This paper proposes a detailed and extensive comparison of the Trust Region Policy Optimization and DeepQ-Network with Normalized Advantage Functions with respect to other state of the art algorithms, namely Deep Deterministic Policy Gradient and Vanilla Policy Gradient. Comparisons demonstrate that the former have better performances then the latter when asking robotic arms to accomplish manipulation tasks such as reaching a random target pose and pick &placing an object. Both simulated and real-world experiments are provided. Simulation lets us show the procedures that we adopted to precisely estimate the algorithms hyper-parameters and to correctly design good policies. Real-world experiments let show that our polices, if correctly trained on simulation, can be transferred and executed in a real environment with almost no changes.
Australasian Conference on Robotics and Automation, 2023
Applications using Model-Free Reinforcement Learning (MFRL) have grown exponentially and have shown remarkable results in the last decade. The application of MFRL to robots shows significant promise for its capability to solve complex control problems, at least virtually or in simulation. Due to the practical challenges of training in a real-world environment, there is limited work bridging the gap to real physical robots. This article benchmarks the state-of-the-art MFRL algorithms training on an open-source robotic manipulation testbed consisting of a fully actuated, 4-Degrees of Freedom (DoF), two-fingered robot gripper to understand the limitations and challenges involved in real-world applications. Experimental analysis using two different statespace representations is presented to understand their impact on executing a dexterous manipulation task. The source code, the CAD files of the robotic manipulation testbed, and a handy guide on how to approach MFRL's application to real-world are provided to facilitate replication of the results and further experimentation by other research groups.
International Journal of Intelligent Robotics and Applications, 2020
For robot manipulation, reinforcement learning has provided an effective end to end approach in controlling the complicated dynamic system. Model-free reinforcement learning methods ignore the model of system dynamics and are limited to simple behavior control. By contrast, model-based methods can quickly reach optimal trajectory planning by building a dynamic system model. However, it is not easy to build an accurate and efficient system model with high generalization ability, especially when facing complex dynamic system and various manipulation tasks. Furthermore, when the rewards provided by the environment are sparse, the agent will also lose effective guidance and fail to optimize the policy efficiently, which results in considerably decreased sample efficiency. In this paper, a model-based deep reinforcement learning algorithm, in which a deep neural network model is utilized to simulate the system dynamics, is designed for robot manipulation. The proposed deep neural network model is robust enough to deal with complex control tasks and possesses the generalization ability. Moreover, a curiosity-based experience replay method is incorporated to solve the sparse reward problem and improve the sample efficiency in reinforcement learning. The agent who manipulates a robotic hand, will be encouraged to explore optimal trajectories according to the failure experience. Simulation experiment results show great effectiveness of the proposed method. Various manipulation tasks are achieved successfully in such a complex dynamic system and the sample efficiency gets improved even in a sparse reward environment, as the learning time gets reduced considerably.
A Survey on Deep Reinforcement Learning Algorithms for Robotic Manipulation
Sensors
Robotic manipulation challenges, such as grasping and object manipulation, have been tackled successfully with the help of deep reinforcement learning systems. We give an overview of the recent advances in deep reinforcement learning algorithms for robotic manipulation tasks in this review. We begin by outlining the fundamental ideas of reinforcement learning and the parts of a reinforcement learning system. The many deep reinforcement learning algorithms, such as value-based methods, policy-based methods, and actor–critic approaches, that have been suggested for robotic manipulation tasks are then covered. We also examine the numerous issues that have arisen when applying these algorithms to robotics tasks, as well as the various solutions that have been put forth to deal with these issues. Finally, we highlight several unsolved research issues and talk about possible future directions for the subject.
IAES International Journal of Robotics and Automation, 2024
Forestry cranes are an important tool for safe and efficient timber harvesting with forestry machines. However, their complex manual control often led to inefficiencies and excessive energy usage, due to the many joysticks and buttons that must be used in a precise sequence to perform efficient movements. To address this, the industry is increasingly turning to partial automation, making manual control more intuitive for the operator and, consequently, achieving improvements in energy efficiency. This article introduces a novel approach to energy-optimal motion planning that can be used along with a feedback control system to automate crane motions, taking over portions of the operator's work. Our method combines dynamic movement primitives (DMPs) and an energy-optimization algorithm. DMPs is a machine learning technique for motion planning based on human demonstrations, while the optimization algorithm exploits the crane's redundancy to find energy-optimal trajectories. Simulation results show that DMPs can replicate human-like controlled motions with a 25% reduction in energy consumption. However, our energy optimization algorithm shows improvements of over 40%, providing substantial energy savings and a promising pathway towards environmentally friendly partially automated machines.
On Training Flexible Robots using Deep Reinforcement Learning
2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019
The use of robotics in controlled environments has flourished over the last several decades and training robots to perform tasks using control strategies developed from dynamical models of their hardware have proven very effective. However, in many real-world settings, the uncertainties of the environment, the safety requirements and generalized capabilities that are expected of robots make rigid industrial robots unsuitable. This created great research interest into developing control strategies for flexible robot hardware for which building dynamical models are challenging. In this paper, inspired by the success of deep reinforcement learning (DRL) in other areas, we systematically study the efficacy of policy search methods using DRL in training flexible robots. Our results indicate that DRL is successfully able to learn efficient and robust policies for complex tasks at various degrees of flexibility. We also note that DRL using Deep Deterministic Policy Gradients can be sensitive to the choice of sensors and adding more informative sensors does not necessarily make the task easier to learn.
Deep Reinforcement Learning for Contact-Rich Skills Using Compliant Movement Primitives
2020
In recent years, industrial robots have been installed in various industries to handle advanced manufacturing and high precision tasks. However, further integration of industrial robots is hampered by their limited flexibility, adaptability and decision making skills compared to human operators. Assembly tasks are especially challenging for robots since they are contact-rich and sensitive to even small uncertainties. While reinforcement learning (RL) offers a promising framework to learn contact-rich control policies from scratch, its applicability to high-dimensional continuous state-action spaces remains rather limited due to high brittleness and sample complexity. To address those issues, we propose different pruning methods that facilitate convergence and generalization. In particular, we divide the task into free and contact-rich sub-tasks, perform the control in Cartesian rather than joint space, and parameterize the control policy. Those pruning methods are naturally implemen...
Control of rough terrain vehicles using deep reinforcement learning
ArXiv, 2021
We explore the potential to control terrain vehicles using deep reinforcement in scenarios where human operators and traditional control methods are inadequate. This letter presents a controller that perceives, plans, and successfully controls a 16-tonne forestry vehicle with two frame articulation joints, six wheels, and their actively articulated suspensions to traverse rough terrain. The carefully shaped reward signal promotes safe, environmental, and efficient driving, which leads to the emergence of unprecedented driving skills. We test learned skills in a virtual environment, including terrains reconstructed from high-density laser scans of forest sites. The controller displays the ability to handle obstructing obstacles, slopes up to 27â—¦, and a variety of natural terrains, all with limited wheel slip, smooth, and upright traversal with intelligent use of the active suspensions. The results confirm that deep reinforcement learning has the potential to enhance control of vehicl...
Autonomous Robots, 2022
This paper presents a learning-based method that uses simulation data to learn an object manipulation task using two model-free reinforcement learning (RL) algorithms. The learning performance is compared across on-policy and off-policy algorithms: Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC). In order to accelerate the learning process, the fine-tuning procedure is proposed that demonstrates the continuous adaptation of on-policy RL to new environments, allowing the learned policy to adapt and execute the (partially) modified task. A dense reward function is designed for the task to enable an efficient learning of the agent. A grasping task involving a Franka Emika Panda manipulator is considered as the reference task to be learned. The learned control policy is demonstrated to be generalizable across multiple object geometries and initial robot/parts configurations. The approach is finally tested on a real Franka Emika Panda robot, showing the possibility to tran...
Deep Agency: Towards human guided robotic training for assembly tasks in timber construction
Proceedings of the 42 International Conference on Education and Research in Computer Aided Architectural Design in Europe (eCAADe) [Volume 1] , 2024
Automating robotic assembly in architectural construction is challenging due to material uncertainties and the buildup of tolerance experienced in assembling many parts. Implementing AI technologies, including various machine learning algorithms in robotic assembly, has demonstrated significant potential for robots to respond to this uncertainty. This research builds upon previous implementations of singular algorithms by combining haptic teaching with deep reinforcement learning (DRL) in a single workflow to improve robot autonomy in responding to uncertainties in timber assembly. Haptic teaching bridges the gap between simulation and reality inherent to DRL, while DRL improves the generalizability of haptic teaching and speeds up the agents' learning process. The developed workflow is tested through various lap joint assembly experiments. The effectiveness of this combined approach is assessed through various experiments that refine and evaluate the methodology, providing valuable insights into enhancing robot capabilities to manage material uncertainties and deviations. Additionally, this research considers the evolving role of human workers in collaborative construction environments with robots.