Deep Q-Network Based Dynamic Movement Strategy in a UAV-Assisted Network (original) (raw)
Related papers
DQN-Based Proactive Trajectory Planning of UAVs in Multi-Access Edge Computing
Computers, Materials & Continua
The main aim of future mobile networks is to provide secure, reliable, intelligent, and seamless connectivity. It also enables mobile network operators to ensure their customer's a better quality of service (QoS). Nowadays, Unmanned Aerial Vehicles (UAVs) are a significant part of the mobile network due to their continuously growing use in various applications. For better coverage, cost-effective, and seamless service connectivity and provisioning, UAVs have emerged as the best choice for telco operators. UAVs can be used as flying base stations, edge servers, and relay nodes in mobile networks. On the other side, Multi-access Edge Computing (MEC) technology also emerged in the 5G network to provide a better quality of experience (QoE) to users with different QoS requirements. However, UAVs in a mobile network for coverage enhancement and better QoS face several challenges such as trajectory designing, path planning, optimization, QoS assurance, mobility management, etc. The efficient and proactive path planning and optimization in a highly dynamic environment containing buildings and obstacles are challenging. So, an automated Artificial Intelligence (AI) enabled QoSaware solution is needed for trajectory planning and optimization. Therefore, this work introduces a well-designed AI and MEC-enabled architecture for a UAVs-assisted future network. It has an efficient Deep Reinforcement Learning (DRL) algorithm for real-time and proactive trajectory planning and optimization. It also fulfills QoS-aware service provisioning. A greedypolicy approach is used to maximize the long-term reward for serving more users with QoS. Simulation results reveal the superiority of the proposed DRL mechanism for energy-efficient and QoS-aware trajectory planning over the existing models.
A Deep Q-Learning Based UAV Detouring Algorithm in a Constrained Wireless Sensor Network Environment
MDPI Electronics, 2024
Unmanned aerial vehicles (UAVs) play a crucial role in various applications, including environmental monitoring, disaster management, and surveillance, where timely data collection is vital. However, their effectiveness is often hindered by the limitations of wireless sensor networks (WSNs), which can restrict communications due to bandwidth constraints and limited energy resources. Thus, the operational context of the UAV is intertwined with the constraints on WSNs, influencing how they are deployed and the strategies used to optimize their performance in these environments. Considering the issues, this paper addresses the challenge of efficient UAV navigation in constrained environments while reliably collecting data from WSN nodes, recharging the sensor nodes’ power supplies, and ensuring the UAV detours around obstacles in the flight path. First, an integer linear programming (ILP) optimization problem named deadline and obstacle-constrained energy minimization (DOCEM) is defined and formulated to minimize the total energy consumption of the UAV. Then, a deep reinforcement learning-based algorithm, named the DQN-based UAV detouring algorithm, is proposed to enable the UAV to make intelligent detour decisions in the constrained environment. The UAV must finish its tour (data collection and recharging sensors) without exceeding its battery capacity, ensuring each sensor has the minimum residual energy and consuming energy for transmitting and generating data, after being recharged by the UAV at the end of the tour. Finally, simulation results demonstrate the effectiveness of the proposed DQN-based UAV detouring algorithm in data collection and recharging the sensors while minimizing the total energy consumption of the UAV. Compared to other baseline algorithm variants, the proposed algorithm outperforms all of them
International Journal of Computer Networks and Applications (IJCNA), 2023
The design of efficient routing strategies for Unmanned Aerial Vehicle (UAV) Networks is a multidomain task that involves analysis of node-level & network-level parameters, and mapping them with communication & contextual conditions. Existing path planning optimization models either showcase higher complexity or cannot be scaled for larger network scenarios. Moreover, the efficiency of these models also reduces w.r.t. the number of communication requests, which limits their scalability levels. To get a better result over these challenges, this article provides an idea to design an efficient Q-Learning model to improve the routing efficiency of UAV networks via bioinspired optimizations. The model initially collects temporal routing performance data samples for individual nodes and uses them to form coarse routes via Q-Learning optimizations. These routes are further processed via a Mayfly Optimization (MO) Model, which assists in the selection of optimal routing paths for high Quality of Service (QoS) even under large-scale routing requests. The MO Model can identify alternate paths via the evaluation of a highdensity routing fitness function that assists the router in case the selected paths are occupied during current routing requests. This assists in improving temporal routing performance even under dense network conditions. Due to these optimizations, the model is capable of reducing the routing delay by 8.5%, improving energy efficiency by 4.9%, and reducing the routing jitter by 3.5% when compared with existing routing techniques by taking similar routing conditions.
Powering UAV with Deep Q-Network for Air Quality Tracking
Sensors
Tracking the source of air pollution plumes and monitoring the air quality during emergency events in real-time is crucial to support decision-makers in making an appropriate evacuation plan. Internet of Things (IoT) based air quality tracking and monitoring platforms have used stationary sensors around the environment. However, fixed IoT sensors may not be enough to monitor the air quality in a vast area during emergency situations. Therefore, many applications consider utilizing Unmanned Aerial Vehicles (UAVs) to monitor the air pollution plumes environment. However, finding an unhealthy location in a vast area requires a long navigation time. For time efficiency, we employ deep reinforcement learning (Deep RL) to assist UAVs to find air pollution plumes in an equal-sized grid space. The proposed Deep Q-network (DQN) based UAV Pollution Tracking (DUPT) is utilized to guide the multi-navigation direction of the UAV to find the pollution plumes’ location in a vast area within a shor...
arXiv (Cornell University), 2022
Unmanned aerial vehicles (UAVs) are increasingly deployed to provide wireless connectivity to static and mobile ground users in situations of increased network demand or points of failure in existing terrestrial cellular infrastructure. However, UAVs are energy-constrained and experience the challenge of interference from nearby UAV cells sharing the same frequency spectrum, thereby impacting the system's energy efficiency (EE). Recent approaches focus on optimising the system's EE by optimising the trajectory of UAVs serving only static ground users and neglecting mobile users. Several others neglect the impact of interference from nearby UAV cells, assuming an interference-free network environment. Furthermore, some works assume global spatial knowledge of ground users' location via a central controller (CC) that periodically scans the network perimeter and provides real-time updates to the UAVs for decision-making. However, this assumption may be unsuitable in disaster scenarios since it requires significant information exchange between the UAVs and CC. Moreover, it may not be possible to track users' locations in a disaster scenario. Despite growing research interest in decentralised control over centralised UAVs' control, direct collaboration among UAVs to improve coordination while optimising the systems' EE has not been adequately explored. To address this, we propose a direct collaborative communication-enabled multi-agent decentralised double deep Qnetwork (CMAD-DDQN) approach. The CMAD-DDQN is a collaborative algorithm that allows UAVs to explicitly share their telemetry via existing 3GPP guidelines by communicating with their nearest neighbours. This allows the agent-controlled UAVs to optimise their 3D flight trajectories by filling up knowledge gaps and converging to optimal policies. We account for the mobility of ground users, the UAVs' limited energy budget and interference in the environment. Our approach can maximise the system's EE without hampering performance gains in the network. Simulation results show that the proposed approach outperforms existing baselines in terms of maximising the systems' EE without degrading coverage performance in the network. The CMAD-DDQN approach outperforms the MAD-DDQN that neglects direct collaboration among UAVs, the multi-agent deep deterministic policy gradient (MADDPG) and random policy approaches that consider a 2D UAV deployment design while neglecting interference from nearby UAV cells by about 15%, 65% and 85%, respectively.
Optimising Energy Efficiency in UAV-Assisted Networks using Deep Reinforcement Learning
arXiv (Cornell University), 2022
In this letter, we study the energy efficiency (EE) optimisation of unmanned aerial vehicles (UAVs) providing wireless coverage to static and mobile ground users. Recent multiagent reinforcement learning approaches optimise the system's EE using a 2D trajectory design, neglecting interference from nearby UAV cells. We aim to maximise the system's EE by jointly optimising each UAV's 3D trajectory, number of connected users, and the energy consumed, while accounting for interference. Thus, we propose a cooperative Multi-Agent Decentralised Double Deep Q-Network (MAD-DDQN) approach. Our approach outperforms existing baselines in terms of EE by as much as 55-80%.
Optimizing Energy Efficiency in UAV-Assisted Networks Using Deep Reinforcement Learning
IEEE Wireless Communications Letters
In this letter, we study the energy efficiency (EE) optimization of unmanned aerial vehicles (UAVs) providing wireless coverage to static and mobile ground users. Recent multiagent reinforcement learning approaches optimise the system's EE using a 2D trajectory design, neglecting interference from nearby UAV cells. We aim to maximize the system's EE by jointly optimizing each UAV's 3D trajectory, number of connected users, and the energy consumed, while accounting for interference. Thus, we propose a cooperative Multi-Agent Decentralized Double Deep Q-Network (MAD-DDQN) approach. Our approach outperforms existing baselines in terms of EE by as much as 55-80%.
Model-aided Deep Reinforcement Learning for Sample-efficient UAV Trajectory Design in IoT Networks
2021
Deep Reinforcement Learning (DRL) is gaining attention as a potential approach to design trajectories for autonomous unmanned aerial vehicles (UAV) used as flying access points in the context of cellular or Internet of Things (IoT) connectivity. DRL solutions offer the advantage of on-the-go learning hence relying on very little prior contextual information. A corresponding drawback however lies in the need for many learning episodes which severely restricts the applicability of such approach in real-world timeand energy-constrained missions. Here, we propose a model-aided deep Q-learning approach that, in contrast to previous work, considerably reduces the need for extensive training data samples, while still achieving the overarching goal of DRL, i.e to guide a battery-limited UAV on an efficient data harvesting trajectory, without prior knowledge of wireless channel characteristics and limited knowledge of wireless node locations. The key idea consists in using a small subset of ...