Distributed Uplink Beamforming in Cell-Free Networks Using Deep Reinforcement Learning (original) (raw)
Related papers
Deep Reinforcement Learning-based Power Allocation in Uplink Cell-Free Massive MIMO
2022 IEEE Wireless Communications and Networking Conference (WCNC)
A cell-free massive multiple-input multiple-output (MIMO) uplink is investigated in this paper. We address a power allocation design problem that considers two conflicting metrics, namely the sum rate and fairness. Different weights are allocated to the sum rate and fairness of the system, based on the requirements of the mobile operator. The knowledge of the channel statistics is exploited to optimize power allocation. We propose to employ large scale-fading (LSF) coefficients as the input of a twin delayed deep deterministic policy gradient (TD3). This enables us to solve the non-convex sum rate fairness trade-off optimization problem efficiently. Then, we exploit a use-and-then-forget (UatF) technique, which provides a closedform expression for the achievable rate. The sum rate fairness trade-off optimization problem is subsequently solved through a sequential convex approximation (SCA) technique. Numerical results demonstrate that the proposed algorithms outperform conventional power control algorithms in terms of both the sum rate and minimum user rate. Furthermore, the TD3-based approach can increase the median of sum rate by 16%-46% and the median of minimum user rate by 11%-60% compared to the proposed SCA-based technique. Finally, we investigate the complexity and convergence of the proposed scheme. cc Index terms-Cell-free massive MIMO, deep reinforcement learning, fairness, power control, sequential convex approximation.
2021
Heterogeneous network (HetNet) is now considered to be a promising technique for enhancing the coverage and reducing the transmit power consumption of the next 5G system. Deploying small cells such as femtocells in the current macrocell networks achieves great spatial reuse at the cost of severe cross-tier interference from concurrent transmission. In this situation, two novel energy efficient power control and resource allocation schemes in terms of energy efficiency (EE)-fairness and EE-maximum, respectively, are investigated in this paper. In the EE-fairness scheme, we aim to maximize the minimum EE of the femtocell base stations (FBSs). Generalized Dinkelbach's algorithm (GDA) is utilized to tackle this optimization problem and a distributed algorithm is proposed to solve the subproblem in GDA with limited intercell coordination, in which only a few scalars are shared among FBSs. In the EE-maximum scheme, we aim to maximize the global EE of all femtocells which is defined as the aggregate capacity over the aggregate power consumption in the femtocell networks. Leveraged by means of the lower-bound of logarithmic function, a centralized algorithm with limited computational complexity is proposed to solve the global EE maximization problem. Simulation results show that the proposed algorithms outperform previous schemes in terms of the minimum EE, fairness and global EE.
Distributed Deep Reinforcement Learning with Wideband Sensing for Dynamic Spectrum Access
2020 IEEE Wireless Communications and Networking Conference (WCNC), 2020
Dynamic Spectrum Access (DSA) improves spectrum utilization by allowing secondary users (SUs) to opportunistically access temporary idle periods in the primary user (PU) channels. Previous studies on utility maximizing spectrum access strategies mostly require complete network state information, therefore, may not be practical. Model-free reinforcement learning (RL) based methods, such as Q-learning, on the other hand, are promising adaptive solutions that do not require complete network information. In this paper, we tackle this research dilemma and propose deep Q-learning originated spectrum access (DQLS) based decentralized and centralized channel selection methods for network utility maximization, namely DEcentralized Spectrum Allocation (DESA) and Centralized Spectrum Allocation (CSA), respectively. Actions that are generated through centralized deep Q-network (DQN) are utilized in CSA whereas the DESA adopts a non-cooperative approach in spectrum decisions. We use extensive si...
Power Allocation in Multi-User Cellular Networks: Deep Reinforcement Learning Approaches
IEEE Transactions on Wireless Communications
The model-based power allocation algorithm has been investigated for decades, but it requires the mathematical models to be analytically tractable and it usually has high computational complexity. Recently, the data-driven model-free machine learning enabled approaches are being rapidly developed to obtain near-optimal performance with affordable computational complexity, and deep reinforcement learning (DRL) is regarded as of great potential for future intelligent networks. In this paper, the DRL approaches are considered for power control in multiuser wireless communication cellular networks. Considering the cross-cell cooperation, the off-line/on-line centralized training and the distributed execution, we present a mathematical analysis for the DRL-based top-level design. The concrete DRL design is further developed based on this foundation, and policy-based REINFORCE, value-based deep Q learning (DQL), actor-critic deep deterministic policy gradient (DDPG) algorithms are proposed. Simulation results show that the proposed data-driven approaches outperform the state-of-art modelbased methods on sum-rate performance, with good generalization power and faster processing speed. Furthermore, the proposed DDPG outperforms the REINFORCE and DQL in terms of both sum-rate performance and robustness, and can be incorporated into existing resource allocation schemes due to its generality.
IEEE Access, 2021
Exploiting the millimeter wave (mmWave) band is an attractive solution to accommodate the bandwidth-intensive applications in device-to-device (D2D) communications. The directional nature of communications at mmWave frequencies and mobility of devices require beam alignment at both transmitter and receiver ends. The beam alignment signaling overhead leads to a loss in the network's throughput. There exists a trade-off between antenna beamwidth and the achievable throughput. Although a narrower antenna beam increases the directivity gain, it leads to a higher signaling overhead and less stable D2D links which reduce the network's throughput. Therefore, optimizing the antenna beamwidth is crucial to maintain the D2D users' quality-of-experience (QoE). In this paper, we propose a novel distributed antenna beamwidth optimization algorithm based on multi-agent deep reinforcement learning. We model D2D links as agents that interact with the communication environment concurrently and learn to refine their antenna beamwidth policies. Agents aim to maximize the network sum-throughput and maintain reliable communication links while taking into account the application-specific quality-of-service (QoS) requirements and the cost associated with beam alignment. Online deployment of the proposed algorithm is distributed and does not require any coordination among users. The performance of the proposed antenna beamwidth optimization algorithm is compared with other widely used baseline algorithms. Numerical results show that our proposed algorithm improves the network performance significantly and outperforms existing approaches.
Limited-Fronthaul Cell-Free Hybrid Beamforming with Distributed Deep Neural Network
2021
Cell-free massive MIMO (CF-mMIMO) systems represent a promising approach to increase the spectral efficiency of wireless communication systems. However, near-optimal solutions require a large amount of signaling exchange between access points (APs) and the network controller (NC). In addition, the use of hybrid beamforming in each AP reduces the number of power hungry RF chains, but imposes a large computational complexity to find near-optimal precoders. In this letter, we propose two unsupervised deep neural networks (DNN) architectures, fully and partially distributed, that can perform coordinated hybrid beamforming with zero or limited communication overhead between APs and NC, while achieving near-optimal sum-rate with a reduced computational complexity compared to conventional near-optimal solutions.
Spectrum Sharing between Cellular and Wi-Fi Networks based on Deep Reinforcement Learning
International journal of Computer Networks & Communications
Recently, mobile traffic is growing rapidly and spectrum resources are becoming scarce in wireless networks. Due to this, the wireless network capacity will not meet the traffic demand. To address this problem, using cellular systems in an unlicensed spectrum emerged as an effective solution. In this case, cellular systems need to coexist with Wi-Fi and other systems. For that, we propose an efficient channel assignment method for Wi-Fi AP and cellular NB, based on the DRL method. To train the DDQN model, we implement an emulator as an environment for spectrum sharing in densely deployed NB and APs in wireless heterogeneous networks. Our proposed DDQN algorithm improves the average throughput from 25.5% to 48.7% in different user arrival rates compared to the conventional method. We evaluated the generalization performance of the trained agent, to confirm channel allocation efficiency in terms of average throughput under the different user arrival rates.
Wireless Communications and Mobile Computing, 2020
Enhanced licensed-assisted access (eLAA) is an operational mode that allows the use of unlicensed band to support long-term evolution (LTE) service via carrier aggregation technology. The extension of additional bandwidth is beneficial to meet the demands of the growing mobile traffic. In the uplink eLAA, which is prone to unexpected interference from WiFi access points, resource scheduling by the base station, and then performing a listen before talk (LBT) mechanism by the users can seriously affect the resource utilization. In this paper, we present a decentralized deep reinforcement learning (DRL)-based approach in which each user independently learns dynamic band selection strategy that maximizes its own rate. Through extensive simulations, we show that the proposed DRL-based band selection scheme improves resource utilization while supporting certain minimum quality of service (QoS).
Deep Reinforcement Learning for Joint Spectrum and Power Allocation in Cellular Networks
2021 IEEE Globecom Workshops (GC Wkshps), 2021
A wireless network operator typically divides the radio spectrum it possesses into a number of subbands. In a cellular network those subbands are then reused in many cells. To mitigate co-channel interference, a joint spectrum and power allocation problem is often formulated to maximize a sum-rate objective. The best known algorithms for solving such problems generally require instantaneous global channel state information and a centralized optimizer. In fact those algorithms have not been implemented in practice in large networks with time-varying subbands. Deep reinforcement learning algorithms are promising tools for solving complex resource management problems. A major challenge here is that spectrum allocation involves discrete subband selection, whereas power allocation involves continuous variables. In this paper, a learning framework is proposed to optimize both discrete and continuous decision variables. Specifically, two separate deep reinforcement learning algorithms are designed to be executed and trained simultaneously to maximize a joint objective. Simulation results show that the proposed scheme outperforms both the state-of-the-art fractional programming algorithm and a previous solution based on deep reinforcement learning.