FedControl: When Control Theory Meets Federated Learning (original) (raw)
Related papers
A strategy to the reduction of communication overhead and overfitting in Federated Learning
Anais do XXVI Workshop de Gerência e Operação de Redes e Serviços (WGRS 2021), 2021
Federated learning (FL) is a framework to train machine learning models using decentralized data, especially unbalanced and non-iid. Adaptive methods can be used to accelerate convergence, reducing the number of rounds of local computation and communication to a centralized server. This paper proposes an adaptive controller to adapt the number of epochs needed that employs Poisson distribution to avoid overfitting of the aggregated model, promoting fast convergence. Our results indicate that increasing the local update of the model should be avoided, but yet some complementary mechanism is needed to model performance. We evaluate the impact of an increasing number of epochs of FedAVG and FedADAM.
Federated Asymptotics: a model to compare federated learning algorithms
2021
We propose an asymptotic framework to analyze the performance of (personalized) federated learning algorithms. In this new framework, we formulate federated learning as a multi-criterion objective, where the goal is to minimize each client’s loss using information from all of the clients. We analyze a linear regression model where, for a given client, we may theoretically compare the performance of various algorithms in the high-dimensional asymptotic limit. This asymptotic multicriterion approach naturally models the high-dimensional, many-device nature of federated learning. These tools make fairly precise predictions about the benefits of personalization and information sharing in federated scenarios—at least in our (stylized) model—including that Federated Averaging with simple client fine-tuning achieves the same asymptotic risk as the more intricate metalearning and proximal-regularized approaches and outperforming Federated Averaging without personalization. We evaluate these...
Adaptive Personalized Federated Learning
ArXiv, 2020
Investigation of the degree of personalization in federated learning algorithms has shown that only maximizing the performance of the global model will confine the capacity of the local models to personalize. In this paper, we advocate an adaptive personalized federated learning (APFL) algorithm, where each client will train their local models while contributing to the global model. Information theoretically, we prove that the mixture of local and global models can reduce the generalization error. We also propose a communication-reduced bilevel optimization method, which reduces the communication rounds to O(sqrtT)O(\sqrt{T})O(sqrtT) and show that under strong convexity and smoothness assumptions, the proposed algorithm can achieve a convergence rate of O(1/T)O(1/T)O(1/T) with some residual error. The residual error is related to the gradient diversity among local models, and the gap between optimal local and global models. The extensive experiments demonstrate the effectiveness of our personalization, as ...
Optimization in Federated Learning
2019
Federated learning (FL) is an emerging branch of machine learning (ML) research, that is examining the methods for scenarios, where individual nodes possess parts of the data, and the task is to form a single common model that fits to the whole distribution. In FL, we generally use mini batch gradient descent for optimizing weights of the models that appears to work very well for federated scenarios. For traditional machine learning setups, a number of modifications has been proposed to accelerate the learning process and to help to get over challenges posed by the high dimensionality and nonconvexity of search spaces of the parameters. In this paper we present our experiments on applying different popular optimization methods for training neural networks in a federated manner. 1 Federated Learning Federated learning (FL) [1] is a new paradigm in Machine Learning (ML), that is dealing with an increasingly important distributed optimization setting, that came into view with the sprea...
FEDERATED LEARNING OPTIMIZATION TECHNIQUES FOR NON-IID DATA: A REVIEW
IAEME PUBLICATION, 2020
Federated learning is a distributed machine learning setting that can effectively assist multiple clients (e.g. mobile phones, IoT devices and organizations) to conduct isolated data use and machine learning modeling in accordance with user privacy protection, data security and government regulations. The participants can benefit from the federated learning training process or outcomes without sharing any local raw data. However, this special structure also determines that federated learning is bound to encounter statistical challenge which principally performs as non-independently and identically distributed (non-IID) data problem. This issue can directly affect the performance of federated learning, resulting in inferior performance of machine learning models, like poor prediction accuracy, additional communication overhead and low convergence rate. Therefore, it is of great significance for federated learning to study this non-IID data issue. At present, many optimization techniques have been proposed to tackle this problem, nonetheless they are various and lack a unified classification standard. To provide that standard, this paper reviews and classifies the optimization approaches which tackle this issue of federated learning, and summarizes the characteristics of each type of method based on several major influencing factors
Federated Learning Algorithms to Optimize the Client and Cost Selections
Mathematical Problems in Engineering
In recent years, federated learning has received widespread attention as a technology to solve the problem of data islands, and it has begun to be applied in fields such as finance, healthcare, and smart cities. The federated learning algorithm is systematically explained from three levels. First, federated learning is defined through the definition, architecture, classification of federated learning, and comparison with traditional distributed knowledge. Then, based on machine learning and deep learning, the current types of federated learning algorithms are classified, compared, and analyzed in-depth. Finally, the communication from the perspectives of cost, client selection, and aggregation method optimization, the federated learning optimization algorithms are classified. Finally, the current research status of federated learning is summarized. Finally, the three major problems and solutions of communication, system heterogeneity, and data heterogeneity faced by federated learni...
FedPerf: A Practitioners’ Guide to Performance of Federated Learning Algorithms
Federated Learning (FL) enables the edge devices to collaboratively train a joint model without sharing their local data. This decentralised and distributed approach improves user privacy, security, and trust. Different variants of FL algorithms have presented promising results on both IID and skewed Non-IID data. However, the performance of FL algorithms is found to be sensitive to the FL system parameters and hyperparameters of the used model. In practice, tuning the right set of parameter settings for an FL algorithm is an expensive task. In this preregister paper, we propose an empirical investigation on five prominent FL algorithms to discover the relation between the FL System Parameters (FLSPs) and their performance. The FLSPs adds extra complexity to FL algorithms over a traditional ML system. We hypothesise that choosing the best FL algorithm for the given FLSP is not a trivial problem. Further, we endeavour to formulate a single easy-to-use metric which can describe the pe...
Node Selection Toward Faster Convergence for Federated Learning on Non-IID Data
IEEE Transactions on Network Science and Engineering, 2022
Federated Learning (FL) is a distributed learning paradigm that enables a large number of resource-limited nodes to collaboratively train a model without data sharing. The non-independent-and-identically-distributed (non-i.i.d.) data samples invoke discrepancy between global and local objectives, making the FL model slow to converge. In this paper, we proposed Optimal Aggregation algorithm for better aggregation, which finds out the optimal subset of local updates of participating nodes in each global round, by identifying and excluding the adverse local updates via checking the relationship between the local gradient and the global gradient. Then, we proposed a Probabilistic Node Selection framework (FedPNS) to dynamically change the probability for each node to be selected based on the output of Optimal Aggregation. FedPNS can preferentially select nodes that propel faster model convergence. The convergence rate improvement of FedPNS over the commonly adopted Federated Averaging (FedAvg) algorithm is analyzed theoretically. Experimental results demonstrate the effectiveness of FedPNS in accelerating the FL convergence rate, as compared to FedAvg with random node selection.
Fine-tuning in Federated Learning: a simple but tough-to-beat baseline
2021
We study the performance of federated learning algorithms and their variants in an asymptotic framework. Our starting point is the formulation of federated learning as a multi-criterion objective, where the goal is to minimize each client’s loss using information from all of the clients. We analyze a linear regression model, where, for a given client, we theoretically compare the performance of various algorithms in the high-dimensional asymptotic limit. This asymptotic multi-criterion approach naturally models the high-dimensional, many-device nature of federated learning and suggests that personalization is central to federated learning. In this paper, we investigate how some sophisticated personalization algorithms fare against simple fine-tuning baselines. In particular, our theory suggests that Federated Averaging with client fine-tuning is competitive than more intricate meta-learning and proximal-regularized approaches. In addition to being conceptually simpler, our fine-tuni...
ArXiv, 2021
Federated Learning allows training of data stored in distributed devices without the need for centralizing training data, thereby maintaining data privacy. Addressing the ability to handle data heterogeneity (non-identical and independent distribution or non-IID) is a key enabler for the wider deployment of Federated Learning. In this paper, we propose a novel Divide-and-Conquer training methodology that enables the use of the popular FedAvg aggregation algorithm by overcoming the acknowledged FedAvg limitations in nonIID environments. We propose a novel use of Cosine-distance based Weight Divergence metric to determine the exact point where a Deep Learning network can be divided into class agnostic initial layers and class-specific deep layers for performing a Divide and Conquer training. We show that the methodology achieves trained model accuracy at par (and in certain cases exceeding) with numbers achieved by state-of-theart Aggregation algorithms like FedProx, FedMA, etc. Also,...