FedControl: When Control Theory Meets Federated Learning (original) (raw)

A strategy to the reduction of communication overhead and overfitting in Federated Learning

Anais do XXVI Workshop de Gerência e Operação de Redes e Serviços (WGRS 2021), 2021

Federated learning (FL) is a framework to train machine learning models using decentralized data, especially unbalanced and non-iid. Adaptive methods can be used to accelerate convergence, reducing the number of rounds of local computation and communication to a centralized server. This paper proposes an adaptive controller to adapt the number of epochs needed that employs Poisson distribution to avoid overfitting of the aggregated model, promoting fast convergence. Our results indicate that increasing the local update of the model should be avoided, but yet some complementary mechanism is needed to model performance. We evaluate the impact of an increasing number of epochs of FedAVG and FedADAM.

Federated Asymptotics: a model to compare federated learning algorithms

2021

We propose an asymptotic framework to analyze the performance of (personalized) federated learning algorithms. In this new framework, we formulate federated learning as a multi-criterion objective, where the goal is to minimize each client’s loss using information from all of the clients. We analyze a linear regression model where, for a given client, we may theoretically compare the performance of various algorithms in the high-dimensional asymptotic limit. This asymptotic multicriterion approach naturally models the high-dimensional, many-device nature of federated learning. These tools make fairly precise predictions about the benefits of personalization and information sharing in federated scenarios—at least in our (stylized) model—including that Federated Averaging with simple client fine-tuning achieves the same asymptotic risk as the more intricate metalearning and proximal-regularized approaches and outperforming Federated Averaging without personalization. We evaluate these...

Not All Federated Learning Algorithms Are Created Equal: A Performance Evaluation Study

arXiv (Cornell University), 2024

Federated Learning (FL) emerged as a practical approach to training a model from decentralized data. The proliferation of FL led to the development of numerous FL algorithms and mechanisms. Many prior efforts have given their primary focus on accuracy of those approaches, but there exists little understanding of other aspects such as computational overheads, performance and training stability, etc. To bridge this gap, we conduct extensive performance evaluation on several canonical FL algorithms (FedAvg, FedProx, FedYogi, FedAdam, SCAFFOLD, and FedDyn) by leveraging an open-source federated learning framework called Flame. Our comprehensive measurement study reveals that no single algorithm works best across different performance metrics. A few key observations are: (1) While some state-of-the-art algorithms achieve higher accuracy than others, they incur either higher computation overheads (FedDyn) or communication overheads (SCAFFOLD). (2) Recent algorithms present smaller standard deviation in accuracy across clients than FedAvg, indicating that the advanced algorithms' performances are stable. (3) However, algorithms such as FedDyn and SCAFFOLD are more prone to catastrophic failures without the support of additional techniques such as gradient clipping. We hope that our empirical study can help the community to build best practices in evaluating FL algorithms.

Adaptive Personalized Federated Learning

ArXiv, 2020

Investigation of the degree of personalization in federated learning algorithms has shown that only maximizing the performance of the global model will confine the capacity of the local models to personalize. In this paper, we advocate an adaptive personalized federated learning (APFL) algorithm, where each client will train their local models while contributing to the global model. Information theoretically, we prove that the mixture of local and global models can reduce the generalization error. We also propose a communication-reduced bilevel optimization method, which reduces the communication rounds to O(sqrtT)O(\sqrt{T})O(sqrtT) and show that under strong convexity and smoothness assumptions, the proposed algorithm can achieve a convergence rate of O(1/T)O(1/T)O(1/T) with some residual error. The residual error is related to the gradient diversity among local models, and the gap between optimal local and global models. The extensive experiments demonstrate the effectiveness of our personalization, as ...

Optimization in Federated Learning

2019

Federated learning (FL) is an emerging branch of machine learning (ML) research, that is examining the methods for scenarios, where individual nodes possess parts of the data, and the task is to form a single common model that fits to the whole distribution. In FL, we generally use mini batch gradient descent for optimizing weights of the models that appears to work very well for federated scenarios. For traditional machine learning setups, a number of modifications has been proposed to accelerate the learning process and to help to get over challenges posed by the high dimensionality and nonconvexity of search spaces of the parameters. In this paper we present our experiments on applying different popular optimization methods for training neural networks in a federated manner. 1 Federated Learning Federated learning (FL) [1] is a new paradigm in Machine Learning (ML), that is dealing with an increasingly important distributed optimization setting, that came into view with the sprea...

A Generalized Look at Federated Learning: Survey and Perspectives

arXiv (Cornell University), 2023

Federated learning (FL) refers to a distributed machine learning framework involving learning from several decentralized edge clients without sharing local dataset. This distributed strategy prevents data leakage and enables on-device training as it updates the global model based on the local model updates. Despite offering several advantages, including data privacy and scalability, FL poses challenges such as statistical and system heterogeneity of data in federated networks, communication bottlenecks, privacy and security issues. This survey contains a systematic summarization of previous work, studies, and experiments on FL and presents a list of possibilities for FL across a range of applications and use cases. Other than that, various challenges of implementing FL and promising directions revolving around the corresponding challenges are provided.

FEDERATED LEARNING OPTIMIZATION TECHNIQUES FOR NON-IID DATA: A REVIEW

IAEME PUBLICATION, 2020

Federated learning is a distributed machine learning setting that can effectively assist multiple clients (e.g. mobile phones, IoT devices and organizations) to conduct isolated data use and machine learning modeling in accordance with user privacy protection, data security and government regulations. The participants can benefit from the federated learning training process or outcomes without sharing any local raw data. However, this special structure also determines that federated learning is bound to encounter statistical challenge which principally performs as non-independently and identically distributed (non-IID) data problem. This issue can directly affect the performance of federated learning, resulting in inferior performance of machine learning models, like poor prediction accuracy, additional communication overhead and low convergence rate. Therefore, it is of great significance for federated learning to study this non-IID data issue. At present, many optimization techniques have been proposed to tackle this problem, nonetheless they are various and lack a unified classification standard. To provide that standard, this paper reviews and classifies the optimization approaches which tackle this issue of federated learning, and summarizes the characteristics of each type of method based on several major influencing factors

Federated Learning Algorithms to Optimize the Client and Cost Selections

Mathematical Problems in Engineering

In recent years, federated learning has received widespread attention as a technology to solve the problem of data islands, and it has begun to be applied in fields such as finance, healthcare, and smart cities. The federated learning algorithm is systematically explained from three levels. First, federated learning is defined through the definition, architecture, classification of federated learning, and comparison with traditional distributed knowledge. Then, based on machine learning and deep learning, the current types of federated learning algorithms are classified, compared, and analyzed in-depth. Finally, the communication from the perspectives of cost, client selection, and aggregation method optimization, the federated learning optimization algorithms are classified. Finally, the current research status of federated learning is summarized. Finally, the three major problems and solutions of communication, system heterogeneity, and data heterogeneity faced by federated learni...

FedPerf: A Practitioners’ Guide to Performance of Federated Learning Algorithms

Federated Learning (FL) enables the edge devices to collaboratively train a joint model without sharing their local data. This decentralised and distributed approach improves user privacy, security, and trust. Different variants of FL algorithms have presented promising results on both IID and skewed Non-IID data. However, the performance of FL algorithms is found to be sensitive to the FL system parameters and hyperparameters of the used model. In practice, tuning the right set of parameter settings for an FL algorithm is an expensive task. In this preregister paper, we propose an empirical investigation on five prominent FL algorithms to discover the relation between the FL System Parameters (FLSPs) and their performance. The FLSPs adds extra complexity to FL algorithms over a traditional ML system. We hypothesise that choosing the best FL algorithm for the given FLSP is not a trivial problem. Further, we endeavour to formulate a single easy-to-use metric which can describe the pe...

Node Selection Toward Faster Convergence for Federated Learning on Non-IID Data

IEEE Transactions on Network Science and Engineering, 2022

Federated Learning (FL) is a distributed learning paradigm that enables a large number of resource-limited nodes to collaboratively train a model without data sharing. The non-independent-and-identically-distributed (non-i.i.d.) data samples invoke discrepancy between global and local objectives, making the FL model slow to converge. In this paper, we proposed Optimal Aggregation algorithm for better aggregation, which finds out the optimal subset of local updates of participating nodes in each global round, by identifying and excluding the adverse local updates via checking the relationship between the local gradient and the global gradient. Then, we proposed a Probabilistic Node Selection framework (FedPNS) to dynamically change the probability for each node to be selected based on the output of Optimal Aggregation. FedPNS can preferentially select nodes that propel faster model convergence. The convergence rate improvement of FedPNS over the commonly adopted Federated Averaging (FedAvg) algorithm is analyzed theoretically. Experimental results demonstrate the effectiveness of FedPNS in accelerating the FL convergence rate, as compared to FedAvg with random node selection.