Towards Intelligent Federated Learning Systems

Abstract

With the introduction of data privacy laws such as GDPR, the privacy challenges of traditional machine learning have become more visible. Recent works leverage edge computing to preserve data privacy by keeping the data where it is (not shared during the training process), so-called "Edge Computing". In 2016, Google extended this idea to distributed machine learning and termed it as "Federated Learning". In a federated learning system, any device could participate in the training – regardless of its data distribution held or system performance, which brings us to a problem: how to deal with these heterogeneities? Or we take a step back. Does the device even have enough resources to initialise the training process, and if it doesn’t, can we split the workload to multiple devices wisely? In this thesis, I investigate the prior works available towards a federated learning system, using a top-down approach: from aggregation to devices, from devices to models, and from the typical federated learning paradigm (centralised horizontal federated learning) to non-typical federated learning (vertical federated learning and decentralised federated learning). We then optimise a federated learning system from top to bottom. We first maximise the resource utilisation rate on powerful devices for better accuracy, time-to-accuracy efficiency and consistency. We then investigate the paradigm of GAN training and apply the idea of federated learning and split-learning, as well as maximise the extent of parallel computing to reduce the job-completion-time for such a system. We finally investigate the problem of vertical federated learning, where the data distribution is i.i.d while the feature space is partitioned across different devices, and propose a framework called HoVeFL. We proposed these techniques to develop a more intelligent federated learning system. Our experimental results not only empirically show the feasibility of our algorithms as well as suitable scenarios for these optimisation techniques, but also highlight some useful future directions toward more intelligent federated learning systems. We hope that our research will encourage people to further work on these types of optimisations.