Federated learning (FL) has garnered significant attention from both research and industrial communities due to its distinctive ability to facilitate collaborative learning from large-scale datasets without compromising users’ data privacy. However, current FL practices predominantly focus on supervised learning tasks, necessitating the availability of high-quality, domain-specific labels alongside the data. This prerequisite constrains the implementation of FL in numerous real-world applications where access to such labels at the edge is limited. Self-supervised learning (SSL) enables the acquisition of representations from unlabelled data, which can subsequently be employed to address various downstream tasks. Integrating SSL with FL presents considerable advantages beyond privacy-preserving training, including robust distributed representation learning, enhanced scalability, and resilience to noisy data. Despite its potential, research on SSL within the context of FL remains scarce.

This thesis endeavours to bridge this research gap by illuminating the underlying challenges and proposing potential solutions to advance the training of SSL models in FL environments, specifically within the speech, video, and image domains. First, we present a systematic investigation into the feasibility and complexities of implementing speech SSL in FL contexts concerning hardware limitations and algorithmic aspects, and provide an elementary solution to the efficiency issue of training with short input sequences. Second, we delve into the unexplored area of video-SSL in FL and propose a novel FL framework, incorporating stochastic weighted averaging during aggregation and partial weights updating, which achieves new state-of-the-art performance on downstream tasks. Third, we examine the prevalent issue of model divergence, instigated by clients’ bias in the area of image-federated SSL. We introduce a novel aggregation scheme, designed to mitigate this problem by utilising angular divergence as a contributing coefficient for weighting clients’ models at the layer level. Finally, we revisit the efficiency challenge in FL-SSL and incorporate sparsification into federated SSL model training to accelerate the deployment of such models on FL edge devices.

Overall, the original contributions of this thesis address the task of integrating SSL model training into FL environments across three pervasive domains (speech, video, and image). This work lays the groundwork for transferring SSL training to local edge devices for a wide array of real-world applications.