Graph Neural Networks for Multi-Agent Learning

Abstract

Over time, machine learning research has placed an increasing emphasis on utilising relational inductive biases. By focusing on the underlying relationships in graph structured data, it has become possible to create models with superior performance and generalisation. Given different graph topologies, these models can manifest in the form of CNNs (on grid graphs), RNNs (on line graphs), and Transformers (on fully connected graphs). However, all of these architectures can be subsumed as special cases under graph neural networks (GNNs), a framework for operating over any graph-structured data. Taking advantage of relational inductive biases, GNNs use local filters to learn functions that generalise over high-dimensional data. They are particularly useful in the context of multi-agent learning, where most data is structured as a graph (e.g., communication links in a multi-robot team generate a graph connectivity). In this thesis, we study the field of multi-agent learning. Recent advances in the domain are promising, but further innovation is required to tackle problems such as learning under partial observability and facilitating collaborative behaviour. GNNs provide a useful framework for solving these problems, as their decentralised architecture allows them to utilise global information while maintaining generalisation and sample-efficiency. Despite these benefits, existing GNN-based approaches to multi-agent problems have only been applied with a limited scope. In this thesis, we investigate the weaknesses of current GNN architectures, and propose extensions to improve their capabilities. Furthermore, we branch out into new learning paradigms, allowing GNN-based approaches to tackle new applications. We start this thesis by developing a modular framework to analyse existing GNN-based approaches to multi-robot tasks. By ablating over the submodules within our framework, we can draw conclusions about the best allocation of representational complexity within a GNN architecture. Our analysis highlights the need for mapping to a learned latent space prior to aggregation, allowing the network to preserve the most important information. To avoid the loss of information through naive aggregation, our subsequent work strives to find an architecture that allows the aggregator itself to be learned. We introduce a novel method that parametrises the space of all standard aggregators, and validate its performance in graph learning problems. Next, we expand our focus to the related problem of learning over sets. We introduce a novel set autoencoder, allowing a bijective mapping from sets to fixed-size embeddings to be learned in an unsupervised manner. To demonstrate the usefulness of this architecture, we use it to create a task-agnostic multi-agent communication strategy. In our final work, we use GNNs to tackle the credit assignment problem in multi-agent reinforcement learning. Leveraging the decentralised manner in which GNNs combine local and aggregated neighbouring information, we perform value factorisation with a GNN-based architecture. This approach maintains the ability to represent non-factorisable value functions, yet performs factorisation when it is possible. We conclude this thesis by reflecting over our contributions, which span the fields of supervised learning, unsupervised learning, and reinforcement learning. The new architectures that we have developed open up new avenues of research---not only in applications, but in extensions to our methods. For each research topic in our thesis, we propose future work that can further impact the field.