Topics in Deep Generative Modelling Mathematical and Computational Aspects of Diffusion Models and Generative Adversarial Networks

Abstract

This thesis explores the theoretical and practical aspects of deep generative models with a special emphasis on score-based diffusion models. It also includes studies of theoretical underpinnings of generative adversarial networks and introduces novel algorithmic improvements to variational autoencoders. The work contributes both theoretical insights and novel algorithms, addressing areas like conditional generation, dimensionality estimation, and reduction. Firstly, we examine diffusion models from a mean-field perspective, which leads to a new theoretical insight into the differences between stochastic and deterministic sampling schemes for these models. We establish a theoretical upper bound on the Wasserstein 2-distance between distributions induced by stochastic and deterministic dynamics, linking it to the Fokker-Planck equation and its residual. Furthermore, the thesis explores the interplay between diffusion models and data manifolds. We elucidate a geometric connection between diffusion models and data manifolds, by proving that a diffusion model encodes the data manifold by approximating its normal bundle. Using this insight, we developed an new technique that employs singular value decomposition to infer intrinsic dimensionality of the underlying data manifold from a trained diffusion model. We have conducted a thorough comparison and theoretical analysis of various methods for learning conditional probability distributions using score-based diffusion models. We have proven results that offer a solid theoretical justification for one of the most effective estimators of the conditional score. Additionally, we have extended the diffusion modelling framework to a multi-speed diffusion setting, which has led to the creation of an new estimator for the conditional score. Furthermore, the research introduces a new method that integrates diffusion models with variational autoencoders (VAEs). This hybrid model can be perceived as either an enhanced VAE with a diffusion-based decoder or a method to derive a latent space from a pre-trained diffusion model. The thesis also critically evaluates the existing theory behind Wasserstein GANs, highlighting discrepancies between theory and algorithmic practice and questioning the desirability of Wasserstein distance as a loss function. Lastly, the research investigates the impact of imputation quality on machine learning classifiers in datasets with missing values. This study, initiated during the COVID-19 pandemic, assesses both classical and modern deep generative model-based imputation techniques. We quantified how the downstream classification performance is influenced by the imputation method, classification method and data missingness rate. Moreover, we examined how faithfully do different data imputation methods reproduce the distribution of the underlying dataset. Our findings suggest that many commonly used metrics for evaluating modern imputation methods are not indicative of their effectiveness in downstream classification tasks. A new evaluation approach based on sliced Wasserstein distance is proposed, proving to be a more accurate predictor of classification performance. This work was motivated by real-world clinical needs, with experiments conducted on both clinical and synthetic data sets.