In this thesis we investigate the use of deep learning for cross-modality and multi-modal image-based profiling applications. In particular, we explore the utility of the brightfield image modality with deep generative models, and also propose new methods to integrate metadata labels freely obtained in high-content screening into deep learning architectures.

The use of automated microscopy in high-content phenotypic screening of cells treated with compounds or genetic perturbations produces a large amount of high-dimensional data. One example which we focus on in this thesis is Cell Painting, a standardized pipeline used to capture rich cell morphology. Typically, fluorescent stained images are the focus of image-based profiling, where the aim is to extract meaningful features from the images which can be used to represent the biology of the cells, and compare the effects of the treatments. Image-based profiling is central to screening in drug discovery, and is used to guide the selection of drug candidates to take to clinical trials.

With an abundance of large databases of high-dimensional images, deep learning for image-based profiling has risen as its own sub-field, and there are now entire drug discovery pipelines selecting drugs to take to clinical trials built upon deep learning foundations. This is possible due to advances in deep learning in computer vision. In this thesis we explore and adapt recent and powerful deep learning approaches including the generative adversarial network, self-supervised vision transformers, and the diffusion denoising probabilistic model.

A major challenge in this space is to use state-of-the-art frameworks from computer vision in a way which is sensitive to the challenges of drug discovery. Image-based profiling, and particularly deep learning in image-based profiling, is a new and maturing field. In this thesis we tackle the unaddressed challenge of incorporating the cheaper, easier to obtain but classically less informative brightfield modality, as well as using underutilised but freely available metadata alongside the images to guide the training of deep neural networks. As we are working in interdisciplinary science, we do this while making models which are visually interpretable to the many people working in drug discovery who are not familiar with, or potentially sceptical of deep learning.

This thesis presents the first study to predict all five fluorescent Cell Painting channels from brightfield images. We explore the potential, benefits and limitations of this new approach. Next, we introduce a weakly-self supervised learning framework to learn feature representations which are guided by informative metadata. Finally, we present the first study to use a diffusion model with high-content microscopy images. We generate entire plates of synthetic Cell Painting images of exceptional image quality to make predictions about the information these models are capable of capturing, and investigate if this can also be guided by labels.