Design of Deep Neural Networks Formulated as Optimisation Problems
View / Open Files
Authors
Zhang, Sushen
Date
2021-10-01Awarding Institution
University of Cambridge
Qualification
Doctor of Philosophy (PhD)
Type
Thesis
Metadata
Show full item recordCitation
Zhang, S. (2021). Design of Deep Neural Networks Formulated as Optimisation Problems (Doctoral thesis). https://doi.org/10.17863/CAM.82337
Abstract
The design of deep neural networks (DNNs) involves the explicit definition
of network architecture as well as the training of the network weights. Each
process can be formulated into an optimisation algorithm and can be
investigated with regard to optimisation performance. The training of the
network weights is defined as a minimisation of the objective function with
regard to network parameters. The architecture search is an optimisation of
the objective function with regard to the presence/absence of layers or
neurons. I draw similarity between the two scenarios, and propose
frameworks that define either the training or the architectural optimisation of
DNNs, or a combination of both. The contribution of the thesis is six-fold, in
which I propose: 1) a quasi-Newton training algorithm based on Truncated
Newton and Gradient Flow methods, 2) a lifting scheme to allow network
sparsification, 3) a lifting framework to automatically evolve neural
architectures, 4) a multi-scale hierarchical search framework involving
sensitivity analysis suitable for the training of neural networks, 5) a heuristic
search algorithm for architectural optimisation of a dynamic model, and 6) a
dynamic cascade learning model solved in the context of de novo drug design. In each contribution, I define the optimisation problem and solve the
optimisation problem under different frameworks. The ultimate aim of this
research is to facilitate the democratisation of AI, enabling people with less
domain expertise to participate in the design of a deep neural network under
a guided framework.
Keywords
Neural Network, Optimisation, Architecture Search
Relationships
Is supplemented by: MNIST datasetOILDROPLET datasetPSA dataset
Sponsorship
Cambridge Overseas Trust and China Scholarship Council
Identifiers
This record's DOI: https://doi.org/10.17863/CAM.82337
Statistics
Total file downloads (since January 2020). For more information on metrics see the
IRUS guide.