The design of deep neural networks (DNNs) involves the explicit definition of network architecture as well as the training of the network weights. Each process can be formulated into an optimisation algorithm and can be investigated with regard to optimisation performance. The training of the network weights is defined as a minimisation of the objective function with regard to network parameters. The architecture search is an optimisation of the objective function with regard to the presence/absence of layers or neurons. I draw similarity between the two scenarios, and propose frameworks that define either the training or the architectural optimisation of DNNs, or a combination of both. The contribution of the thesis is six-fold, in which I propose: 1) a quasi-Newton training algorithm based on Truncated Newton and Gradient Flow methods, 2) a lifting scheme to allow network sparsification, 3) a lifting framework to automatically evolve neural architectures, 4) a multi-scale hierarchical search framework involving sensitivity analysis suitable for the training of neural networks, 5) a heuristic search algorithm for architectural optimisation of a dynamic model, and 6) a dynamic cascade learning model solved in the context of de novo drug design. In each contribution, I define the optimisation problem and solve the optimisation problem under different frameworks. The ultimate aim of this research is to facilitate the democratisation of AI, enabling people with less domain expertise to participate in the design of a deep neural network under a guided framework.