Repository logo
 

Structural Priors in Deep Neural Networks

Simple item page
cam.restrictionthesis_access_open
cam.supervisorCipolla, Roberto
cam.supervisorCriminisi, Antonio
cam.supervisor.orcidCipolla, Roberto [0000-0002-8999-2151]
cam.supervisor.orcidCriminisi, Antonio [0000-0001-7976-3374]
cam.thesis.fundingfalse
dc.contributor.authorIoannou, Yani Andrew
dc.contributor.orcidIoannou, Yani Andrew [0000-0002-9797-5888]
dc.date.accessioned2018-08-22T10:19:41Z
dc.date.available2018-08-22T10:19:41Z
dc.date.issued2018-10-20
dc.date.submitted2017-09-28
dc.date.updated2018-08-21T00:46:57Z
dc.description.abstractDeep learning has in recent years come to dominate the previously separate fields of research in machine learning, computer vision, natural language understanding and speech recognition. Despite breakthroughs in training deep networks, there remains a lack of understanding of both the optimization and structure of deep networks. The approach advocated by many researchers in the field has been to train monolithic networks with excess complexity, and strong regularization --- an approach that leaves much to desire in efficiency. Instead we propose that carefully designing networks in consideration of our prior knowledge of the task and learned representation can improve the memory and compute efficiency of state-of-the art networks, and even improve generalization --- what we propose to denote as structural priors. We present two such novel structural priors for convolutional neural networks, and evaluate them in state-of-the-art image classification CNN architectures. The first of these methods proposes to exploit our knowledge of the low-rank nature of most filters learned for natural images by structuring a deep network to learn a collection of mostly small, low-rank, filters. The second addresses the filter/channel extents of convolutional filters, by learning filters with limited channel extents. The size of these channel-wise basis filters increases with the depth of the model, giving a novel sparse connection structure that resembles a tree root. Both methods are found to improve the generalization of these architectures while also decreasing the size and increasing the efficiency of their training and test-time computation. Finally, we present work towards conditional computation in deep neural networks, moving towards a method of automatically learning structural priors in deep networks. We propose a new discriminative learning model, conditional networks, that jointly exploit the accurate representation learning capabilities of deep neural networks with the efficient conditional computation of decision trees. Conditional networks yield smaller models, and offer test-time flexibility in the trade-off of computation vs. accuracy.
dc.description.sponsorshipFunded by a Microsoft Research PhD Scholarship
dc.identifier.doi10.17863/CAM.26357
dc.identifier.urihttps://www.repository.cam.ac.uk/handle/1810/278976
dc.language.isoen
dc.publisher.collegeJesus College
dc.publisher.departmentEngineering
dc.publisher.institutionUniversity of Cambridge
dc.rightsAll rights reserved
dc.rightsAll Rights Reserveden
dc.rights.urihttps://www.rioxx.net/licenses/all-rights-reserved/en
dc.subjectDeep Learning
dc.subjectNeural Networks
dc.subjectMachine Learning
dc.subjectComputer Vision
dc.subjectStructural Priors
dc.subjectFilter Groups
dc.subjectLow Rank
dc.subjectConvolution
dc.subjectDeep Neural Networks
dc.subjectConvolutional Neural Networks
dc.subjectCNN
dc.subjectEfficient
dc.titleStructural Priors in Deep Neural Networks
dc.typeThesis
dc.type.qualificationlevelDoctoral
dc.type.qualificationnameDoctor of Philosophy (PhD)
dc.type.qualificationtitleDoctor of Philosophy in Information Engineering

Files

Original bundle
Now showing 1 - 2 of 2
Thumbnail Image
Name:
thesisv2.pdf
Size:
15.29 MB
Format:
Adobe Portable Document Format
Description:
Thesis - minor changes to correct formatting
Licence
https://www.rioxx.net/licenses/all-rights-reserved/
Download
Thumbnail Image
Name:
thesis.pdf
Size:
15.3 MB
Format:
Adobe Portable Document Format
Description:
Thesis - original
Licence
https://www.rioxx.net/licenses/all-rights-reserved/
Download

Collections

Theses - Engineering