Show simple item record

dc.contributor.authorChitturi, Sathya R
dc.contributor.authorVerpoort, Philipp
dc.contributor.authorLee, Alpha A
dc.contributor.authorWales, David
dc.date.accessioned2020-02-04T00:31:06Z
dc.date.available2020-02-04T00:31:06Z
dc.date.issued2020-06-01
dc.identifier.issn2632-2153
dc.identifier.urihttps://www.repository.cam.ac.uk/handle/1810/301682
dc.description.abstract<jats:title>Abstract</jats:title> <jats:p>We investigate the structure of the loss function landscape for neural networks subject to dataset mislabelling, increased training set diversity, and reduced node connectivity, using various techniques developed for energy landscape exploration. The benchmarking models are classification problems for atomic geometry optimisation and hand-written digit prediction. We consider the effect of varying the size of the atomic configuration space used to generate initial geometries and find that the number of stationary points increases rapidly with the size of the training configuration space. We introduce a measure of node locality to limit network connectivity and perturb permutational weight symmetry, and examine how this parameter affects the resulting landscapes. We find that highly-reduced systems have low capacity and exhibit landscapes with very few minima. On the other hand, small amounts of reduced connectivity can enhance network expressibility and can yield more complex landscapes. Investigating the effect of deliberate classification errors in the training data, we find that the variance in testing AUC, computed over a sample of minima, grows significantly with the training error, providing new insight into the role of the variance-bias trade-off when training under noise. Finally, we illustrate how the number of local minima for networks with two and three hidden layers, but a comparable number of variable edge weights, increases significantly with the number of layers, and as the number of training data decreases. This work helps shed further light on neural network loss landscapes and provides guidance for future work on neural network training and optimisation.</jats:p>
dc.description.sponsorshipepsrc
dc.publisherIOP Publishing
dc.rightsAll rights reserved
dc.titlePerspective: new insights from loss function landscapes of neural networks
dc.typeArticle
prism.endingPage023002
prism.issueIdentifier2
prism.publicationNameMachine Learning: Science and Technology
prism.startingPage023002
prism.volume1
dc.identifier.doi10.17863/CAM.48753
dcterms.dateAccepted2020-01-31
rioxxterms.versionofrecord10.1088/2632-2153/ab7aef
rioxxterms.versionAM
rioxxterms.licenseref.urihttp://www.rioxx.net/licenses/all-rights-reserved
rioxxterms.licenseref.startdate2020-01-31
dc.contributor.orcidVerpoort, Philipp [0000-0003-1319-5006]
dc.contributor.orcidWales, David [0000-0002-3555-6645]
dc.identifier.eissn2632-2153
rioxxterms.typeJournal Article/Review
pubs.funder-project-idEngineering and Physical Sciences Research Council (EP/N035003/1)
cam.issuedOnline2020-04-09
cam.orpheus.successTue Feb 01 18:59:15 GMT 2022 - Embargo updated*
cam.orpheus.counter90
rioxxterms.freetoread.startdate2020-04-09


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record