Show simple item record

dc.contributor.authorColbrook, Matthew
dc.contributor.authorAntun, Vegard
dc.contributor.authorHansen, Anders
dc.date.accessioned2022-04-04T10:40:30Z
dc.date.available2022-04-04T10:40:30Z
dc.date.issued2022-03-22
dc.identifier.issn0027-8424
dc.identifier.urihttps://www.repository.cam.ac.uk/handle/1810/335732
dc.description.abstractDeep learning (DL) has had unprecedented success and is now entering scientific computing with full force. However, current DL methods typically suffer from instability, even when universal approximation properties guarantee the existence of stable neural networks (NNs). We address this paradox by demonstrating basic well-conditioned problems in scientific computing where one can prove the existence of NNs with great approximation qualities; however, there does not exist any algorithm, even randomized, that can train (or compute) such a NN. For any positive integers K>2 and L, there are cases where simultaneously 1) no randomized training algorithm can compute a NN correct to K digits with probability greater than 1/2; 2) there exists a deterministic training algorithm that computes a NN with K –1 correct digits, but any such (even randomized) algorithm needs arbitrarily many training data; and 3) there exists a deterministic training algorithm that computes a NN with K –2 correct digits using no more than L training samples. These results imply a classification theory describing conditions under which (stable) NNs with a given accuracy can be computed by an algorithm. We begin this theory by establishing sufficient conditions for the existence of algorithms that compute stable NNs in inverse problems. We introduce fast iterative restarted networks (FIRENETs), which we both prove and numerically verify are stable. Moreover, we prove that only O(|log (ϵ)|) layers are needed for an ϵ-accurate solution to the inverse problem.
dc.format.mediumPrint-Electronic
dc.publisherProceedings of the National Academy of Sciences
dc.rightsAttribution-NonCommercial-NoDerivatives 4.0 International
dc.rights.urihttps://creativecommons.org/licenses/by-nc-nd/4.0/
dc.subjectAI and deep learning
dc.subjectSmale’s 18th problem
dc.subjectinverse problems
dc.subjectsolvability complexity index hierarchy
dc.subjectstability and accuracy
dc.titleThe difficulty of computing stable and accurate neural networks: On the barriers of deep learning and Smale's 18th problem.
dc.typeArticle
dc.publisher.departmentTrinity College
dc.publisher.departmentDepartment of Applied Mathematics And Theoretical Physics
dc.date.updated2022-03-27T09:23:09Z
prism.issueIdentifier12
prism.publicationDate2022
prism.publicationNameProc Natl Acad Sci U S A
prism.startingPagee2107151119
prism.volume119
dc.identifier.doi10.17863/CAM.83167
dcterms.dateAccepted2021-10-26
rioxxterms.versionofrecord10.1073/pnas.2107151119
rioxxterms.versionVoR
dc.contributor.orcidColbrook, Matthew [0000-0003-4964-9575]
dc.identifier.eissn1091-6490
rioxxterms.typeJournal Article/Review
cam.issuedOnline2022-03-16
cam.depositDate2022-03-27
pubs.licence-identifierapollo-deposit-licence-2-1
pubs.licence-display-nameApollo Repository Deposit Licence Agreement


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record

Attribution-NonCommercial-NoDerivatives 4.0 International
Except where otherwise noted, this item's licence is described as Attribution-NonCommercial-NoDerivatives 4.0 International