Show simple item record

dc.contributor.authorClarke, Ross
dc.contributor.authorOldewage, Elre T
dc.contributor.authorHernández-Lobato, José Miguel
dc.date.accessioned2022-04-07T23:30:25Z
dc.date.available2022-04-07T23:30:25Z
dc.identifier.urihttps://www.repository.cam.ac.uk/handle/1810/335897
dc.description.abstractMachine learning training methods depend plentifully and intricately on hyperparameters, motivating automated strategies for their optimisation. Many existing algorithms restart training for each new hyperparameter choice, at considerable computational cost. Some hypergradient-based one-pass methods exist, but these either cannot be applied to arbitrary optimiser hyperparameters (such as learning rates and momenta) or take several times longer to train than their base models. We extend these existing methods to develop an approximate hypergradient-based hyperparameter optimiser which is applicable to any continuous hyperparameter appearing in a differentiable model weight update, yet requires only one training episode, with no restarts. We also provide a motivating argument for convergence to the true hypergradient, and perform tractable gradient-based optimisation of independent learning rates for each model parameter. Our method performs competitively from varied random hyperparameter initialisations on several UCI datasets and Fashion-MNIST (using a one-layer MLP), Penn Treebank (using an LSTM) and CIFAR-10 (using a ResNet-18), in time only 2-3x greater than vanilla training.
dc.rightsPublisher's own licence
dc.subjectcs.LG
dc.subjectcs.LG
dc.subjectstat.ML
dc.titleScalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation
dc.typeConference Object
dc.publisher.departmentDepartment of Engineering Student
dc.date.updated2022-04-07T12:21:50Z
dc.identifier.doi10.17863/CAM.83331
dcterms.dateAccepted2022-01-28
rioxxterms.versionofrecord10.17863/CAM.83331
rioxxterms.versionAM
dc.contributor.orcidClarke, Ross [0000-0001-9884-046X]
pubs.funder-project-idEngineering and Physical Sciences Research Council (2107369)
pubs.conference-nameInternational Conference on Learning Representations 2022
pubs.conference-start-date2022-04-25
cam.orpheus.counter7*
cam.depositDate2022-04-07
pubs.conference-finish-date2022-04-29
pubs.licence-identifierapollo-deposit-licence-2-1
pubs.licence-display-nameApollo Repository Deposit Licence Agreement
rioxxterms.freetoread.startdate2023-04-07


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record