Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation
View / Open Files
Authors
Clarke, Ross M
Oldewage, Elre T
Hernández-Lobato, José Miguel
Conference Name
International Conference on Learning Representations 2022
Type
Conference Object
This Version
AM
Metadata
Show full item recordCitation
Clarke, R. M., Oldewage, E. T., & Hernández-Lobato, J. M. Scalable One-Pass Optimisation of High-Dimensional Weight-Update
Hyperparameters by Implicit Differentiation. International Conference on Learning Representations 2022. https://doi.org/10.17863/CAM.83331
Abstract
Machine learning training methods depend plentifully and intricately on
hyperparameters, motivating automated strategies for their optimisation. Many
existing algorithms restart training for each new hyperparameter choice, at
considerable computational cost. Some hypergradient-based one-pass methods
exist, but these either cannot be applied to arbitrary optimiser
hyperparameters (such as learning rates and momenta) or take several times
longer to train than their base models. We extend these existing methods to
develop an approximate hypergradient-based hyperparameter optimiser which is
applicable to any continuous hyperparameter appearing in a differentiable model
weight update, yet requires only one training episode, with no restarts. We
also provide a motivating argument for convergence to the true hypergradient,
and perform tractable gradient-based optimisation of independent learning rates
for each model parameter. Our method performs competitively from varied random
hyperparameter initialisations on several UCI datasets and Fashion-MNIST (using
a one-layer MLP), Penn Treebank (using an LSTM) and CIFAR-10 (using a
ResNet-18), in time only 2-3x greater than vanilla training.
Keywords
cs.LG, cs.LG, stat.ML
Sponsorship
Engineering and Physical Sciences Research Council (2107369)
Identifiers
External DOI: https://doi.org/10.17863/CAM.83331
This record's URL: https://www.repository.cam.ac.uk/handle/1810/335897
Rights
Publisher's own licence
Statistics
Total file downloads (since January 2020). For more information on metrics see the
IRUS guide.
Recommended or similar items
The current recommendation prototype on the Apollo Repository will be turned off on 03 February 2023. Although the pilot has been fruitful for both parties, the service provider IKVA is focusing on horizon scanning products and so the recommender service can no longer be supported. We recognise the importance of recommender services in supporting research discovery and are evaluating offerings from other service providers. If you would like to offer feedback on this decision please contact us on: support@repository.cam.ac.uk