Discriminative Bayesian Filtering Lends Momentum to the Stochastic Newton Method for Minimizing Log-Convex Functions
| cam.depositDate | 2022-05-21 | |
| cam.orpheus.counter | 81 | * |
| dc.contributor.author | Burkhart, Michael | |
| dc.contributor.orcid | Burkhart, Michael [0000-0002-2772-5840] | |
| dc.date.accessioned | 2022-05-23T23:30:30Z | |
| dc.date.available | 2022-05-23T23:30:30Z | |
| dc.date.updated | 2022-05-21T08:48:34Z | |
| dc.description.abstract | To minimize the average of a set of log-convex functions, the stochastic Newton method iteratively updates its estimate using subsampled versions of the full objective's gradient and Hessian. We contextualize this optimization problem as sequential Bayesian inference on a latent state-space model with a discriminatively-specified observation process. Applying Bayesian filtering then yields a novel optimization algorithm that considers the entire history of gradients and Hessians when forming an update. We establish matrix-based conditions under which the effect of older observations diminishes over time, in a manner analogous to Polyak's heavy ball momentum. We illustrate various aspects of our approach with an example and review other relevant innovations for the stochastic Newton method. | |
| dc.identifier.doi | 10.17863/CAM.84825 | |
| dc.identifier.issn | 1862-4472 | |
| dc.identifier.uri | https://www.repository.cam.ac.uk/handle/1810/337413 | |
| dc.language.iso | eng | |
| dc.publisher | Springer | |
| dc.publisher.department | Department of Psychology | |
| dc.rights | Publisher's own licence | |
| dc.title | Discriminative Bayesian Filtering Lends Momentum to the Stochastic Newton Method for Minimizing Log-Convex Functions | |
| dc.type | Article | |
| dcterms.dateAccepted | 2022-05-19 | |
| prism.publicationName | Optimization Letters | |
| pubs.licence-display-name | Apollo Repository Deposit Licence Agreement | |
| pubs.licence-identifier | apollo-deposit-licence-2-1 | |
| rioxxterms.type | Journal Article/Review | |
| rioxxterms.version | AM |
Files
Original bundle
1 - 1 of 1
No Thumbnail Available
- Name:
- filtered_stochastic_netwon.pdf
- Size:
- 260.64 KB
- Format:
- Adobe Portable Document Format
- Description:
- Accepted version