Repository logo
 

Predictive Complexity Priors

Published version
Peer-reviewed

Type

Conference Object

Change log

Authors

Nalisnick, E 
Gordon, J 
Hernández-Lobato, JM 

Abstract

Specifying a Bayesian prior is notoriously difficult for complex models such as neural networks. Reasoning about parameters is made challenging by the high-dimensionality and over-parameterization of the space. Priors that seem benign and uninformative can have unintuitive and detrimental effects on a model's predictions. For this reason, we propose predictive complexity priors: a functional prior that is defined by comparing the model's predictions to those of a reference model. Although originally defined on the model outputs, we transfer the prior to the model parameters via a change of variables. The traditional Bayesian workflow can then proceed as usual. We apply our predictive complexity prior to high-dimensional regression, reasoning over neural network depth, and sharing of statistical strength for few-shot learning.

Description

Keywords

Journal Title

Proceedings of Machine Learning Research

Conference Name

24th International Conference on Artificial Intelligence and Statistics (AISTATS)

Journal ISSN

2640-3498
2640-3498

Volume Title

130

Publisher

Rights

All rights reserved