Self-Distribution Distillation: Efficient Uncertainty Estimation
dc.contributor.author | Fathullah, Y | |
dc.contributor.author | Gales, MJF | |
dc.date.accessioned | 2022-06-15T23:30:06Z | |
dc.date.available | 2022-06-15T23:30:06Z | |
dc.identifier.isbn | 9781713863298 | |
dc.identifier.uri | https://www.repository.cam.ac.uk/handle/1810/338131 | |
dc.description.abstract | Deep learning is increasingly being applied in safety-critical domains. For these scenarios it is important to know the level of uncertainty in a model’s prediction to ensure appropriate decisions are made by the system. Deep ensembles are the de-facto standard approach to obtaining various measures of uncertainty. However, ensembles often significantly increase the resources required in the training and/or deployment phases. Approaches have been developed that typically address the costs in one of these phases. In this work we propose a novel training approach, self-distribution distillation (S2D), which is able to efficiently train a single model that can estimate uncertainties. Furthermore it is possible to build ensembles of these models and apply hierarchical ensemble distillation approaches. Experiments on CIFAR-100 showed that S2D models outperformed standard models and Monte-Carlo dropout. Additional out-of-distribution detection experiments on LSUN, Tiny ImageNet, SVHN showed that even a stan- dard deep ensemble can be outperformed using S2D based ensembles and novel distilled models. | |
dc.rights | All Rights Reserved | |
dc.rights.uri | http://www.rioxx.net/licenses/all-rights-reserved | |
dc.title | Self-Distribution Distillation: Efficient Uncertainty Estimation | |
dc.type | Conference Object | |
dc.publisher.department | Department of Engineering | |
dc.date.updated | 2022-05-18T05:02:49Z | |
prism.publicationName | Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence, UAI 2022 | |
dc.identifier.doi | 10.17863/CAM.85540 | |
dcterms.dateAccepted | 2022-05-15 | |
rioxxterms.versionofrecord | 10.17863/CAM.85540 | |
rioxxterms.version | AM | |
pubs.conference-name | Conference on Uncertainty in Artificial Intelligence | |
pubs.conference-start-date | 2022-08-01 | |
cam.orpheus.counter | 23 | * |
cam.depositDate | 2022-05-18 | |
pubs.conference-finish-date | 2022-08-05 | |
pubs.licence-identifier | apollo-deposit-licence-2-1 | |
pubs.licence-display-name | Apollo Repository Deposit Licence Agreement | |
rioxxterms.freetoread.startdate | 2023-06-15 |
Files in this item
This item appears in the following Collection(s)
-
Cambridge University Research Outputs
Research outputs of the University of Cambridge