Answer Uncertainty and Unanswerability for Multiple-Choice Machine Reading Comprehension
View / Open Files
Conference Name
Findings of Association for Computational Linguistics 2022
Type
Conference Object
This Version
AM
Metadata
Show full item recordCitation
Raina, V., & Gales, M. Answer Uncertainty and Unanswerability for Multiple-Choice Machine Reading Comprehension. Findings of Association for Computational Linguistics 2022. https://doi.org/10.17863/CAM.83436
Abstract
Machine reading comprehension (MRC) has drawn a lot of attention as an approach for assessing the ability of systems to understand natural language. Usually systems focus on selecting the correct answer to a question given a contextual paragraph. However, for many applications of multiple-choice MRC systems there are two additional considerations. For multiple-choice exams there is often a negative marking scheme; there is a penalty for an incorrect answer. This means that the system is required to have an idea of the uncertainty in the predicted answer. The second consideration is that many multiple-choice questions have the option of none of the above (NOA) indicating that none of the answers is applicable, rather than there always being the correct answer in the list of choices. This paper investigates both of these issues by making use of predictive uncertainty. It is shown that uncertainty does allow questions that the system is not confident about to be detected. Additionally we show that uncertainty outperforms a system explicitly built with an NOA option for the ReClor corpus.
Embargo Lift Date
2023-04-11
Identifiers
External DOI: https://doi.org/10.17863/CAM.83436
This record's URL: https://www.repository.cam.ac.uk/handle/1810/336004
Statistics
Total file downloads (since January 2020). For more information on metrics see the
IRUS guide.