Repository logo

Human Uncertainty in Concept-Based AI Systems

Published version

Repository DOI


Conference Object

Change log


Placing a human in the loop may help abate the risks of deploying AI systems in safety-critical settings ( e.g., a clinician working with a medical AI system). However, mitigating risks arising from human error and uncertainty within such human-AI interactions is an important and understudied issue. In this work, we study human uncertainty in the context of concept-based models, a family of AI systems that enable human feedback via concept interventions where an expert intervenes on human-interpretable concepts relevant to the task. Prior work in this space often assumes that humans are oracles who are always certain and correct. Yet, real-world decision-making by humans is prone to occasional mistakes and uncertainty. We study how existing concept-based models deal with uncertain interventions from humans using two novel datasets: \texttt{UMNIST}, a visual dataset with controlled simulated uncertainty based on the MNIST dataset, and \texttt{CUB-S}, a relabeling of the popular \texttt{CUB} concept dataset with rich, densely-annotated soft labels from humans. We show that training with uncertain concept labels may help mitigate weaknesses of concept-based systems when handling uncertain interventions. These results allow us to identify several open challenges, which we argue can be tackled through future multidisciplinary research on building interactive uncertainty-aware systems. To facilitate further research, we release a new elicitation platform, \texttt{UElic}, to collect uncertain feedback from humans in collaborative prediction tasks.



46 Information and Computing Sciences, 4608 Human-Centred Computing, Clinical Research, Generic health relevance

Journal Title

AIES 2023 - Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society

Conference Name

AIES '23: AAAI/ACM Conference on AI, Ethics, and Society

Journal ISSN

Volume Title


KMC gratefully acknowledges support from the Marshall Commission and the Cambridge Trust. MEZ acknowledges support from the Gates Cambridge Trust via the Gates Cambridge Scholarship. NR is supported by a Churchill Scholarship. UB acknowledges support from DeepMind and the Leverhulme Trust via the Leverhulme Centre for the Future of Intelligence (CFI), and from the Mozilla Foundation. MJ is supported by the EPSRC grant EP/T019603/1. AW acknowledges support from a Turing AI Fellowship under grant EP/V025279/1, The Alan Turing Institute, and the Leverhulme Trust via CFI. IS is supported by an NSERC fellowship (567554-2022).